2020-07-07 04:25:45,496 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2020-07-07 04:25:45,525 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2020-07-07 04:25:45,534 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2020-07-07 04:25:45,535 INFO [rotate_logs] log files rotated 2020-07-07 04:25:45,541 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2020-07-07 09:02:02,655 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/12/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200707/12/ 2020-07-07 09:02:02,656 INFO [collect_weather] 2020-07-07 2.5km weather forecast 12 collection complete 2020-07-07 09:02:02,659 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-07 09:02:02,953 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['nowcast'], host='localhost') 2020-07-07 09:02:02,962 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2020-07-07 09:02:03,036 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2020-07-07 09:02:03,039 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2020-07-07 09:02:04,926 INFO [collect_weather] running in process 19935 2020-07-07 09:02:04,927 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:02:04,928 INFO [collect_weather] publishing log messages to tcp://*:5562 2020-07-07 09:02:04,928 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:02:04,972 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/18/ 2020-07-07 09:02:05,218 INFO [download_live_ocean] running in process 19934 2020-07-07 09:02:05,225 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:02:05,226 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2020-07-07 09:02:05,227 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:02:05,228 INFO [download_live_ocean] downloading Salish Sea western boundary daily averaged Live Ocean file for 2020-07-07 2020-07-07 09:02:05,253 INFO [grib_to_netcdf] running in process 19933 2020-07-07 09:02:05,254 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:02:05,255 INFO [grib_to_netcdf] publishing log messages to tcp://*:5567 2020-07-07 09:02:05,256 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:02:07,367 INFO [get_NeahBay_ssh] running in process 19932 2020-07-07 09:02:07,368 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:02:07,369 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5568 2020-07-07 09:02:07,369 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:02:09,588 INFO [download_live_ocean] Deflating in up to 6 concurrent sub-processes 2020-07-07 09:02:10,650 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20200707/low_passed_UBC.nc 2020-07-07 09:02:10,651 INFO [download_live_ocean] 2020-07-07 Live Ocean file for Salish Sea western boundary downloaded 2020-07-07 09:02:10,654 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2020-07-07 09:02:10,900 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:02:11,643 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for nowcast complete 2020-07-07 09:02:11,648 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker 2020-07-07 09:02:13,596 INFO [make_live_ocean_files] running in process 20014 2020-07-07 09:02:13,597 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:02:13,598 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564 2020-07-07 09:02:13,598 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:02:13,599 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2020-07-07 Live Ocean run 2020-07-07 09:03:00,404 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: / 2020-07-07 09:03:00,405 INFO [make_live_ocean_files] 2020-07-07 Live Ocean western boundary conditions files created 2020-07-07 09:03:00,407 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2020-07-07 09:03:50,043 INFO [grib_to_netcdf] 2020-07-07 NEMO-atmos forcing file for nowcast+ created 2020-07-07 09:03:50,046 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2020-07-07 09:03:50,054 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2020-07-07 09:03:50,059 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2020-07-07 09:03:50,067 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'nowcast+'], host='localhost') 2020-07-07 09:03:50,072 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2020-07-07 09:03:50,077 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2020-07-07 09:03:50,984 INFO [ping_erddap] running in process 21346 2020-07-07 09:03:50,985 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:03:50,986 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-07 09:03:50,986 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:03:50,988 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2020-07-07 09:03:50,989 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 09:03:51,046 INFO [upload_forcing] running in process 21344 2020-07-07 09:03:51,047 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:03:51,048 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-07 09:03:51,049 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:03:51,050 INFO [upload_forcing] running in process 21342 2020-07-07 09:03:51,051 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:03:51,052 INFO [upload_forcing] publishing log messages to tcp://*:5568 2020-07-07 09:03:51,053 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:03:51,070 INFO [upload_forcing] running in process 21343 2020-07-07 09:03:51,071 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:03:51,072 INFO [upload_forcing] publishing log messages to tcp://*:5569 2020-07-07 09:03:51,073 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:03:51,081 INFO [upload_forcing] running in process 21345 2020-07-07 09:03:51,082 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:03:51,083 INFO [upload_forcing] publishing log messages to tcp://*:5570 2020-07-07 09:03:51,084 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:03:51,560 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 144, in _upload_ssh_files ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 143, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 759, in put return self.putfo(fl, remotepath, file_size, callback, confirm) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 714, in putfo with self.file(remotepath, "wb") as fr: File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 372, in open t, msg = self._request(CMD_OPEN, filename, imode, attrblock) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 813, in _request return self._read_response(num) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 865, in _read_response self._convert_status(msg) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 894, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 102, in upload_forcing _upload_ssh_files(sftp_client, run_type, run_date, config, host_name, host_config) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 151, in _upload_ssh_files fcst.symlink_to(localpath) File "/SalishSeaCast/nowcast-env/lib/python3.8/pathlib.py", line 1374, in symlink_to self._accessor.symlink(target, self, target_is_directory) File "/SalishSeaCast/nowcast-env/lib/python3.8/pathlib.py", line 442, in symlink return os.symlink(a, b) FileExistsError: [Errno 17] File exists: '/results/forcing/sshNeahBay/obs/ssh_y2020m07d06.nc' -> '/results/forcing/sshNeahBay/fcst/ssh_y2020m07d06.nc' 2020-07-07 09:03:55,336 INFO [upload_forcing] nowcast+ 2020-07-07 forcing files upload to orcinus-nowcast-agrif completed 2020-07-07 09:03:55,340 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 09:03:56,619 INFO [upload_forcing] nowcast+ 2020-07-07 forcing files upload to arbutus.cloud-nowcast completed 2020-07-07 09:03:56,623 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 09:03:56,627 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:03:57,591 INFO [make_forcing_links] running in process 21380 2020-07-07 09:03:57,592 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:03:57,593 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2020-07-07 09:03:57,594 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:03:58,814 INFO [make_forcing_links] nowcast+ 2020-07-07 forcing file links on arbutus.cloud-nowcast created 2020-07-07 09:03:58,818 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-07 09:03:58,825 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 09:04:01,093 INFO [run_NEMO] running in process 31337 2020-07-07 09:04:01,094 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:04:01,095 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-07 09:04:01,586 INFO [upload_forcing] nowcast+ 2020-07-07 forcing files upload to graham-hindcast completed 2020-07-07 09:04:01,593 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 09:04:01,978 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:04:02,065 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/07jul20nowcast_2020-07-07T090401.154876-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-07 09:04:02,081 INFO [run_NEMO] nowcast NEMO run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 09:04:02,089 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-07 09:04:02,100 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-07 09:04:04,062 INFO [watch_NEMO] running in process 31602 2020-07-07 09:04:04,063 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:04:04,064 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-07 09:04:04,065 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:04:04,176 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-07 09:09:04,279 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7504271 = 2020-07-07 04:46:40 UTC, 19.9% complete 2020-07-07 09:14:04,393 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7504762 = 2020-07-07 10:14:00 UTC, 42.7% complete 2020-07-07 09:19:04,495 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7505247 = 2020-07-07 15:37:20 UTC, 65.1% complete 2020-07-07 09:24:04,596 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7505726 = 2020-07-07 20:56:40 UTC, 87.3% complete 2020-07-07 09:29:04,700 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2020-07-07 09:29:04,705 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-07 09:29:05,003 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast'], host='localhost') 2020-07-07 09:29:05,008 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-07 09:29:05,015 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-07 09:29:05,019 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:29:06,119 INFO [download_results] running in process 21829 2020-07-07 09:29:06,123 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:06,124 INFO [download_results] publishing log messages to tcp://*:5564 2020-07-07 09:29:06,125 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:06,126 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/07jul20 to /results/SalishSea/nowcast-blue.201905 2020-07-07 09:29:07,607 INFO [get_NeahBay_ssh] running in process 21826 2020-07-07 09:29:07,608 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:07,609 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5567 2020-07-07 09:29:07,610 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:10,151 INFO [make_fvcom_boundary] running in process 31868 2020-07-07 09:29:10,152 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:10,153 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580 2020-07-07 09:29:10,154 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:10,154 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 nowcast run from 2020-07-07 NEMO run 2020-07-07 09:29:10,157 INFO [make_fvcom_boundary] running in process 31870 2020-07-07 09:29:10,158 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:10,159 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5581 2020-07-07 09:29:10,159 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:10,160 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for r12 nowcast run from 2020-07-07 NEMO run 2020-07-07 09:29:11,819 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast complete 2020-07-07 09:29:11,823 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker 2020-07-07 09:29:11,832 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'ssh'], host='localhost') 2020-07-07 09:29:12,750 INFO [upload_forcing] running in process 21843 2020-07-07 09:29:12,751 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:12,752 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-07 09:29:12,753 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:13,984 INFO [upload_forcing] ssh 2020-07-07 forcing files upload to arbutus.cloud-nowcast completed 2020-07-07 09:29:13,990 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 09:29:14,002 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:29:14,991 INFO [make_forcing_links] running in process 21850 2020-07-07 09:29:14,992 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:14,993 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2020-07-07 09:29:14,994 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:16,067 INFO [make_forcing_links] ssh 2020-07-07 forcing file links on arbutus.cloud-nowcast created 2020-07-07 09:29:16,072 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-07 09:29:16,082 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 09:29:18,451 INFO [run_NEMO] running in process 32206 2020-07-07 09:29:18,453 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:18,454 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-07 09:29:18,455 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:19,579 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/07jul20forecast_2020-07-07T092918.544415-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-07 09:29:19,600 INFO [run_NEMO] forecast NEMO run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 09:29:19,605 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-07 09:29:19,616 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-07 09:29:21,684 INFO [watch_NEMO] running in process 32471 2020-07-07 09:29:21,686 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:29:21,687 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-07 09:29:21,688 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:29:21,778 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-07 09:31:03,121 INFO [download_results] nowcast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 09:31:03,128 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-07 09:31:03,144 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:31:03,152 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2020-07-06'], host='localhost') 2020-07-07 09:31:03,227 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:31:06,565 INFO [make_CHS_currents_file] running in process 21865 2020-07-07 09:31:06,567 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:06,568 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2020-07-07 09:31:06,569 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:08,106 INFO [make_plots] running in process 21863 2020-07-07 09:31:08,107 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:08,107 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-07 09:31:08,108 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:08,113 INFO [make_plots] running in process 21864 2020-07-07 09:31:08,167 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:08,176 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-07 09:31:08,182 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:08,646 INFO [make_plots] preparing render list for 2020-07-06 NEMO nowcast-blue comparison figures 2020-07-07 09:31:08,672 INFO [make_plots] preparing render list for 2020-07-07 NEMO nowcast-blue research figures 2020-07-07 09:31:14,844 INFO [make_plots] added salinity_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-blue research render list 2020-07-07 09:31:20,174 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-blue research render list 2020-07-07 09:31:20,176 INFO [make_plots] added Currents_sections_and_surface figure to 2020-07-07 NEMO nowcast-blue research render list 2020-07-07 09:31:20,177 INFO [make_plots] added Currents_at_VENUS_Central figure to 2020-07-07 NEMO nowcast-blue research render list 2020-07-07 09:31:20,178 INFO [make_plots] added Currents_at_VENUS_East figure to 2020-07-07 NEMO nowcast-blue research render list 2020-07-07 09:31:20,179 INFO [make_plots] starting to render nemo nowcast research 07jul20 figures 2020-07-07 09:31:20,280 INFO [make_plots] added SH_wind figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,282 INFO [make_plots] added Compare_VENUS_East figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,282 INFO [make_plots] added Compare_VENUS_Central figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,283 INFO [make_plots] added Compare_VENUS_Delta_DDL figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,284 INFO [make_plots] added Central_ADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,288 INFO [make_plots] added Central_depavADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,301 INFO [make_plots] added Central_timeavADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,302 INFO [make_plots] added East_ADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,303 INFO [make_plots] added East_depavADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,305 INFO [make_plots] added East_timeavADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,306 INFO [make_plots] added ddl_ADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,307 INFO [make_plots] added ddl_depavADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,307 INFO [make_plots] added ddl_timeavADCP figure to 2020-07-06 NEMO nowcast-blue comparison render list 2020-07-07 09:31:20,308 INFO [make_plots] starting to render nemo nowcast comparison 06jul20 figures 2020-07-07 09:31:31,849 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/bdy_r12_nowcast_brcl_20200707.nc 2020-07-07 09:31:31,854 INFO [make_fvcom_boundary] FVCOM r12 nowcast run boundary condition file for 2020-07-07 created on arbutus.cloud-nowcast 2020-07-07 09:31:31,861 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker 2020-07-07 09:31:31,879 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['r12', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:31:31,883 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 09:31:35,841 INFO [make_fvcom_atmos_forcing] running in process 21910 2020-07-07 09:31:35,842 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:35,843 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5569 2020-07-07 09:31:35,844 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:35,845 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for r12 nowcast run from 2020-07-07 HRDPS GRIB files 2020-07-07 09:31:36,147 INFO [make_fvcom_rivers_forcing] running in process 32564 2020-07-07 09:31:36,148 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:36,148 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582 2020-07-07 09:31:36,150 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:36,150 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for r12 nowcast run from 2020-07-07 NEMO runoff forcing files 2020-07-07 09:31:36,373 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/rivers_r12_nowcast_20200707.nc 2020-07-07 09:31:36,374 INFO [make_fvcom_rivers_forcing] FVCOM r12 nowcast run rivers forcing file for 2020-07-07 created on arbutus.cloud-nowcast 2020-07-07 09:31:36,380 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker 2020-07-07 09:31:43,172 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_nowcast_brcl_20200707.nc 2020-07-07 09:31:43,177 INFO [make_fvcom_boundary] FVCOM x2 nowcast run boundary condition file for 2020-07-07 created on arbutus.cloud-nowcast 2020-07-07 09:31:43,185 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker 2020-07-07 09:31:43,195 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:31:43,361 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 09:31:46,236 INFO [make_fvcom_atmos_forcing] running in process 21917 2020-07-07 09:31:46,237 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:46,238 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5570 2020-07-07 09:31:46,239 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:46,240 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 nowcast run from 2020-07-07 HRDPS GRIB files 2020-07-07 09:31:47,476 INFO [make_fvcom_rivers_forcing] running in process 32656 2020-07-07 09:31:47,478 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:31:47,479 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582 2020-07-07 09:31:47,480 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:31:47,481 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 nowcast run from 2020-07-07 NEMO runoff forcing files 2020-07-07 09:31:47,698 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_nowcast_20200707.nc 2020-07-07 09:31:47,700 INFO [make_fvcom_rivers_forcing] FVCOM x2 nowcast run rivers forcing file for 2020-07-07 created on arbutus.cloud-nowcast 2020-07-07 09:31:47,703 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker 2020-07-07 09:32:03,734 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_hfx_20200707.nc 2020-07-07 09:32:09,738 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_hfx_20200707.nc 2020-07-07 09:32:12,096 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_precip_20200707.nc 2020-07-07 09:32:16,578 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_precip_20200707.nc 2020-07-07 09:32:24,390 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_wnd_20200707.nc 2020-07-07 09:32:24,391 INFO [make_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing file for 2020-07-07 created 2020-07-07 09:32:24,396 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker 2020-07-07 09:32:24,413 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:32:25,675 INFO [upload_fvcom_atmos_forcing] running in process 21924 2020-07-07 09:32:25,676 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:32:25,677 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5569 2020-07-07 09:32:25,678 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:32:25,679 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-07 r12 nowcast run to arbutus.cloud-nowcast 2020-07-07 09:32:26,907 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_wnd_20200707.nc 2020-07-07 09:32:26,909 INFO [make_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing file for 2020-07-07 created 2020-07-07 09:32:26,913 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker 2020-07-07 09:32:26,924 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 09:32:28,172 INFO [upload_fvcom_atmos_forcing] running in process 21930 2020-07-07 09:32:28,173 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:32:28,174 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5570 2020-07-07 09:32:28,175 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:32:28,176 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-07 x2 nowcast run to arbutus.cloud-nowcast 2020-07-07 09:32:31,783 INFO [upload_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing files for 2020-07-07 uploaded to arbutus.cloud-nowcast 2020-07-07 09:32:31,787 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker 2020-07-07 09:32:31,799 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 09:32:33,380 INFO [upload_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing files for 2020-07-07 uploaded to arbutus.cloud-nowcast 2020-07-07 09:32:33,388 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker 2020-07-07 09:32:33,407 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 09:32:34,291 INFO [run_fvcom] running in process 823 2020-07-07 09:32:34,292 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:32:34,293 INFO [run_fvcom] publishing log messages to tcp://*:5584 2020-07-07 09:32:34,294 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:32:34,346 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200707-16h32m34.335281s/VHFR_FVCOM.sh on arbutus.cloud-nowcast 2020-07-07 09:32:34,370 INFO [run_fvcom] r12 nowcast FVCOM VH-FR run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 09:32:34,375 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker 2020-07-07 09:32:34,389 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-07 09:32:35,863 INFO [run_fvcom] running in process 945 2020-07-07 09:32:35,864 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:32:35,865 INFO [run_fvcom] publishing log messages to tcp://*:5584 2020-07-07 09:32:35,866 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:32:35,924 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200707-16h32m35.914599s/VHFR_FVCOM.sh on arbutus.cloud-nowcast 2020-07-07 09:32:35,954 INFO [run_fvcom] x2 nowcast FVCOM VH-FR run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 09:32:35,971 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker 2020-07-07 09:32:35,994 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-07 09:32:36,585 INFO [watch_fvcom] running in process 1015 2020-07-07 09:32:36,586 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:32:36,587 INFO [watch_fvcom] publishing log messages to tcp://*:5586 2020-07-07 09:32:36,588 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:32:36,631 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch... 2020-07-07 09:32:38,245 INFO [watch_fvcom] running in process 1124 2020-07-07 09:32:38,246 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 09:32:38,248 INFO [watch_fvcom] publishing log messages to tcp://*:5587 2020-07-07 09:32:38,249 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-07 09:32:38,289 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch... 2020-07-07 09:32:57,265 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-07 for nowcast 2020-07-07 09:32:57,270 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2020-07-07 09:33:46,262 WARNING [make_plots] VENUS Delta DDL node CTD comparison figure failed: No observations available 2020-07-07 09:33:46,870 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2020-07-07 09:33:47,445 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2020-07-07 09:33:48,045 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2020-07-07 09:33:48,711 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2020-07-07 09:33:49,319 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2020-07-07 09:33:49,951 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2020-07-07 09:33:50,609 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2020-07-07 09:33:51,211 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2020-07-07 09:33:51,847 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2020-07-07 09:33:51,850 INFO [make_plots] finished rendering nemo nowcast comparison 06jul20 figures 2020-07-07 09:33:51,958 INFO [make_plots] nemo comparison plots for 2020-07-06 nowcast completed 2020-07-07 09:33:51,963 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 09:34:21,879 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7506386 = 2020-07-08 04:16:40 UTC, 11.9% complete 2020-07-07 09:34:31,436 INFO [make_plots] finished rendering nemo nowcast research 07jul20 figures 2020-07-07 09:34:31,439 INFO [make_plots] nemo research plots for 2020-07-07 nowcast completed 2020-07-07 09:34:31,460 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 09:37:36,734 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40181446 = 2020-07-07 00:06:18 UTC estimated time to finish: 0000:16:31:34 2020-07-07 09:37:38,391 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16074012 = 2020-07-07 00:30:12 UTC estimated time to finish: 0000:03:46:44 2020-07-07 09:39:21,981 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7506849 = 2020-07-08 09:25:20 UTC, 26.2% complete 2020-07-07 09:42:36,836 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40182593 = 2020-07-07 00:13:57 UTC estimated time to finish: 0000:16:10:28 2020-07-07 09:42:38,494 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16075716 = 2020-07-07 00:58:36 UTC estimated time to finish: 0000:03:45:17 2020-07-07 09:44:22,082 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7507276 = 2020-07-08 14:10:00 UTC, 39.4% complete 2020-07-07 09:47:36,939 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40183699 = 2020-07-07 00:21:19 UTC estimated time to finish: 0000:16:17:28 2020-07-07 09:47:38,597 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16077570 = 2020-07-07 01:29:30 UTC estimated time to finish: 0000:03:41:27 2020-07-07 09:49:22,183 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7507701 = 2020-07-08 18:53:20 UTC, 52.5% complete 2020-07-07 09:52:37,043 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40184682 = 2020-07-07 00:27:52 UTC estimated time to finish: 0000:16:18:18 2020-07-07 09:52:38,693 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16079410 = 2020-07-07 02:00:10 UTC estimated time to finish: 0000:03:36:59 2020-07-07 09:54:22,284 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7508144 = 2020-07-08 23:48:40 UTC, 66.2% complete 2020-07-07 09:57:37,147 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40185830 = 2020-07-07 00:35:32 UTC estimated time to finish: 0000:16:03:42 2020-07-07 09:57:38,772 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16081305 = 2020-07-07 02:31:45 UTC estimated time to finish: 0000:03:31:35 2020-07-07 09:59:22,385 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7508553 = 2020-07-09 04:21:20 UTC, 78.8% complete 2020-07-07 10:02:37,250 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40186936 = 2020-07-07 00:42:54 UTC estimated time to finish: 0000:15:58:36 2020-07-07 10:02:38,853 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16083000 = 2020-07-07 03:00:00 UTC estimated time to finish: 0000:03:26:36 2020-07-07 10:04:22,486 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7509007 = 2020-07-09 09:24:00 UTC, 92.8% complete 2020-07-07 10:07:37,353 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40188082 = 2020-07-07 00:50:32 UTC estimated time to finish: 0000:15:50:17 2020-07-07 10:07:38,933 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16084849 = 2020-07-07 03:30:49 UTC estimated time to finish: 0000:03:21:41 2020-07-07 10:09:22,591 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2020-07-07 10:09:22,595 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-07 10:09:22,875 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=['--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:09:22,878 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:09:24,111 INFO [download_results] running in process 24339 2020-07-07 10:09:24,113 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:24,114 INFO [download_results] publishing log messages to tcp://*:5567 2020-07-07 10:09:24,115 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:24,115 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/07jul20 to /results/SalishSea/forecast.201905 2020-07-07 10:09:24,328 INFO [make_turbidity_file] running in process 24338 2020-07-07 10:09:24,329 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:24,330 INFO [make_turbidity_file] publishing log messages to tcp://*:5568 2020-07-07 10:09:24,331 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:24,332 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2020-07-07 2020-07-07 10:09:24,682 INFO [make_turbidity_file] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-07 10:09:24,682 INFO [make_turbidity_file] NumExpr defaulting to 8 threads. 2020-07-07 10:09:26,200 INFO [make_turbidity_file] 2020-07-07 Fraser River turbidity file creation complete 2020-07-07 10:09:26,202 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2020-07-07 10:09:26,232 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity'], host='localhost') 2020-07-07 10:09:26,236 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity'], host='localhost') 2020-07-07 10:09:26,239 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'turbidity'], host='localhost') 2020-07-07 10:09:26,242 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity'], host='localhost') 2020-07-07 10:09:27,164 INFO [upload_forcing] running in process 24362 2020-07-07 10:09:27,165 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:27,166 INFO [upload_forcing] publishing log messages to tcp://*:5568 2020-07-07 10:09:27,167 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:27,234 INFO [upload_forcing] running in process 24360 2020-07-07 10:09:27,235 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:27,236 INFO [upload_forcing] publishing log messages to tcp://*:5569 2020-07-07 10:09:27,238 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:27,238 INFO [upload_forcing] running in process 24361 2020-07-07 10:09:27,239 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:27,240 INFO [upload_forcing] publishing log messages to tcp://*:5570 2020-07-07 10:09:27,241 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:27,296 INFO [upload_forcing] running in process 24363 2020-07-07 10:09:27,297 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:27,298 INFO [upload_forcing] publishing log messages to tcp://*:5571 2020-07-07 10:09:27,298 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:27,710 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 144, in _upload_ssh_files ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 143, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 759, in put return self.putfo(fl, remotepath, file_size, callback, confirm) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 714, in putfo with self.file(remotepath, "wb") as fr: File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 372, in open t, msg = self._request(CMD_OPEN, filename, imode, attrblock) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 813, in _request return self._read_response(num) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 865, in _read_response self._convert_status(msg) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 894, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 102, in upload_forcing _upload_ssh_files(sftp_client, run_type, run_date, config, host_name, host_config) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 151, in _upload_ssh_files fcst.symlink_to(localpath) File "/SalishSeaCast/nowcast-env/lib/python3.8/pathlib.py", line 1374, in symlink_to self._accessor.symlink(target, self, target_is_directory) File "/SalishSeaCast/nowcast-env/lib/python3.8/pathlib.py", line 442, in symlink return os.symlink(a, b) FileExistsError: [Errno 17] File exists: '/results/forcing/sshNeahBay/obs/ssh_y2020m07d06.nc' -> '/results/forcing/sshNeahBay/fcst/ssh_y2020m07d06.nc' 2020-07-07 10:09:27,885 INFO [upload_forcing] turbidity 2020-07-07 forcing files upload to orcinus-nowcast-agrif completed 2020-07-07 10:09:27,889 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 10:09:27,906 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:09:28,463 INFO [upload_forcing] turbidity 2020-07-07 forcing files upload to arbutus.cloud-nowcast completed 2020-07-07 10:09:28,467 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 10:09:28,494 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:09:28,897 INFO [make_forcing_links] running in process 24392 2020-07-07 10:09:28,898 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:28,899 INFO [make_forcing_links] publishing log messages to tcp://*:5569 2020-07-07 10:09:28,900 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:29,479 INFO [make_forcing_links] running in process 24395 2020-07-07 10:09:29,480 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:29,481 INFO [make_forcing_links] publishing log messages to tcp://*:5570 2020-07-07 10:09:29,482 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:29,507 INFO [make_forcing_links] nowcast-agrif 2020-07-07 forcing file links on orcinus-nowcast-agrif created 2020-07-07 10:09:29,511 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-07 10:09:29,529 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:09:29,929 INFO [upload_forcing] turbidity 2020-07-07 forcing files upload to graham-hindcast completed 2020-07-07 10:09:29,932 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-07 10:09:30,358 INFO [run_NEMO_agrif] running in process 24405 2020-07-07 10:09:30,360 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:30,361 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5568 2020-07-07 10:09:30,361 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:30,704 INFO [make_forcing_links] nowcast-green 2020-07-07 forcing file links on arbutus.cloud-nowcast created 2020-07-07 10:09:30,708 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-07 10:09:30,727 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 10:09:33,264 INFO [run_NEMO] running in process 3717 2020-07-07 10:09:33,265 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:33,266 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-07 10:09:33,300 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:34,686 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/07jul20nowcast-green_2020-07-07T100933.335105-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-07 10:09:34,735 INFO [run_NEMO] nowcast-green NEMO run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 10:09:34,740 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-07 10:09:34,758 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2020-07-07 10:09:36,746 INFO [watch_NEMO] running in process 3983 2020-07-07 10:09:36,748 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:09:36,749 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-07 10:09:36,750 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:09:36,851 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-07 10:10:03,999 INFO [run_NEMO_agrif] job id for 07jul20nowcast-agrif: 10848799.orca2.ibb 2020-07-07 10:10:04,000 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-07-07 queued on orcinus-nowcast-agrif 2020-07-07 10:10:04,006 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2020-07-07 10:10:04,040 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10848799.orca2.ibb'], host='localhost') 2020-07-07 10:10:05,005 INFO [watch_NEMO_agrif] running in process 24420 2020-07-07 10:10:05,006 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:10:05,007 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5568 2020-07-07 10:10:05,007 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:10:05,531 INFO [watch_NEMO_agrif] watching 07jul20nowcast-agrif job 10848799 on orcinus-nowcast-agrif 2020-07-07 10:10:05,705 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif job 10848799 is queued on orcinus-nowcast-agrif 2020-07-07 10:11:05,969 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif job 10848799 is queued on orcinus-nowcast-agrif 2020-07-07 10:12:06,905 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2020-07-07 10:12:09,598 INFO [download_results] forecast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 10:12:09,607 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-07 10:12:09,654 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:12:12,162 INFO [make_CHS_currents_file] running in process 24431 2020-07-07 10:12:12,164 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:12:12,165 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5567 2020-07-07 10:12:12,165 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:12:37,461 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40189270 = 2020-07-07 00:58:28 UTC estimated time to finish: 0000:15:26:47 2020-07-07 10:12:39,037 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16086702 = 2020-07-07 04:01:42 UTC estimated time to finish: 0000:03:16:47 2020-07-07 10:14:36,949 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7503971 = 2020-07-07 01:26:40 UTC, 6.0% complete 2020-07-07 10:15:09,166 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-07 for forecast 2020-07-07 10:15:09,169 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2020-07-07 10:15:09,185 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:15:10,165 INFO [update_forecast_datasets] running in process 24462 2020-07-07 10:15:10,166 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:15:10,167 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567 2020-07-07 10:15:10,168 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:15:10,168 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/07jul20 run 2020-07-07 10:15:10,444 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/07jul20 run: /results/SalishSea/rolling-forecasts/nemo 2020-07-07 10:15:10,445 INFO [update_forecast_datasets] nemo 2020-07-07 forecast rolling forecast datasets updated 2020-07-07 10:15:10,447 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-07 10:15:10,471 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2020-07-07 10:15:10,475 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:15:10,478 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:15:11,294 INFO [ping_erddap] running in process 24466 2020-07-07 10:15:11,295 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:15:11,295 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-07 10:15:11,296 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:15:11,348 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2020-07-07 10:15:11,353 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 10:15:11,696 INFO [make_surface_current_tiles] running in process 24468 2020-07-07 10:15:11,696 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:15:11,697 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567 2020-07-07 10:15:11,698 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:15:11,945 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-07 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/07jul20 2020-07-07 10:15:12,242 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/07jul20 2020-07-07 10:15:12,503 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,525 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,546 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,565 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,585 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,605 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,628 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,648 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,670 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,692 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,712 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,734 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,755 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,776 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,798 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,818 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,839 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,864 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,896 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,918 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-07 10:15:12,919 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/07jul20 2020-07-07 10:15:12,920 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-07 forecast into /results/nowcast-sys/figures/surface_currents/forecast/07jul20 2020-07-07 10:15:12,921 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-07 forecast completed 2020-07-07 10:15:12,927 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2020-07-07 10:15:14,395 INFO [make_plots] running in process 24467 2020-07-07 10:15:14,397 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:15:14,402 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-07 10:15:14,403 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:15:14,807 INFO [make_plots] preparing render list for 2020-07-07 NEMO forecast publish figures 2020-07-07 10:15:14,888 INFO [make_plots] added Website_thumbnail figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,889 INFO [make_plots] added Threshold_website figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,890 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,890 INFO [make_plots] added Boundary Bay figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,891 INFO [make_plots] added Campbell River figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,892 INFO [make_plots] added Cherry Point figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,893 INFO [make_plots] added Friday Harbor figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,893 INFO [make_plots] added Halfmoon Bay figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,894 INFO [make_plots] added Nanaimo figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,895 INFO [make_plots] added Neah Bay figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,896 INFO [make_plots] added New Westminster figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,897 INFO [make_plots] added Patricia Bay figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,897 INFO [make_plots] added Point Atkinson figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,898 INFO [make_plots] added Port Renfrew figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,898 INFO [make_plots] added Sand Heads figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,899 INFO [make_plots] added Sandy Cove figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,900 INFO [make_plots] added Squamish figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,900 INFO [make_plots] added Victoria figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,901 INFO [make_plots] added Woodwards Landing figure to 2020-07-07 NEMO forecast publish render list 2020-07-07 10:15:14,902 INFO [make_plots] starting to render nemo forecast publish 07jul20 figures 2020-07-07 10:15:23,398 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-07 10:15:23,399 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-07 10:17:07,480 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4633434 = 2020-07-07 02:35:20 UTC, 10.8% complete 2020-07-07 10:17:37,569 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40190335 = 2020-07-07 01:05:34 UTC estimated time to finish: 0000:15:28:16 2020-07-07 10:17:39,104 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16088400 = 2020-07-07 04:30:00 UTC estimated time to finish: 0000:03:14:12 2020-07-07 10:19:37,050 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7504118 = 2020-07-07 03:04:40 UTC, 12.8% complete 2020-07-07 10:20:49,310 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:22:07,911 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4633690 = 2020-07-07 05:26:00 UTC, 22.6% complete 2020-07-07 10:22:37,679 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40191441 = 2020-07-07 01:12:56 UTC estimated time to finish: 0000:15:24:53 2020-07-07 10:22:39,179 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16090137 = 2020-07-07 04:58:57 UTC estimated time to finish: 0000:03:08:55 2020-07-07 10:23:05,315 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:23:24,764 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:23:44,333 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:24:04,047 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:24:37,151 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7504266 = 2020-07-07 04:43:20 UTC, 19.7% complete 2020-07-07 10:26:25,201 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:26:41,960 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:27:08,346 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4633949 = 2020-07-07 08:18:40 UTC, 34.6% complete 2020-07-07 10:27:37,793 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40192547 = 2020-07-07 01:20:18 UTC estimated time to finish: 0000:15:22:51 2020-07-07 10:27:39,260 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16092018 = 2020-07-07 05:30:18 UTC estimated time to finish: 0000:03:03:27 2020-07-07 10:28:56,383 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:29:37,252 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7504414 = 2020-07-07 06:22:00 UTC, 26.5% complete 2020-07-07 10:31:14,514 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:31:33,988 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:32:09,004 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4634205 = 2020-07-07 11:09:20 UTC, 46.5% complete 2020-07-07 10:32:37,899 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40193640 = 2020-07-07 01:27:36 UTC estimated time to finish: 0000:15:20:09 2020-07-07 10:32:39,342 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16093871 = 2020-07-07 06:01:11 UTC estimated time to finish: 0000:02:58:06 2020-07-07 10:33:48,163 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:34:37,354 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7504562 = 2020-07-07 08:00:40 UTC, 33.4% complete 2020-07-07 10:36:06,731 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:37:09,489 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4634460 = 2020-07-07 13:59:20 UTC, 58.3% complete 2020-07-07 10:37:38,011 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40194637 = 2020-07-07 01:34:14 UTC estimated time to finish: 0000:15:17:18 2020-07-07 10:37:39,418 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16095767 = 2020-07-07 06:32:47 UTC estimated time to finish: 0000:02:52:53 2020-07-07 10:38:24,831 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:38:44,783 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:39:37,455 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7504710 = 2020-07-07 09:39:20 UTC, 40.3% complete 2020-07-07 10:41:01,180 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-02 00:00:00 to 2020-07-09 11:55:00 2020-07-07 10:42:10,018 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4634719 = 2020-07-07 16:52:00 UTC, 70.3% complete 2020-07-07 10:42:38,121 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40195783 = 2020-07-07 01:41:53 UTC estimated time to finish: 0000:15:12:36 2020-07-07 10:42:39,504 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16097457 = 2020-07-07 07:00:57 UTC estimated time to finish: 0000:02:48:09 2020-07-07 10:43:10,137 INFO [make_plots] finished rendering nemo forecast publish 07jul20 figures 2020-07-07 10:43:10,139 INFO [make_plots] nemo publish plots for 2020-07-07 forecast completed 2020-07-07 10:43:10,150 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 10:43:10,179 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 10:43:12,889 INFO [make_feeds] running in process 24740 2020-07-07 10:43:12,890 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 10:43:12,891 INFO [make_feeds] publishing log messages to tcp://*:5567 2020-07-07 10:43:12,892 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2020-07-07 10:43:21,774 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-07 10:43:21,775 INFO [make_feeds] NumExpr defaulting to 8 threads. 2020-07-07 10:43:27,989 INFO [make_feeds] ATOM feeds for 2020-07-07 forecast run completed 2020-07-07 10:43:27,993 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2020-07-07 10:44:37,557 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7504858 = 2020-07-07 11:18:00 UTC, 47.1% complete 2020-07-07 10:47:10,595 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4634974 = 2020-07-07 19:42:00 UTC, 82.1% complete 2020-07-07 10:47:38,231 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40196849 = 2020-07-07 01:48:59 UTC estimated time to finish: 0000:15:09:59 2020-07-07 10:47:39,616 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16099230 = 2020-07-07 07:30:30 UTC estimated time to finish: 0000:02:43:37 2020-07-07 10:49:37,656 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505005 = 2020-07-07 12:56:00 UTC, 53.9% complete 2020-07-07 10:52:11,048 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4635230 = 2020-07-07 22:32:40 UTC, 94.0% complete 2020-07-07 10:52:38,347 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40197913 = 2020-07-07 01:56:05 UTC estimated time to finish: 0000:15:09:07 2020-07-07 10:52:39,658 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16100851 = 2020-07-07 07:57:31 UTC estimated time to finish: 0000:02:40:30 2020-07-07 10:54:37,757 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505153 = 2020-07-07 14:34:40 UTC, 60.8% complete 2020-07-07 10:57:11,512 INFO [watch_NEMO_agrif] 07jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4635360 = 2020-07-07 23:59:20 UTC, 100.0% complete 2020-07-07 10:57:38,455 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40198897 = 2020-07-07 02:02:38 UTC estimated time to finish: 0000:15:05:38 2020-07-07 10:57:39,741 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16102460 = 2020-07-07 08:24:20 UTC estimated time to finish: 0000:02:37:12 2020-07-07 10:59:37,765 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505299 = 2020-07-07 16:12:00 UTC, 67.5% complete 2020-07-07 11:02:11,691 INFO [watch_NEMO_agrif] NEMO AGRIF run 10848799.orca2.ibb on orcinus-nowcast-agrif completed 2020-07-07 11:02:11,695 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2020-07-07 11:02:11,982 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 11:02:13,053 INFO [download_results] running in process 24904 2020-07-07 11:02:13,054 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:02:13,055 INFO [download_results] publishing log messages to tcp://*:5567 2020-07-07 11:02:13,055 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:02:13,057 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/07jul20 to /results/SalishSea/nowcast-agrif.201702 2020-07-07 11:02:38,574 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40199921 = 2020-07-07 02:09:28 UTC estimated time to finish: 0000:15:04:17 2020-07-07 11:02:39,851 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16104082 = 2020-07-07 08:51:22 UTC estimated time to finish: 0000:02:33:41 2020-07-07 11:02:51,793 INFO [download_results] nowcast-agrif 2020-07-07 results files from orcinus-nowcast-agrif downloaded 2020-07-07 11:02:51,803 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-07 11:02:51,846 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 11:02:55,604 INFO [make_plots] running in process 24910 2020-07-07 11:02:55,605 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:02:55,606 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-07 11:02:55,613 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:02:55,983 INFO [make_plots] preparing render list for 2020-07-07 NEMO nowcast-agrif research figures 2020-07-07 11:02:55,984 INFO [make_plots] added baynes_sound_surface figure to 2020-07-07 NEMO nowcast-agrif research render list 2020-07-07 11:02:55,985 INFO [make_plots] starting to render nemo nowcast-agrif research 07jul20 figures 2020-07-07 11:03:05,105 INFO [make_plots] finished rendering nemo nowcast-agrif research 07jul20 figures 2020-07-07 11:03:05,106 INFO [make_plots] nemo research plots for 2020-07-07 nowcast-agrif completed 2020-07-07 11:03:05,110 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 11:04:37,864 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505448 = 2020-07-07 17:51:20 UTC, 74.4% complete 2020-07-07 11:07:38,682 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40200986 = 2020-07-07 02:16:34 UTC estimated time to finish: 0000:15:02:13 2020-07-07 11:07:39,908 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16105540 = 2020-07-07 09:15:40 UTC estimated time to finish: 0000:02:30:23 2020-07-07 11:09:37,965 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505594 = 2020-07-07 19:28:40 UTC, 81.2% complete 2020-07-07 11:12:38,795 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40202051 = 2020-07-07 02:23:40 UTC estimated time to finish: 0000:14:59:38 2020-07-07 11:12:39,992 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16107150 = 2020-07-07 09:42:30 UTC estimated time to finish: 0000:02:26:34 2020-07-07 11:14:38,067 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505743 = 2020-07-07 21:08:00 UTC, 88.1% complete 2020-07-07 11:17:38,909 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40203075 = 2020-07-07 02:30:30 UTC estimated time to finish: 0000:14:52:29 2020-07-07 11:17:40,108 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16108811 = 2020-07-07 10:10:11 UTC estimated time to finish: 0000:02:22:26 2020-07-07 11:19:38,168 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7505893 = 2020-07-07 22:48:00 UTC, 95.0% complete 2020-07-07 11:22:39,022 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40204304 = 2020-07-07 02:38:41 UTC estimated time to finish: 0000:14:45:38 2020-07-07 11:22:40,148 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16110666 = 2020-07-07 10:41:06 UTC estimated time to finish: 0000:02:16:54 2020-07-07 11:24:38,269 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7506000 = 2020-07-07 23:59:20 UTC, 100.0% complete 2020-07-07 11:27:39,131 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40205491 = 2020-07-07 02:46:36 UTC estimated time to finish: 0000:14:33:17 2020-07-07 11:27:40,221 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16112316 = 2020-07-07 11:08:36 UTC estimated time to finish: 0000:02:12:01 2020-07-07 11:29:38,378 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2020-07-07 11:29:38,383 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-07 11:29:38,690 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-07 11:29:38,696 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-07 11:29:38,700 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['salish-nowcast', 'nowcast+', '--shared-storage'], host='localhost') 2020-07-07 11:29:38,704 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 11:29:39,888 INFO [download_results] running in process 25367 2020-07-07 11:29:39,889 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:29:39,890 INFO [download_results] publishing log messages to tcp://*:5564 2020-07-07 11:29:39,891 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:29:39,893 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/07jul20 to /results2/SalishSea/nowcast-green.201905 2020-07-07 11:29:39,899 INFO [make_forcing_links] running in process 25366 2020-07-07 11:29:39,900 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:29:39,901 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2020-07-07 11:29:39,903 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:29:40,958 INFO [make_forcing_links] nowcast+ 2020-07-07 forcing file links on salish-nowcast created 2020-07-07 11:29:40,962 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-07 11:29:41,011 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['salish-nowcast', 'nowcast-dev', '--run-date', '2020-07-07'], host='salish-nowcast') 2020-07-07 11:29:42,693 INFO [make_ww3_wind_file] running in process 4403 2020-07-07 11:29:43,236 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:29:43,612 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2020-07-07 11:29:44,051 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:29:44,516 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-07 forecast run 2020-07-07 11:29:45,836 INFO [make_ww3_current_file] running in process 4402 2020-07-07 11:29:46,306 INFO [run_NEMO] running in process 23449 2020-07-07 11:29:46,887 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:29:47,544 INFO [run_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:29:48,300 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2020-07-07 11:29:48,677 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-07 11:29:48,692 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:29:48,693 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:29:48,694 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-07 forecast run 2020-07-07 11:29:48,721 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-07 forecast run 2020-07-07 11:29:48,722 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2020-07-07 11:29:48,728 WARNING [run_NEMO] There are uncommitted changes in /SalishSeaCast/SalishSeaNowcast 2020-07-07 11:29:49,902 INFO [run_NEMO] nowcast-dev: launching /SalishSeaCast/runs/07jul20nowcast-dev_2020-07-07T112943.280986-0700/SalishSeaNEMO.sh on salish-nowcast 2020-07-07 11:29:50,935 INFO [run_NEMO] nowcast-dev NEMO run for 2020-07-07 on salish-nowcast started 2020-07-07 11:29:52,295 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-07 11:29:52,716 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['salish-nowcast', 'nowcast-dev'], host='salish-nowcast') 2020-07-07 11:29:57,824 INFO [watch_NEMO] running in process 23693 2020-07-07 11:29:58,361 INFO [watch_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:29:58,824 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-07 11:29:59,286 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:29:59,305 INFO [watch_NEMO] nowcast-dev on salish-nowcast: time.step not found; continuing to watch... 2020-07-07 11:30:25,559 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-07 forecast run 2020-07-07 11:30:25,564 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2020-07-07 11:30:25,616 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 11:30:27,614 INFO [run_ww3] running in process 4621 2020-07-07 11:30:27,615 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:30:27,616 INFO [run_ww3] publishing log messages to tcp://*:5572 2020-07-07 11:30:27,617 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:30:27,619 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-07T113027.611909-0700 2020-07-07 11:30:27,621 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-07T113027.611909-0700/SoGWW3.sh on arbutus.cloud-nowcast 2020-07-07 11:30:27,637 INFO [run_ww3] nowcast WaveWatch3 run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 11:30:27,643 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2020-07-07 11:30:27,692 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-07 11:30:29,737 INFO [watch_ww3] running in process 4727 2020-07-07 11:30:29,738 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:30:29,740 INFO [watch_ww3] publishing log messages to tcp://*:5573 2020-07-07 11:30:29,741 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:30:29,775 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2020-07-07 11:32:39,240 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40206980 = 2020-07-07 02:56:32 UTC estimated time to finish: 0000:14:14:44 2020-07-07 11:32:40,303 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16114130 = 2020-07-07 11:38:50 UTC estimated time to finish: 0000:02:06:46 2020-07-07 11:34:57,988 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633224 = 2020-07-07 00:15:20 UTC, 1.1% complete 2020-07-07 11:35:29,876 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete 2020-07-07 11:37:39,350 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40208482 = 2020-07-07 03:06:32 UTC estimated time to finish: 0000:13:56:48 2020-07-07 11:37:40,390 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16115983 = 2020-07-07 12:09:43 UTC estimated time to finish: 0000:02:01:21 2020-07-07 11:39:58,089 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633272 = 2020-07-07 00:47:20 UTC, 3.3% complete 2020-07-07 11:40:29,976 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 207, 47.9% complete 2020-07-07 11:42:39,418 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40210025 = 2020-07-07 03:16:50 UTC estimated time to finish: 0000:13:39:01 2020-07-07 11:42:40,467 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16117837 = 2020-07-07 12:40:37 UTC estimated time to finish: 0000:01:55:56 2020-07-07 11:44:58,190 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633323 = 2020-07-07 01:21:20 UTC, 5.7% complete 2020-07-07 11:45:30,078 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 315, 72.9% complete 2020-07-07 11:46:20,180 INFO [download_results] nowcast-green 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 11:46:20,191 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-07 11:46:20,246 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 11:46:20,251 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2020-07-07 11:46:22,008 INFO [ping_erddap] running in process 25657 2020-07-07 11:46:22,009 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:46:22,011 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-07 11:46:22,012 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:46:22,018 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2020-07-07 11:46:22,021 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 11:46:28,912 INFO [make_plots] running in process 25656 2020-07-07 11:46:28,915 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:46:28,916 INFO [make_plots] publishing log messages to tcp://*:5564 2020-07-07 11:46:28,918 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:46:30,194 INFO [make_plots] preparing render list for 2020-07-07 NEMO nowcast-green research figures 2020-07-07 11:46:33,857 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:35,286 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:36,471 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:37,637 INFO [make_plots] added silicon_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:39,042 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:40,459 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:41,826 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:43,232 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:44,581 INFO [make_plots] added ciliates_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:45,940 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:47,333 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:48,713 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:50,118 INFO [make_plots] added Fraser_tracer_thalweg_and_surface figures to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:50,212 INFO [make_plots] added temperature_salinity_timeseries figure to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:50,213 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:50,215 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:50,216 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2020-07-07 NEMO nowcast-green research render list 2020-07-07 11:46:50,216 INFO [make_plots] starting to render nemo nowcast-green research 07jul20 figures 2020-07-07 11:47:39,529 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40211363 = 2020-07-07 03:25:45 UTC estimated time to finish: 0000:13:25:15 2020-07-07 11:47:40,543 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16119528 = 2020-07-07 13:08:48 UTC estimated time to finish: 0000:01:51:01 2020-07-07 11:49:58,291 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633377 = 2020-07-07 01:57:20 UTC, 8.2% complete 2020-07-07 11:50:30,178 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 417, 96.5% complete 2020-07-07 11:52:39,641 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40212866 = 2020-07-07 03:35:46 UTC estimated time to finish: 0000:13:10:56 2020-07-07 11:52:40,624 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16121341 = 2020-07-07 13:39:01 UTC estimated time to finish: 0000:01:45:47 2020-07-07 11:54:58,388 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633431 = 2020-07-07 02:33:20 UTC, 10.7% complete 2020-07-07 11:55:30,280 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2020-07-07 11:55:30,285 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2020-07-07 11:55:30,632 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 11:55:30,638 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 11:55:32,828 INFO [run_ww3] running in process 4902 2020-07-07 11:55:32,829 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:55:32,831 INFO [run_ww3] publishing log messages to tcp://*:5572 2020-07-07 11:55:32,832 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:55:32,836 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-07T115532.826316-0700 2020-07-07 11:55:32,838 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-07T115532.826316-0700/SoGWW3.sh on arbutus.cloud-nowcast 2020-07-07 11:55:32,851 INFO [run_ww3] forecast WaveWatch3 run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 11:55:32,858 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2020-07-07 11:55:32,911 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-07 11:55:33,369 INFO [download_wwatch3_results] running in process 25844 2020-07-07 11:55:33,372 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:55:33,378 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567 2020-07-07 11:55:33,379 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:55:34,897 INFO [watch_ww3] running in process 5023 2020-07-07 11:55:35,093 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 11:55:35,094 INFO [watch_ww3] publishing log messages to tcp://*:5573 2020-07-07 11:55:35,095 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-07 11:55:35,097 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2020-07-07 11:56:55,891 INFO [download_wwatch3_results] nowcast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 11:56:55,895 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2020-07-07 11:57:39,753 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40214327 = 2020-07-07 03:45:30 UTC estimated time to finish: 0000:12:58:00 2020-07-07 11:57:40,704 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16123195 = 2020-07-07 14:09:55 UTC estimated time to finish: 0000:01:40:26 2020-07-07 11:59:58,488 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633485 = 2020-07-07 03:09:20 UTC, 13.2% complete 2020-07-07 12:00:35,035 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete 2020-07-07 12:02:39,873 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40215829 = 2020-07-07 03:55:31 UTC estimated time to finish: 0000:12:45:04 2020-07-07 12:02:40,784 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16125007 = 2020-07-07 14:40:07 UTC estimated time to finish: 0000:01:35:15 2020-07-07 12:04:58,589 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633539 = 2020-07-07 03:45:20 UTC, 15.7% complete 2020-07-07 12:05:35,137 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 207, 31.9% complete 2020-07-07 12:07:39,988 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40217330 = 2020-07-07 04:05:32 UTC estimated time to finish: 0000:12:32:37 2020-07-07 12:07:40,865 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16126860 = 2020-07-07 15:11:00 UTC estimated time to finish: 0000:01:29:58 2020-07-07 12:09:58,690 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633593 = 2020-07-07 04:21:20 UTC, 18.2% complete 2020-07-07 12:10:35,238 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 315, 48.6% complete 2020-07-07 12:12:40,101 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40218710 = 2020-07-07 04:14:44 UTC estimated time to finish: 0000:12:21:19 2020-07-07 12:12:40,946 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16128510 = 2020-07-07 15:38:30 UTC estimated time to finish: 0000:01:25:16 2020-07-07 12:14:58,783 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633647 = 2020-07-07 04:57:20 UTC, 20.7% complete 2020-07-07 12:15:35,339 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 423, 65.3% complete 2020-07-07 12:17:40,214 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40220280 = 2020-07-07 04:25:12 UTC estimated time to finish: 0000:12:08:52 2020-07-07 12:17:41,025 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16130323 = 2020-07-07 16:08:43 UTC estimated time to finish: 0000:01:20:05 2020-07-07 12:19:58,881 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633700 = 2020-07-07 05:32:40 UTC, 23.1% complete 2020-07-07 12:20:35,439 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 531, 81.9% complete 2020-07-07 12:22:40,328 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40221823 = 2020-07-07 04:35:29 UTC estimated time to finish: 0000:11:57:10 2020-07-07 12:22:41,107 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16131944 = 2020-07-07 16:35:44 UTC estimated time to finish: 0000:01:15:45 2020-07-07 12:24:58,981 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633754 = 2020-07-07 06:08:40 UTC, 25.6% complete 2020-07-07 12:25:35,743 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 639, 98.6% complete 2020-07-07 12:27:40,350 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40223325 = 2020-07-07 04:45:30 UTC estimated time to finish: 0000:11:46:41 2020-07-07 12:27:41,186 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16133512 = 2020-07-07 17:01:52 UTC estimated time to finish: 0000:01:11:32 2020-07-07 12:29:59,006 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633808 = 2020-07-07 06:44:40 UTC, 28.1% complete 2020-07-07 12:30:35,635 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2020-07-07 12:30:35,639 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2020-07-07 12:30:35,973 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 12:30:44,318 INFO [download_wwatch3_results] running in process 26271 2020-07-07 12:30:44,609 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 12:30:44,610 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567 2020-07-07 12:30:44,611 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 12:32:37,508 INFO [download_wwatch3_results] forecast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 12:32:37,513 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2020-07-07 12:32:37,861 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 12:32:40,464 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40224705 = 2020-07-07 04:54:42 UTC estimated time to finish: 0000:11:37:00 2020-07-07 12:32:41,224 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16134930 = 2020-07-07 17:25:30 UTC estimated time to finish: 0000:01:07:43 2020-07-07 12:32:43,352 INFO [update_forecast_datasets] running in process 26315 2020-07-07 12:32:43,471 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 12:32:43,471 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567 2020-07-07 12:32:43,472 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-07 12:32:43,473 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/07jul20 run 2020-07-07 12:32:43,474 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/07jul20 run 2020-07-07 12:32:43,475 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/07jul20 run 2020-07-07 12:32:43,750 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/07jul20 run: /results/SalishSea/rolling-forecasts/wwatch3 2020-07-07 12:32:44,016 INFO [update_forecast_datasets] wwatch3 2020-07-07 forecast rolling forecast datasets updated 2020-07-07 12:32:44,050 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-07 12:32:44,076 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2020-07-07 12:32:48,021 INFO [ping_erddap] running in process 26319 2020-07-07 12:32:48,023 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 12:32:48,053 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-07 12:32:48,102 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 12:32:48,143 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2020-07-07 12:32:48,432 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 12:32:48,434 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 12:33:22,300 INFO [make_plots] running in process 26323 2020-07-07 12:33:22,306 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 12:33:22,307 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-07 12:33:22,308 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 12:33:22,309 INFO [make_plots] preparing render list for 2020-07-07 WaveWatch3 forecast publish figures 2020-07-07 12:33:22,310 INFO [make_plots] added Halibut Bank figure to 2020-07-07 WaveWatch3 forecast publish render list 2020-07-07 12:33:22,311 INFO [make_plots] added Sentry Shoal figure to 2020-07-07 WaveWatch3 forecast publish render list 2020-07-07 12:33:22,312 INFO [make_plots] starting to render wwatch3 forecast publish 07jul20 figures 2020-07-07 12:34:27,835 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-07 12:34:27,835 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-07 12:34:27,848 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2020-07-07 12:34:59,107 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633860 = 2020-07-07 07:19:20 UTC, 30.5% complete 2020-07-07 12:35:33,522 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2020-07-07 12:35:36,658 INFO [make_plots] finished rendering wwatch3 forecast publish 07jul20 figures 2020-07-07 12:35:36,659 INFO [make_plots] wwatch3 publish plots for 2020-07-07 forecast completed 2020-07-07 12:35:36,900 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 12:37:40,579 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40226207 = 2020-07-07 05:04:42 UTC estimated time to finish: 0000:11:27:22 2020-07-07 12:37:41,344 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16136511 = 2020-07-07 17:51:51 UTC estimated time to finish: 0000:01:03:23 2020-07-07 12:39:59,208 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633913 = 2020-07-07 07:54:40 UTC, 33.0% complete 2020-07-07 12:42:40,694 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40227900 = 2020-07-07 05:16:00 UTC estimated time to finish: 0000:11:14:19 2020-07-07 12:42:41,428 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16138201 = 2020-07-07 18:20:01 UTC estimated time to finish: 0000:00:58:37 2020-07-07 12:44:59,309 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4633967 = 2020-07-07 08:30:40 UTC, 35.5% complete 2020-07-07 12:47:40,809 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40229606 = 2020-07-07 05:27:22 UTC estimated time to finish: 0000:11:01:37 2020-07-07 12:47:41,509 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16139934 = 2020-07-07 18:48:54 UTC estimated time to finish: 0000:00:53:41 2020-07-07 12:49:59,323 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634021 = 2020-07-07 09:06:40 UTC, 38.0% complete 2020-07-07 12:52:40,928 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40231136 = 2020-07-07 05:37:34 UTC estimated time to finish: 0000:10:50:45 2020-07-07 12:52:41,589 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16141596 = 2020-07-07 19:16:36 UTC estimated time to finish: 0000:00:48:59 2020-07-07 12:54:59,424 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634074 = 2020-07-07 09:42:00 UTC, 40.4% complete 2020-07-07 12:57:41,043 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40232880 = 2020-07-07 05:49:12 UTC estimated time to finish: 0000:10:38:37 2020-07-07 12:57:41,669 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16143247 = 2020-07-07 19:44:07 UTC estimated time to finish: 0000:00:44:16 2020-07-07 12:59:59,521 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634128 = 2020-07-07 10:18:00 UTC, 42.9% complete 2020-07-07 13:02:41,161 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40234576 = 2020-07-07 06:00:30 UTC estimated time to finish: 0000:10:27:33 2020-07-07 13:02:41,749 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16144827 = 2020-07-07 20:10:27 UTC estimated time to finish: 0000:00:39:45 2020-07-07 13:04:59,622 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634181 = 2020-07-07 10:53:20 UTC, 45.4% complete 2020-07-07 13:07:41,280 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40236270 = 2020-07-07 06:11:48 UTC estimated time to finish: 0000:10:16:44 2020-07-07 13:07:41,830 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16146559 = 2020-07-07 20:39:19 UTC estimated time to finish: 0000:00:34:47 2020-07-07 13:09:59,721 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634235 = 2020-07-07 11:29:20 UTC, 47.9% complete 2020-07-07 13:12:41,399 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40238004 = 2020-07-07 06:23:21 UTC estimated time to finish: 0000:10:05:52 2020-07-07 13:12:41,912 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16148249 = 2020-07-07 21:07:29 UTC estimated time to finish: 0000:00:29:56 2020-07-07 13:14:59,792 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634289 = 2020-07-07 12:05:20 UTC, 50.4% complete 2020-07-07 13:17:41,518 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40239506 = 2020-07-07 06:33:22 UTC estimated time to finish: 0000:09:57:00 2020-07-07 13:17:41,992 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16149790 = 2020-07-07 21:33:10 UTC estimated time to finish: 0000:00:25:29 2020-07-07 13:20:00,092 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634343 = 2020-07-07 12:41:20 UTC, 52.9% complete 2020-07-07 13:22:41,639 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40241200 = 2020-07-07 06:44:40 UTC estimated time to finish: 0000:09:47:07 2020-07-07 13:22:42,074 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16151480 = 2020-07-07 22:01:20 UTC estimated time to finish: 0000:00:20:37 2020-07-07 13:24:59,993 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634397 = 2020-07-07 13:17:20 UTC, 55.4% complete 2020-07-07 13:27:41,759 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40242905 = 2020-07-07 06:56:02 UTC estimated time to finish: 0000:09:37:34 2020-07-07 13:27:42,152 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16153183 = 2020-07-07 22:29:43 UTC estimated time to finish: 0000:00:15:42 2020-07-07 13:30:00,093 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634452 = 2020-07-07 13:54:00 UTC, 57.9% complete 2020-07-07 13:32:41,880 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40244599 = 2020-07-07 07:07:19 UTC estimated time to finish: 0000:09:28:08 2020-07-07 13:32:42,234 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16154874 = 2020-07-07 22:57:54 UTC estimated time to finish: 0000:00:10:48 2020-07-07 13:35:00,131 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634506 = 2020-07-07 14:30:00 UTC, 60.4% complete 2020-07-07 13:36:26,038 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/matplotlib/cbook/__init__.py", line 1349, in _check_1d ndim = x[:, None].ndim File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 630, in __getitem__ return self.isel(indexers=self._item_key_to_dict(key)) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 594, in _item_key_to_dict key = indexing.expanded_indexer(key, self.ndim) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 43, in expanded_indexer raise IndexError("too many indices") IndexError: too many indices During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots checklist = _render_figures( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures fig = _calc_figure(fig_func, args, kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/time_series_plots.py", line 77, in make_figure _plot_timeseries(axl, plot_data.left_var, left_variable, theme) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/time_series_plots.py", line 121, in _plot_timeseries ax.plot(plot_data.time, plot_data, color=theme.COLOURS["time series"][variable]) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/matplotlib/axes/_axes.py", line 1646, in plot lines = [*self._get_lines(*args, data=data, **kwargs)] File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 216, in __call__ yield from self._plot_args(this, kwargs) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 332, in _plot_args y = _check_1d(tup[-1]) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/matplotlib/cbook/__init__.py", line 1361, in _check_1d return np.atleast_1d(x) File "<__array_function__ internals>", line 5, in atleast_1d File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/shape_base.py", line 67, in atleast_1d ary = asanyarray(ary) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/_asarray.py", line 138, in asanyarray return array(a, dtype, copy=False, order=order, subok=True) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/common.py", line 132, in __array__ return np.asarray(self.values, dtype=dtype) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 557, in values return self.variable.values File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/variable.py", line 451, in values return _as_array_or_item(self._data) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/variable.py", line 254, in _as_array_or_item data = np.asarray(data) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 677, in __array__ self._ensure_cached() File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 674, in _ensure_cached self.array = NumpyIndexingAdapter(np.asarray(self.array)) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 653, in __array__ return np.asarray(self.array, dtype=dtype) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 557, in __array__ return np.asarray(array[self.key], dtype=None) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/coding/variables.py", line 72, in __array__ return self.func(self.array) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/coding/variables.py", line 138, in _apply_mask data = np.asarray(data, dtype=dtype) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 557, in __array__ return np.asarray(array[self.key], dtype=None) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 72, in __getitem__ return indexing.explicit_indexing_adapter( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 837, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 85, in _getitem array = getitem(original_array, key) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/common.py", line 54, in robust_getitem return array[key] File "netCDF4/_netCDF4.pyx", line 4408, in netCDF4._netCDF4.Variable.__getitem__ File "netCDF4/_netCDF4.pyx", line 5352, in netCDF4._netCDF4.Variable._get File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: DAP failure 2020-07-07 13:37:42,002 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40246182 = 2020-07-07 07:17:52 UTC estimated time to finish: 0000:09:19:21 2020-07-07 13:37:42,315 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16156414 = 2020-07-07 23:23:34 UTC estimated time to finish: 0000:00:06:20 2020-07-07 13:40:00,229 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634560 = 2020-07-07 15:06:00 UTC, 62.9% complete 2020-07-07 13:42:42,123 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40247876 = 2020-07-07 07:29:10 UTC estimated time to finish: 0000:09:10:22 2020-07-07 13:42:42,395 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 16158105 = 2020-07-07 23:51:45 UTC estimated time to finish: 0000:00:01:26 2020-07-07 13:45:00,328 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634614 = 2020-07-07 15:42:00 UTC, 65.4% complete 2020-07-07 13:47:42,246 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40250240 = 2020-07-07 07:44:56 UTC estimated time to finish: 0000:08:54:01 2020-07-07 13:47:42,455 INFO [watch_fvcom] x2 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed 2020-07-07 13:47:42,459 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker 2020-07-07 13:47:43,154 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 13:47:43,560 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 13:47:50,276 INFO [download_fvcom_results] running in process 26862 2020-07-07 13:47:50,278 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:47:50,279 INFO [download_fvcom_results] publishing log messages to tcp://*:5567 2020-07-07 13:47:50,280 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:47:52,809 INFO [make_fvcom_boundary] running in process 5554 2020-07-07 13:47:52,885 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:47:52,886 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580 2020-07-07 13:47:52,887 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:47:52,887 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 forecast run from 2020-07-07 NEMO run 2020-07-07 13:49:21,648 INFO [download_fvcom_results] VHFR FVCOM x2 nowcast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 13:49:21,654 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker 2020-07-07 13:49:21,693 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-07'], host='localhost') 2020-07-07 13:49:21,697 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'research', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 13:49:51,603 INFO [get_vfpa_hadcp] running in process 26874 2020-07-07 13:49:51,865 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:49:51,866 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5567 2020-07-07 13:49:51,867 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:49:51,868 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-07 2020-07-07 13:49:56,539 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-07 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc 2020-07-07 13:49:56,540 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file 2020-07-07 13:49:56,542 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2020-07-07 13:49:56,580 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2020-07-07 13:50:00,429 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634668 = 2020-07-07 16:18:00 UTC, 67.9% complete 2020-07-07 13:50:05,677 INFO [ping_erddap] running in process 26883 2020-07-07 13:50:05,740 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:50:05,741 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-07 13:50:05,742 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:50:05,743 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2020-07-07 13:50:05,746 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 13:50:05,796 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 13:50:06,621 INFO [make_plots] running in process 26875 2020-07-07 13:50:06,623 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:50:06,624 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-07 13:50:06,626 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:50:06,635 INFO [make_plots] preparing render list for 2020-07-07 VHFR FVCOM nowcast-x2 research figures 2020-07-07 13:50:06,638 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-07 VHFR FVCOM nowcast-x2 research render list 2020-07-07 13:50:06,641 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-07 VHFR FVCOM nowcast-x2 research render list 2020-07-07 13:50:06,643 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-07 VHFR FVCOM nowcast-x2 research render list 2020-07-07 13:50:06,646 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-07 VHFR FVCOM nowcast-x2 research render list 2020-07-07 13:50:06,649 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-07 VHFR FVCOM nowcast-x2 research render list 2020-07-07 13:50:06,652 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-07 VHFR FVCOM nowcast-x2 research render list 2020-07-07 13:50:06,653 INFO [make_plots] starting to render fvcom nowcast-x2 research 07jul20 figures 2020-07-07 13:50:44,983 INFO [make_plots] running in process 26928 2020-07-07 13:50:45,292 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:50:45,293 INFO [make_plots] publishing log messages to tcp://*:5569 2020-07-07 13:50:45,294 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:50:48,216 INFO [make_plots] preparing render list for 2020-07-07 VHFR FVCOM nowcast-x2 publish figures 2020-07-07 13:50:48,347 INFO [make_plots] added Calamity Point figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,348 INFO [make_plots] added Indian Arm Head figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,349 INFO [make_plots] added New Westminster figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,349 INFO [make_plots] added Port Moody figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,350 INFO [make_plots] added Sand Heads figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,351 INFO [make_plots] added Sandy Cove figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,351 INFO [make_plots] added Vancouver Harbour figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,352 INFO [make_plots] added Woodwards Landing figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,352 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 13:50:48,353 INFO [make_plots] starting to render fvcom nowcast-x2 publish 07jul20 figures 2020-07-07 13:50:48,397 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:51:05,493 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:51:06,686 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_forecast_brcl_20200708.nc 2020-07-07 13:51:06,695 INFO [make_fvcom_boundary] FVCOM x2 forecast run boundary condition file for 2020-07-07 created on arbutus.cloud-nowcast 2020-07-07 13:51:06,705 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker 2020-07-07 13:51:06,755 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 13:51:06,789 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 13:51:09,836 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:51:10,841 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:51:10,895 INFO [make_fvcom_rivers_forcing] running in process 5735 2020-07-07 13:51:10,897 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:51:10,898 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582 2020-07-07 13:51:10,899 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:51:10,899 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 forecast run from 2020-07-07 NEMO runoff forcing files 2020-07-07 13:51:11,090 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_forecast_20200708.nc 2020-07-07 13:51:11,092 INFO [make_fvcom_rivers_forcing] FVCOM x2 forecast run rivers forcing file for 2020-07-07 created on arbutus.cloud-nowcast 2020-07-07 13:51:11,098 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker 2020-07-07 13:51:13,186 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:51:38,344 INFO [make_fvcom_atmos_forcing] running in process 26953 2020-07-07 13:51:38,345 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:51:38,346 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5570 2020-07-07 13:51:38,347 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:51:38,381 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 forecast run from 2020-07-07 HRDPS GRIB files 2020-07-07 13:51:40,101 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:51:46,375 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:52:09,325 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:52:13,115 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:52:15,366 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_hfx_20200708.nc 2020-07-07 13:52:25,405 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_precip_20200708.nc 2020-07-07 13:52:36,795 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:52:40,000 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_wnd_20200708.nc 2020-07-07 13:52:40,129 INFO [make_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing file for 2020-07-07 created 2020-07-07 13:52:40,131 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker 2020-07-07 13:52:40,271 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 13:52:40,775 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:52:42,369 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40252847 = 2020-07-07 08:02:18 UTC estimated time to finish: 0000:08:35:46 2020-07-07 13:52:45,220 INFO [upload_fvcom_atmos_forcing] running in process 26965 2020-07-07 13:52:45,222 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:52:45,224 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5570 2020-07-07 13:52:45,226 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:52:45,227 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-07 x2 forecast run to arbutus.cloud-nowcast 2020-07-07 13:52:50,357 INFO [upload_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing files for 2020-07-07 uploaded to arbutus.cloud-nowcast 2020-07-07 13:52:50,358 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker 2020-07-07 13:52:50,391 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-07'], host='arbutus.cloud-nowcast') 2020-07-07 13:52:52,537 INFO [run_fvcom] running in process 6085 2020-07-07 13:52:52,538 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:52:52,539 INFO [run_fvcom] publishing log messages to tcp://*:5584 2020-07-07 13:52:52,540 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:52:52,583 INFO [run_fvcom] forecast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200707-20h52m52.575957s/VHFR_FVCOM.sh on arbutus.cloud-nowcast 2020-07-07 13:52:52,601 INFO [run_fvcom] x2 forecast FVCOM VH-FR run for 2020-07-07 on arbutus.cloud-nowcast started 2020-07-07 13:52:52,607 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker 2020-07-07 13:52:52,662 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-07 13:52:54,534 INFO [watch_fvcom] running in process 6187 2020-07-07 13:52:54,535 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 13:52:54,536 INFO [watch_fvcom] publishing log messages to tcp://*:5587 2020-07-07 13:52:54,537 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-07 13:52:54,572 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch... 2020-07-07 13:53:08,573 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:53:12,259 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:53:37,153 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:53:42,845 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 13:54:06,757 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 13:54:12,729 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 07jul20 figures 2020-07-07 13:54:13,327 INFO [make_plots] fvcom publish plots for 2020-07-07 nowcast-x2 completed 2020-07-07 13:54:13,499 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 13:55:00,528 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634722 = 2020-07-07 16:54:00 UTC, 70.4% complete 2020-07-07 13:56:39,571 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:56:47,837 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:56:55,935 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:02,930 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:10,080 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:17,057 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:24,169 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:31,146 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:38,398 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:42,495 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40254500 = 2020-07-07 08:13:20 UTC estimated time to finish: 0000:08:27:14 2020-07-07 13:57:45,481 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:52,641 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:57:54,674 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16160535 = 2020-07-08 00:32:15 UTC estimated time to finish: 0000:05:17:53 2020-07-07 13:57:59,543 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:06,553 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:13,417 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:20,441 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:27,716 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:34,705 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:41,699 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:48,602 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:58:55,760 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:02,650 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:09,833 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:16,781 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:23,770 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:30,597 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:37,553 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:44,340 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:51,578 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 13:59:59,301 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:00,628 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634776 = 2020-07-07 17:30:00 UTC, 73.0% complete 2020-07-07 14:00:06,273 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:13,485 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:20,555 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:28,447 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:36,316 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:43,747 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:50,927 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:00:58,199 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:05,127 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:11,907 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:18,835 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:25,666 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:32,475 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:39,983 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:47,013 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:01:53,656 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:00,800 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:07,890 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:14,922 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:21,958 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:29,070 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:36,122 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:42,596 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40256280 = 2020-07-07 08:25:12 UTC estimated time to finish: 0000:08:18:34 2020-07-07 14:02:43,442 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:50,489 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:02:54,777 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16162551 = 2020-07-08 01:05:51 UTC estimated time to finish: 0000:05:11:57 2020-07-07 14:02:57,343 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:04,597 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:11,948 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:19,033 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:26,393 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:33,237 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:39,986 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:47,278 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:03:54,116 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:01,073 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:08,193 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:15,361 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:22,378 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:29,607 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:36,506 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:43,638 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:50,495 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:04:57,230 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:00,728 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634831 = 2020-07-07 18:06:40 UTC, 75.5% complete 2020-07-07 14:05:04,146 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:08,732 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:13,045 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:17,550 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:21,831 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:25,935 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:30,280 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:34,565 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:38,915 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:42,988 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:47,122 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:51,578 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:05:55,844 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:00,228 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:04,551 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:08,735 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:13,361 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:17,483 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:21,585 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:25,704 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:29,889 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:34,024 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:38,540 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:43,363 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:48,086 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:52,809 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:06:57,692 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:02,018 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:06,432 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:10,795 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:15,812 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:20,035 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:24,937 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:29,266 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:33,505 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:38,059 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:42,387 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:42,721 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40258092 = 2020-07-07 08:37:16 UTC estimated time to finish: 0000:08:09:51 2020-07-07 14:07:47,113 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:51,467 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:07:54,880 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16164556 = 2020-07-08 01:39:16 UTC estimated time to finish: 0000:05:08:15 2020-07-07 14:07:55,762 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:00,106 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:04,268 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:08,368 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:12,747 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:17,234 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:21,947 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:26,460 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:30,862 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:35,366 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:39,471 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:44,080 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:48,637 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:52,937 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:08:57,302 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:01,653 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:06,223 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:11,284 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:15,778 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:20,182 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:24,675 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:28,726 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:33,028 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:37,271 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:41,817 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:46,370 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:51,512 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:09:55,822 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:00,328 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:00,828 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634885 = 2020-07-07 18:42:40 UTC, 78.0% complete 2020-07-07 14:10:04,509 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:08,744 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:13,246 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:17,426 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:21,584 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:26,744 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:31,664 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:36,941 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:41,998 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:47,235 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:52,545 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:10:57,562 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:02,525 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:07,414 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:12,694 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:17,907 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:22,856 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:27,694 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:32,640 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:37,612 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:42,582 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:47,484 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:52,630 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:11:57,688 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:02,607 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:07,613 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:12,896 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:17,741 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:22,811 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:27,811 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:32,794 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:37,894 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:42,848 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40259880 = 2020-07-07 08:49:12 UTC estimated time to finish: 0000:08:01:24 2020-07-07 14:12:43,100 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:48,147 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:53,063 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:12:54,984 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16166369 = 2020-07-08 02:09:29 UTC estimated time to finish: 0000:05:04:25 2020-07-07 14:12:58,063 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:03,074 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:08,222 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:13,332 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:18,067 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:22,961 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:27,757 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:32,640 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:37,513 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:42,748 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:48,078 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:52,905 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:13:57,697 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:02,633 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:07,733 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:12,524 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:17,121 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:21,993 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:27,106 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:32,481 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:37,429 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:42,501 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:47,453 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:52,637 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:14:57,573 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:00,928 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634939 = 2020-07-07 19:18:40 UTC, 80.5% complete 2020-07-07 14:15:02,194 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:06,937 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:11,712 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:16,573 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:21,508 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:26,249 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:31,087 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:35,888 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:40,640 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:45,575 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:50,461 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:15:55,454 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:16:00,582 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:16:05,453 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:16:10,654 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:16:15,668 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:16:20,552 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 126, in _prep_plot_data xx, zz, vi, _, __, ___ = fpp.vertical_transect_snap( File "/SalishSeaCast/OPPTools/OPPTools/utils/fvcom_postprocess.py", line 843, in vertical_transect_snap timestr = datetime.strftime(nctime(f,ktime), ' %Y-%m-%d %H:%M UTC') TypeError: descriptor 'strftime' for 'datetime.date' objects doesn't apply to a 'cftime._cftime.DatetimeGregorian' object 2020-07-07 14:16:20,659 INFO [make_plots] finished rendering fvcom nowcast-x2 research 07jul20 figures 2020-07-07 14:16:20,661 INFO [make_plots] fvcom research plots for 2020-07-07 nowcast-x2 completed 2020-07-07 14:16:20,998 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 14:17:42,976 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40261655 = 2020-07-07 09:01:02 UTC estimated time to finish: 0000:07:53:17 2020-07-07 14:17:55,089 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16168414 = 2020-07-08 02:43:34 UTC estimated time to finish: 0000:04:59:28 2020-07-07 14:20:01,028 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4634993 = 2020-07-07 19:54:40 UTC, 83.0% complete 2020-07-07 14:22:43,103 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40263226 = 2020-07-07 09:11:30 UTC estimated time to finish: 0000:07:46:10 2020-07-07 14:22:55,194 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16170471 = 2020-07-08 03:17:51 UTC estimated time to finish: 0000:04:54:37 2020-07-07 14:25:01,063 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635047 = 2020-07-07 20:30:40 UTC, 85.5% complete 2020-07-07 14:27:43,231 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40265001 = 2020-07-07 09:23:20 UTC estimated time to finish: 0000:07:38:16 2020-07-07 14:27:55,300 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16172516 = 2020-07-08 03:51:56 UTC estimated time to finish: 0000:04:49:28 2020-07-07 14:30:01,081 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635102 = 2020-07-07 21:07:20 UTC, 88.1% complete 2020-07-07 14:32:43,359 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40266819 = 2020-07-07 09:35:27 UTC estimated time to finish: 0000:07:30:13 2020-07-07 14:32:55,386 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16174370 = 2020-07-08 04:22:50 UTC estimated time to finish: 0000:04:44:43 2020-07-07 14:35:01,169 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635156 = 2020-07-07 21:43:20 UTC, 90.6% complete 2020-07-07 14:37:43,436 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40268594 = 2020-07-07 09:47:17 UTC estimated time to finish: 0000:07:22:33 2020-07-07 14:37:55,435 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16176414 = 2020-07-08 04:56:54 UTC estimated time to finish: 0000:04:39:18 2020-07-07 14:40:01,268 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635211 = 2020-07-07 22:20:00 UTC, 93.1% complete 2020-07-07 14:42:43,565 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40270178 = 2020-07-07 09:57:51 UTC estimated time to finish: 0000:07:15:45 2020-07-07 14:42:55,542 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16178459 = 2020-07-08 05:30:59 UTC estimated time to finish: 0000:04:34:18 2020-07-07 14:45:01,368 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635265 = 2020-07-07 22:56:00 UTC, 95.6% complete 2020-07-07 14:47:43,696 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40271953 = 2020-07-07 10:09:41 UTC estimated time to finish: 0000:07:08:20 2020-07-07 14:47:55,651 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16180516 = 2020-07-08 06:05:16 UTC estimated time to finish: 0000:04:29:08 2020-07-07 14:50:01,463 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635319 = 2020-07-07 23:32:00 UTC, 98.1% complete 2020-07-07 14:52:43,825 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40273728 = 2020-07-07 10:21:31 UTC estimated time to finish: 0000:07:00:53 2020-07-07 14:52:55,759 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16182521 = 2020-07-08 06:38:41 UTC estimated time to finish: 0000:04:24:26 2020-07-07 14:55:01,564 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4635360 = 2020-07-07 23:59:20 UTC, 100.0% complete 2020-07-07 14:57:43,955 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40275546 = 2020-07-07 10:33:38 UTC estimated time to finish: 0000:06:53:15 2020-07-07 14:57:55,869 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16184334 = 2020-07-08 07:08:54 UTC estimated time to finish: 0000:04:20:05 2020-07-07 15:00:01,681 INFO [watch_NEMO] nowcast-dev NEMO run on salish-nowcast completed 2020-07-07 15:00:01,745 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-07 15:02:44,087 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40277160 = 2020-07-07 10:44:24 UTC estimated time to finish: 0000:06:46:34 2020-07-07 15:02:55,978 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16186380 = 2020-07-08 07:43:00 UTC estimated time to finish: 0000:04:15:07 2020-07-07 15:03:47,677 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/18/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200707/18/ 2020-07-07 15:03:47,678 INFO [collect_weather] 2020-07-07 2.5km weather forecast 18 collection complete 2020-07-07 15:03:47,681 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-07 15:03:48,278 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km', '--no-verify-certs'], host='localhost') 2020-07-07 15:03:48,282 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km', '--no-verify-certs'], host='localhost') 2020-07-07 15:03:48,286 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2020-07-07 15:03:54,562 INFO [download_weather] running in process 29495 2020-07-07 15:03:54,665 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 15:03:54,666 INFO [download_weather] publishing log messages to tcp://*:5562 2020-07-07 15:03:54,667 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-07 15:03:54,669 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20200707 2020-07-07 15:03:54,670 INFO [collect_weather] running in process 29496 2020-07-07 15:03:54,671 INFO [download_weather] running in process 29494 2020-07-07 15:03:54,671 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 15:03:54,672 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 15:03:54,673 INFO [collect_weather] publishing log messages to tcp://*:5567 2020-07-07 15:03:54,673 INFO [download_weather] publishing log messages to tcp://*:5569 2020-07-07 15:03:54,674 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-07 15:03:54,675 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-07 15:03:54,677 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200707 2020-07-07 15:03:54,678 ERROR [download_weather] /results/forcing/atmospheric/GEM1.0/GRIB/20200707/00 directory already exists; not overwriting 2020-07-07 15:03:54,680 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/00/ 2020-07-07 15:03:55,852 CRITICAL [download_weather] 2020-07-07 1km weather forecast 00 downloads failed 2020-07-07 15:07:44,219 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40278945 = 2020-07-07 10:56:18 UTC estimated time to finish: 0000:06:39:23 2020-07-07 15:07:56,089 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16188396 = 2020-07-08 08:16:36 UTC estimated time to finish: 0000:04:10:16 2020-07-07 15:12:44,351 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40280720 = 2020-07-07 11:08:08 UTC estimated time to finish: 0000:06:32:18 2020-07-07 15:12:56,200 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16190400 = 2020-07-08 08:50:00 UTC estimated time to finish: 0000:04:05:27 2020-07-07 15:15:47,327 INFO [download_weather] running in process 29654 2020-07-07 15:15:47,328 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 15:15:47,328 INFO [download_weather] publishing log messages to tcp://*:5569 2020-07-07 15:15:47,330 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-07 15:15:47,331 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200707 2020-07-07 15:17:44,484 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40282536 = 2020-07-07 11:20:14 UTC estimated time to finish: 0000:06:25:03 2020-07-07 15:17:56,312 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16192213 = 2020-07-08 09:20:13 UTC estimated time to finish: 0000:04:01:08 2020-07-07 15:22:44,781 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40284272 = 2020-07-07 11:31:48 UTC estimated time to finish: 0000:06:18:16 2020-07-07 15:22:56,425 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16194190 = 2020-07-08 09:53:10 UTC estimated time to finish: 0000:03:56:36 2020-07-07 15:27:44,752 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40285884 = 2020-07-07 11:42:33 UTC estimated time to finish: 0000:06:11:58 2020-07-07 15:27:56,538 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16196193 = 2020-07-08 10:26:33 UTC estimated time to finish: 0000:03:51:50 2020-07-07 15:32:44,888 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40287660 = 2020-07-07 11:54:24 UTC estimated time to finish: 0000:06:05:09 2020-07-07 15:32:56,653 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16198200 = 2020-07-08 11:00:00 UTC estimated time to finish: 0000:03:46:50 2020-07-07 15:37:45,023 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40289475 = 2020-07-07 12:06:30 UTC estimated time to finish: 0000:05:58:11 2020-07-07 15:37:56,767 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16200011 = 2020-07-08 11:30:11 UTC estimated time to finish: 0000:03:42:22 2020-07-07 15:42:45,159 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40291250 = 2020-07-07 12:18:20 UTC estimated time to finish: 0000:05:51:26 2020-07-07 15:42:56,883 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16202029 = 2020-07-08 12:03:49 UTC estimated time to finish: 0000:03:37:23 2020-07-07 15:47:16,090 INFO [download_weather] 2020-07-07 1km weather forecast 12 downloads complete 2020-07-07 15:47:16,094 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2020-07-07 15:47:45,295 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40292876 = 2020-07-07 12:29:10 UTC estimated time to finish: 0000:05:45:13 2020-07-07 15:47:57,002 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16204033 = 2020-07-08 12:37:13 UTC estimated time to finish: 0000:03:32:20 2020-07-07 15:52:45,430 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40294651 = 2020-07-07 12:41:00 UTC estimated time to finish: 0000:05:38:34 2020-07-07 15:52:57,114 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16206090 = 2020-07-08 13:11:30 UTC estimated time to finish: 0000:03:27:09 2020-07-07 15:57:45,567 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40296427 = 2020-07-07 12:52:50 UTC estimated time to finish: 0000:05:32:02 2020-07-07 15:57:57,231 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16208096 = 2020-07-08 13:44:56 UTC estimated time to finish: 0000:03:22:06 2020-07-07 16:02:45,705 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40298202 = 2020-07-07 13:04:40 UTC estimated time to finish: 0000:05:25:29 2020-07-07 16:02:57,348 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16209949 = 2020-07-08 14:15:49 UTC estimated time to finish: 0000:03:17:23 2020-07-07 16:07:45,843 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40299977 = 2020-07-07 13:16:30 UTC estimated time to finish: 0000:05:19:00 2020-07-07 16:07:57,467 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16211953 = 2020-07-08 14:49:13 UTC estimated time to finish: 0000:03:12:23 2020-07-07 16:12:45,979 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40301561 = 2020-07-07 13:27:04 UTC estimated time to finish: 0000:05:13:17 2020-07-07 16:12:57,586 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16213970 = 2020-07-08 15:22:50 UTC estimated time to finish: 0000:03:07:20 2020-07-07 16:17:46,120 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40303297 = 2020-07-07 13:38:38 UTC estimated time to finish: 0000:05:07:06 2020-07-07 16:17:57,705 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16215973 = 2020-07-08 15:56:13 UTC estimated time to finish: 0000:03:02:20 2020-07-07 16:22:46,259 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40305071 = 2020-07-07 13:50:28 UTC estimated time to finish: 0000:05:00:44 2020-07-07 16:22:57,826 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16217827 = 2020-07-08 16:27:07 UTC estimated time to finish: 0000:02:57:36 2020-07-07 16:27:46,398 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 40306806 = 2020-07-07 14:02:02 UTC estimated time to finish: 0000:04:54:34 2020-07-07 16:27:57,952 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16219912 = 2020-07-08 17:01:52 UTC estimated time to finish: 0000:02:52:14 2020-07-07 17:49:36,401 ERROR [download_weather] giving up; download from https://dd.alpha.meteo.gc.ca/model_hrdps/west/1km/grib2/00/001/CMC_hrdps_west_UGRD_TGL_10_rotated_latlon0.009x0.009_20200707T00Z_P001-00.grib2 failed 43 times 2020-07-07 17:49:38,658 CRITICAL [download_weather] 2020-07-07 1km weather forecast 00 downloads failed 2020-07-07 19:23:02,406 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker 2020-07-07 19:23:02,873 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 19:23:05,955 INFO [download_fvcom_results] running in process 2618 2020-07-07 19:23:05,956 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:23:05,957 INFO [download_fvcom_results] publishing log messages to tcp://*:5562 2020-07-07 19:23:05,957 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:24:37,431 INFO [download_fvcom_results] VHFR FVCOM x2 forecast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 19:24:37,438 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker 2020-07-07 19:24:37,502 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-07'], host='localhost') 2020-07-07 19:24:37,506 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'research', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 19:24:37,509 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['fvcom', 'forecast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 19:24:41,066 INFO [update_forecast_datasets] running in process 2632 2020-07-07 19:24:41,067 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:24:41,068 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2020-07-07 19:24:41,069 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:24:41,070 INFO [update_forecast_datasets] updating fvcom most_recent_forecast directory from forecast/07jul20 run 2020-07-07 19:24:41,075 INFO [update_forecast_datasets] updated fvcom most_recent_forecast directory from forecast/07jul20 run 2020-07-07 19:24:41,075 INFO [update_forecast_datasets] fvcom 2020-07-07 forecast rolling forecast datasets updated 2020-07-07 19:24:41,283 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-07 19:24:41,358 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['fvcom-forecast'], host='localhost') 2020-07-07 19:24:47,769 INFO [get_vfpa_hadcp] running in process 2630 2020-07-07 19:24:47,770 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:24:47,772 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562 2020-07-07 19:24:47,773 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:24:47,774 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-07 2020-07-07 19:24:51,558 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-07 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc 2020-07-07 19:24:51,558 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file 2020-07-07 19:24:51,562 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2020-07-07 19:24:51,643 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2020-07-07 19:24:53,863 INFO [ping_erddap] running in process 2660 2020-07-07 19:24:53,864 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:24:53,865 INFO [ping_erddap] publishing log messages to tcp://*:5562 2020-07-07 19:24:53,945 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:24:54,068 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2020-07-07 19:24:54,070 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 19:24:54,116 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 19:24:54,166 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 19:24:55,101 INFO [make_plots] running in process 2631 2020-07-07 19:24:55,136 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:24:55,138 INFO [make_plots] publishing log messages to tcp://*:5562 2020-07-07 19:24:55,138 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:25:12,026 INFO [make_plots] preparing render list for 2020-07-07 VHFR FVCOM forecast-x2 research figures 2020-07-07 19:25:12,032 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-07 VHFR FVCOM forecast-x2 research render list 2020-07-07 19:25:12,037 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-07 VHFR FVCOM forecast-x2 research render list 2020-07-07 19:25:12,041 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-07 VHFR FVCOM forecast-x2 research render list 2020-07-07 19:25:12,047 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-07 VHFR FVCOM forecast-x2 research render list 2020-07-07 19:25:12,053 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-07 VHFR FVCOM forecast-x2 research render list 2020-07-07 19:25:12,058 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-07 VHFR FVCOM forecast-x2 research render list 2020-07-07 19:25:12,059 INFO [make_plots] starting to render fvcom forecast-x2 research 07jul20 figures 2020-07-07 19:25:15,085 INFO [make_plots] running in process 2668 2020-07-07 19:25:15,086 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:25:15,086 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-07 19:25:15,087 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:25:15,310 INFO [make_plots] running in process 2669 2020-07-07 19:25:15,311 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 19:25:15,312 INFO [make_plots] publishing log messages to tcp://*:5569 2020-07-07 19:25:15,313 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 19:25:15,660 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 282, in make_plots fvcom_stns_datasets[model_config] = xarray.open_dataset( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/api.py", line 538, in open_dataset ds = maybe_decode_store(store) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/api.py", line 446, in maybe_decode_store ds = conventions.decode_cf( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/conventions.py", line 587, in decode_cf ds = Dataset(vars, attrs=attrs) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataset.py", line 539, in __init__ variables, coord_names, dims, indexes = merge_data_and_coords( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/merge.py", line 466, in merge_data_and_coords return merge_core( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/merge.py", line 552, in merge_core collected = collect_variables_and_indexes(aligned) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/merge.py", line 277, in collect_variables_and_indexes variable = as_variable(variable, name=name) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/variable.py", line 134, in as_variable raise MissingDimensionsError( xarray.core.variable.MissingDimensionsError: 'siglay' has more than 1-dimension and the same name as one of its dimensions ('siglay', 'station'). xarray disallows such variables because they conflict with the coordinates used to label dimensions. 2020-07-07 19:25:17,627 INFO [make_plots] preparing render list for 2020-07-07 VHFR FVCOM nowcast-x2 publish figures 2020-07-07 19:25:17,876 INFO [make_plots] added Calamity Point figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,877 INFO [make_plots] added Indian Arm Head figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,877 INFO [make_plots] added New Westminster figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,878 INFO [make_plots] added Port Moody figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,878 INFO [make_plots] added Sand Heads figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,879 INFO [make_plots] added Sandy Cove figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,879 INFO [make_plots] added Vancouver Harbour figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,880 INFO [make_plots] added Woodwards Landing figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,880 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-07 VHFR FVCOM nowcast-x2 publish render list 2020-07-07 19:25:17,881 INFO [make_plots] starting to render fvcom nowcast-x2 publish 07jul20 figures 2020-07-07 19:25:17,903 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:25:37,019 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:25:40,589 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:25:41,184 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:25:43,249 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:26:14,938 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:26:18,185 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:26:48,624 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:26:52,750 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:27:20,597 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:27:24,484 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:27:53,701 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:28:01,997 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:28:31,571 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:28:35,641 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 19:29:04,382 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 19:29:10,708 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 07jul20 figures 2020-07-07 19:29:11,112 INFO [make_plots] fvcom publish plots for 2020-07-07 nowcast-x2 completed 2020-07-07 19:29:11,268 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 20:22:53,384 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker 2020-07-07 20:22:53,888 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 20:23:01,834 INFO [download_fvcom_results] running in process 4413 2020-07-07 20:23:01,836 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 20:23:01,837 INFO [download_fvcom_results] publishing log messages to tcp://*:5568 2020-07-07 20:23:01,838 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555 2020-07-07 20:26:19,128 INFO [download_fvcom_results] VHFR FVCOM r12 nowcast 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-07 20:26:19,133 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker 2020-07-07 20:26:19,385 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-07'], host='localhost') 2020-07-07 20:26:19,390 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'research', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 20:26:43,026 INFO [get_vfpa_hadcp] running in process 4489 2020-07-07 20:26:43,029 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 20:26:43,174 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5568 2020-07-07 20:26:43,324 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2020-07-07 20:26:43,325 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-07 2020-07-07 20:26:47,794 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-07 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc 2020-07-07 20:26:47,795 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file 2020-07-07 20:26:47,796 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2020-07-07 20:26:47,976 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2020-07-07 20:26:56,012 INFO [ping_erddap] running in process 4505 2020-07-07 20:26:56,237 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 20:26:56,238 INFO [ping_erddap] publishing log messages to tcp://*:5568 2020-07-07 20:26:56,239 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-07 20:26:56,241 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2020-07-07 20:26:56,243 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-07 20:26:56,299 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-07 20:26:58,317 INFO [make_plots] running in process 4490 2020-07-07 20:26:58,580 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 20:26:58,580 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-07 20:26:58,581 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 20:26:58,609 INFO [make_plots] preparing render list for 2020-07-07 VHFR FVCOM nowcast-r12 research figures 2020-07-07 20:26:58,611 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-07 VHFR FVCOM nowcast-r12 research render list 2020-07-07 20:26:58,614 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-07 VHFR FVCOM nowcast-r12 research render list 2020-07-07 20:26:58,617 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-07 VHFR FVCOM nowcast-r12 research render list 2020-07-07 20:26:58,619 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-07 VHFR FVCOM nowcast-r12 research render list 2020-07-07 20:26:58,622 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-07 VHFR FVCOM nowcast-r12 research render list 2020-07-07 20:26:58,625 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-07 VHFR FVCOM nowcast-r12 research render list 2020-07-07 20:26:58,626 INFO [make_plots] starting to render fvcom nowcast-r12 research 07jul20 figures 2020-07-07 20:27:35,847 INFO [make_plots] running in process 4510 2020-07-07 20:27:35,848 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 20:27:35,849 INFO [make_plots] publishing log messages to tcp://*:5569 2020-07-07 20:27:35,850 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-07 20:27:40,141 INFO [make_plots] preparing render list for 2020-07-07 VHFR FVCOM nowcast-r12 publish figures 2020-07-07 20:27:40,142 INFO [make_plots] added Calamity Point figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,143 INFO [make_plots] added Indian Arm Head figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,143 INFO [make_plots] added New Westminster figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,144 INFO [make_plots] added Port Moody figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,145 INFO [make_plots] added Sand Heads figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,146 INFO [make_plots] added Sandy Cove figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,146 INFO [make_plots] added Vancouver Harbour figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,147 INFO [make_plots] added Woodwards Landing figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,151 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-07 VHFR FVCOM nowcast-r12 publish render list 2020-07-07 20:27:40,152 INFO [make_plots] starting to render fvcom nowcast-r12 publish 07jul20 figures 2020-07-07 20:27:40,170 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:28:02,627 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:28:05,229 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:28:07,149 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:28:08,884 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:28:37,389 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:28:41,583 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:29:12,116 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:29:16,279 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:29:46,236 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:29:49,779 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:30:20,122 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:30:24,174 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:30:54,043 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:30:58,541 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-06 23:55:00 to 2020-07-08 00:00:00 2020-07-07 20:31:25,302 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-07 00:00:00 to 2020-07-08 00:00:00 2020-07-07 20:31:34,584 INFO [make_plots] finished rendering fvcom nowcast-r12 publish 07jul20 figures 2020-07-07 20:31:35,178 INFO [make_plots] fvcom publish plots for 2020-07-07 nowcast-r12 completed 2020-07-07 20:31:35,187 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 20:36:21,440 INFO [make_plots] finished rendering fvcom forecast-x2 research 07jul20 figures 2020-07-07 20:36:21,441 INFO [make_plots] fvcom research plots for 2020-07-07 forecast-x2 completed 2020-07-07 20:36:21,443 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-07 21:03:24,304 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/00/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200708/00/ 2020-07-07 21:03:24,370 INFO [collect_weather] 2020-07-08 2.5km weather forecast 00 collection complete 2020-07-07 21:03:24,372 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-07 21:03:24,896 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2020-07-07 21:03:27,771 INFO [collect_weather] running in process 5253 2020-07-07 21:03:27,796 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-07 21:03:27,796 INFO [collect_weather] publishing log messages to tcp://*:5562 2020-07-07 21:03:27,797 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-07 21:03:27,799 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/06/ 2020-07-07 21:47:08,304 INFO [make_plots] finished rendering fvcom nowcast-r12 research 07jul20 figures 2020-07-07 21:47:08,305 INFO [make_plots] fvcom research plots for 2020-07-07 nowcast-r12 completed 2020-07-07 21:47:08,430 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-08 03:09:49,075 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/06/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200708/06/ 2020-07-08 03:09:49,086 INFO [collect_weather] 2020-07-08 2.5km weather forecast 06 collection complete 2020-07-08 03:09:49,090 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-08 03:09:49,731 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Capilano', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:49,987 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ChilliwackVedder', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:49,992 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ClowhomClowhomLake', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,001 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Englishman', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,017 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Fraser', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,031 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['HomathkoMouth', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,036 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SalmonSayward', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,040 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SanJuanPortRenfrew', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,054 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SquamishBrackendale', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,056 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaScotty', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,058 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaBypass', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,062 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaDiversion', '--data-date', '2020-07-07'], host='localhost') 2020-07-08 03:09:50,065 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2020-07-08 03:09:50,071 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2020-07-08 03:09:50,075 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2020-07-08 03:09:50,084 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast2'], host='localhost') 2020-07-08 03:09:50,089 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2020-07-08 03:09:50,092 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2020-07-08 03:09:53,215 INFO [collect_weather] running in process 13688 2020-07-08 03:09:53,242 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:53,243 INFO [collect_weather] publishing log messages to tcp://*:5562 2020-07-08 03:09:53,244 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:53,248 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/12/ 2020-07-08 03:09:55,335 INFO [collect_river_data] running in process 13674 2020-07-08 03:09:55,336 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,337 INFO [collect_river_data] publishing log messages to tcp://*:5564 2020-07-08 03:09:55,338 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,339 INFO [collect_river_data] Collecting Fraser river data for 2020-07-07 2020-07-08 03:09:55,354 INFO [collect_river_data] running in process 13682 2020-07-08 03:09:55,355 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,356 INFO [collect_river_data] publishing log messages to tcp://*:5567 2020-07-08 03:09:55,357 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,357 INFO [collect_river_data] Collecting TheodosiaDiversion river data for 2020-07-07 2020-07-08 03:09:55,358 INFO [collect_river_data] running in process 13669 2020-07-08 03:09:55,359 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,360 INFO [collect_river_data] publishing log messages to tcp://*:5568 2020-07-08 03:09:55,361 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,361 INFO [collect_river_data] Collecting Capilano river data for 2020-07-07 2020-07-08 03:09:55,362 INFO [collect_river_data] running in process 13677 2020-07-08 03:09:55,363 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,364 INFO [collect_river_data] publishing log messages to tcp://*:5569 2020-07-08 03:09:55,365 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,365 INFO [collect_river_data] Collecting SanJuanPortRenfrew river data for 2020-07-07 2020-07-08 03:09:55,367 INFO [collect_river_data] running in process 13673 2020-07-08 03:09:55,368 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,368 INFO [collect_river_data] publishing log messages to tcp://*:5570 2020-07-08 03:09:55,369 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,370 INFO [collect_river_data] Collecting Englishman river data for 2020-07-07 2020-07-08 03:09:55,383 INFO [collect_river_data] running in process 13675 2020-07-08 03:09:55,440 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,442 INFO [collect_river_data] publishing log messages to tcp://*:5571 2020-07-08 03:09:55,443 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,444 INFO [collect_river_data] Collecting HomathkoMouth river data for 2020-07-07 2020-07-08 03:09:55,493 INFO [collect_river_data] running in process 13671 2020-07-08 03:09:55,507 INFO [collect_river_data] running in process 13679 2020-07-08 03:09:55,513 INFO [collect_river_data] running in process 13676 2020-07-08 03:09:55,515 INFO [collect_river_data] running in process 13681 2020-07-08 03:09:55,515 INFO [collect_river_data] running in process 13678 2020-07-08 03:09:55,516 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,517 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,517 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,518 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:55,518 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,520 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,520 INFO [collect_river_data] publishing log messages to tcp://*:5572 2020-07-08 03:09:55,521 INFO [collect_river_data] publishing log messages to tcp://*:5573 2020-07-08 03:09:55,521 INFO [collect_river_data] publishing log messages to tcp://*:5574 2020-07-08 03:09:55,522 INFO [collect_river_data] publishing log messages to tcp://*:5575 2020-07-08 03:09:55,523 INFO [collect_river_data] publishing log messages to tcp://*:5576 2020-07-08 03:09:55,523 INFO [collect_river_data] Appended SanJuanPortRenfrew river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2020-07-08 03:09:55,524 INFO [collect_river_data] Appended TheodosiaDiversion river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2020-07-08 03:09:55,524 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,525 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,526 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,544 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,545 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,546 INFO [collect_river_data] SanJuanPortRenfrew river average discharge for 2020-07-07 calculated and appended to SanJuanPortRenfrew_flow file 2020-07-08 03:09:55,550 INFO [collect_river_data] TheodosiaDiversion river average discharge for 2020-07-07 calculated and appended to TheodosiaDiversion_flow file 2020-07-08 03:09:55,550 INFO [collect_river_data] Collecting ChilliwackVedder river data for 2020-07-07 2020-07-08 03:09:55,551 INFO [collect_river_data] Collecting TheodosiaScotty river data for 2020-07-07 2020-07-08 03:09:55,551 INFO [collect_river_data] Collecting SalmonSayward river data for 2020-07-07 2020-07-08 03:09:55,552 INFO [collect_river_data] Collecting TheodosiaBypass river data for 2020-07-07 2020-07-08 03:09:55,553 INFO [collect_river_data] Collecting SquamishBrackendale river data for 2020-07-07 2020-07-08 03:09:55,557 INFO [collect_river_data] Appended SalmonSayward river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2020-07-08 03:09:55,557 INFO [collect_river_data] Appended TheodosiaScotty river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2020-07-08 03:09:55,558 INFO [collect_river_data] SalmonSayward river average discharge for 2020-07-07 calculated and appended to SalmonSayward_flow file 2020-07-08 03:09:55,559 INFO [collect_river_data] TheodosiaScotty river average discharge for 2020-07-07 calculated and appended to TheodosiaScotty_flow file 2020-07-08 03:09:55,561 INFO [grib_to_netcdf] running in process 13687 2020-07-08 03:09:55,561 INFO [collect_river_data] running in process 13672 2020-07-08 03:09:55,563 INFO [collect_river_data] Appended Capilano river average discharge for 2020-07-07 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2020-07-08 03:09:55,563 INFO [collect_river_data] Appended Fraser river average discharge for 2020-07-07 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2020-07-08 03:09:55,564 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,565 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:55,566 INFO [collect_river_data] Capilano river average discharge for 2020-07-07 calculated and appended to Capilano_flow file 2020-07-08 03:09:55,566 INFO [collect_river_data] Fraser river average discharge for 2020-07-07 calculated and appended to Fraser_flow file 2020-07-08 03:09:55,567 INFO [grib_to_netcdf] publishing log messages to tcp://*:5577 2020-07-08 03:09:55,568 INFO [collect_river_data] publishing log messages to tcp://*:5578 2020-07-08 03:09:55,568 INFO [collect_river_data] Appended Englishman river average discharge for 2020-07-07 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2020-07-08 03:09:55,569 INFO [collect_river_data] Appended HomathkoMouth river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2020-07-08 03:09:55,570 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,571 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:55,571 INFO [collect_river_data] Englishman river average discharge for 2020-07-07 calculated and appended to Englishman_flow file 2020-07-08 03:09:55,572 INFO [collect_river_data] HomathkoMouth river average discharge for 2020-07-07 calculated and appended to HomathkoMouth_flow file 2020-07-08 03:09:55,573 INFO [grib_to_netcdf] forecast section: 20200708/06 2020-07-08 03:09:55,574 INFO [collect_river_data] Collecting ClowhomClowhomLake river data for 2020-07-07 2020-07-08 03:09:55,574 INFO [grib_to_netcdf] next day forecast section: 20200708/06 2020-07-08 03:09:55,575 INFO [collect_river_data] Appended ChilliwackVedder river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2020-07-08 03:09:55,576 INFO [collect_river_data] ChilliwackVedder river average discharge for 2020-07-07 calculated and appended to ChilliwackVedder_flow file 2020-07-08 03:09:55,841 INFO [collect_river_data] Appended SquamishBrackendale river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2020-07-08 03:09:55,842 INFO [collect_river_data] SquamishBrackendale river average discharge for 2020-07-07 calculated and appended to SquamishBrackendale_flow file 2020-07-08 03:09:55,952 INFO [collect_river_data] Appended ClowhomClowhomLake river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2020-07-08 03:09:55,953 INFO [collect_river_data] Appended TheodosiaBypass river average discharge for 2020-07-07 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2020-07-08 03:09:56,023 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:56,024 INFO [collect_river_data] ClowhomClowhomLake river average discharge for 2020-07-07 calculated and appended to ClowhomClowhomLake_flow file 2020-07-08 03:09:56,025 INFO [collect_river_data] TheodosiaBypass river average discharge for 2020-07-07 calculated and appended to TheodosiaBypass_flow file 2020-07-08 03:09:56,079 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:56,376 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:56,536 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:56,703 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2020-07-08 03:09:56,710 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:56,910 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:57,096 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:57,297 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:57,445 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:57,691 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:57,851 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-08 03:09:58,352 INFO [get_onc_ctd] running in process 13683 2020-07-08 03:09:58,353 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:58,354 INFO [get_onc_ctd] publishing log messages to tcp://*:5564 2020-07-08 03:09:58,355 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:58,356 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2020-07-07 2020-07-08 03:09:58,381 INFO [get_NeahBay_ssh] running in process 13686 2020-07-08 03:09:58,382 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:58,383 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5567 2020-07-08 03:09:58,384 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:58,411 INFO [get_onc_ctd] running in process 13684 2020-07-08 03:09:58,433 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:58,434 INFO [get_onc_ctd] publishing log messages to tcp://*:5568 2020-07-08 03:09:58,436 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:58,440 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2020-07-07 2020-07-08 03:09:58,493 INFO [get_onc_ferry] running in process 13685 2020-07-08 03:09:58,504 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:58,505 INFO [get_onc_ferry] publishing log messages to tcp://*:5569 2020-07-08 03:09:58,505 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:58,506 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2020-07-07 2020-07-08 03:09:58,798 INFO [make_runoff_file] running in process 13754 2020-07-08 03:09:58,799 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:09:58,800 INFO [make_runoff_file] publishing log messages to tcp://*:5570 2020-07-08 03:09:58,800 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:09:59,084 INFO [make_runoff_file] calculating runoff forcing for bathymetry type: b201702 2020-07-08 03:09:59,190 INFO [make_runoff_file] 2020-07-08 runoff file creation from Fraser at Hope and climatology elsewhere complete 2020-07-08 03:09:59,193 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2020-07-08 03:10:03,086 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast2 complete 2020-07-08 03:10:03,091 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker 2020-07-08 03:10:21,914 INFO [get_onc_ctd] 2020-07-07 ONC SCVIP CTD T&S file created 2020-07-08 03:10:21,917 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2020-07-08 03:10:22,109 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost') 2020-07-08 03:10:22,349 INFO [get_onc_ctd] 2020-07-07 ONC SEVIP CTD T&S file created 2020-07-08 03:10:22,351 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2020-07-08 03:10:22,544 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2020-07-08 03:10:22,865 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2020-07-07 2020-07-08 03:10:23,498 INFO [ping_erddap] running in process 14103 2020-07-08 03:10:23,500 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:10:23,502 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-08 03:10:23,503 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:10:23,505 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created 2020-07-08 03:10:23,508 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-08 03:10:23,632 INFO [ping_erddap] running in process 14105 2020-07-08 03:10:23,634 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:10:23,636 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-08 03:10:23,638 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:10:23,640 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2020-07-08 03:10:23,668 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-08 03:10:53,646 INFO [grib_to_netcdf] 2020-07-08 NEMO-atmos forcing file for forecast2 created 2020-07-08 03:10:53,649 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2020-07-08 03:10:53,824 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2020-07-08 03:10:53,827 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2020-07-08 03:10:53,830 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'forecast2'], host='localhost') 2020-07-08 03:10:53,834 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2020-07-08 03:10:54,904 INFO [upload_forcing] running in process 14531 2020-07-08 03:10:54,905 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:10:54,906 INFO [upload_forcing] publishing log messages to tcp://*:5564 2020-07-08 03:10:54,906 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:10:54,908 INFO [upload_forcing] running in process 14532 2020-07-08 03:10:54,909 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:10:54,909 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-08 03:10:54,910 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:10:54,914 INFO [upload_forcing] running in process 14530 2020-07-08 03:10:54,914 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:10:54,915 INFO [upload_forcing] publishing log messages to tcp://*:5568 2020-07-08 03:10:54,916 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:10:54,919 INFO [upload_forcing] running in process 14529 2020-07-08 03:10:54,920 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:10:54,920 INFO [upload_forcing] publishing log messages to tcp://*:5570 2020-07-08 03:10:54,921 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:10:55,473 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 144, in _upload_ssh_files ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 143, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 759, in put return self.putfo(fl, remotepath, file_size, callback, confirm) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 714, in putfo with self.file(remotepath, "wb") as fr: File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 372, in open t, msg = self._request(CMD_OPEN, filename, imode, attrblock) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 813, in _request return self._read_response(num) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 865, in _read_response self._convert_status(msg) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 894, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 102, in upload_forcing _upload_ssh_files(sftp_client, run_type, run_date, config, host_name, host_config) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 151, in _upload_ssh_files fcst.symlink_to(localpath) File "/SalishSeaCast/nowcast-env/lib/python3.8/pathlib.py", line 1374, in symlink_to self._accessor.symlink(target, self, target_is_directory) File "/SalishSeaCast/nowcast-env/lib/python3.8/pathlib.py", line 442, in symlink return os.symlink(a, b) FileExistsError: [Errno 17] File exists: '/results/forcing/sshNeahBay/obs/ssh_y2020m07d07.nc' -> '/results/forcing/sshNeahBay/fcst/ssh_y2020m07d07.nc' 2020-07-08 03:10:56,227 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2020-07-07 2020-07-08 03:10:57,554 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d07.nc 2020-07-08 03:10:57,679 INFO [upload_forcing] forecast2 2020-07-08 forcing files upload to orcinus-nowcast-agrif completed 2020-07-08 03:10:57,683 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-08 03:10:59,105 INFO [upload_forcing] forecast2 2020-07-08 forcing files upload to arbutus.cloud-nowcast completed 2020-07-08 03:10:59,110 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-08 03:10:59,310 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-08'], host='localhost') 2020-07-08 03:11:00,248 INFO [make_forcing_links] running in process 14554 2020-07-08 03:11:00,249 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:11:00,250 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2020-07-08 03:11:00,251 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:11:01,527 INFO [make_forcing_links] forecast2 2020-07-08 forcing file links on arbutus.cloud-nowcast created 2020-07-08 03:11:01,532 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-08 03:11:01,741 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-08'], host='arbutus.cloud-nowcast') 2020-07-08 03:11:02,940 INFO [upload_forcing] forecast2 2020-07-08 forcing files upload to graham-hindcast completed 2020-07-08 03:11:02,947 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-08 03:11:04,118 INFO [run_NEMO] running in process 8148 2020-07-08 03:11:04,119 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:11:04,120 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-08 03:11:04,176 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:11:05,198 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/07jul20forecast2_2020-07-08T031104.264834-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-08 03:11:05,216 INFO [run_NEMO] forecast2 NEMO run for 2020-07-08 on arbutus.cloud-nowcast started 2020-07-08 03:11:05,222 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-08 03:11:05,435 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-08 03:11:07,412 INFO [watch_NEMO] running in process 8413 2020-07-08 03:11:07,413 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:11:07,414 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-08 03:11:07,415 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:11:07,511 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-08 03:11:31,786 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2020-07-07 2020-07-08 03:12:03,085 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2020-07-07 2020-07-08 03:12:17,400 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2020-07-07 2020-07-08 03:12:40,718 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2020-07-07 2020-07-08 03:12:52,900 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2020-07-07 2020-07-08 03:13:03,698 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2020-07-07 2020-07-08 03:13:18,101 INFO [get_onc_ferry] 2020-07-07 ONC TWDP ferry data file created 2020-07-08 03:13:18,106 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2020-07-08 03:13:19,206 INFO [ping_erddap] running in process 14605 2020-07-08 03:13:19,207 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:13:19,207 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-08 03:13:19,208 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:13:19,210 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2020-07-08 03:13:19,212 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-08 03:16:07,612 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7508624 = 2020-07-09 05:08:40 UTC, 17.2% complete 2020-07-08 03:21:07,713 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7509150 = 2020-07-09 10:59:20 UTC, 36.6% complete 2020-07-08 03:26:07,730 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7509655 = 2020-07-09 16:36:00 UTC, 55.4% complete 2020-07-08 03:31:07,831 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7510169 = 2020-07-09 22:18:40 UTC, 74.4% complete 2020-07-08 03:36:07,932 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7510657 = 2020-07-10 03:44:00 UTC, 92.5% complete 2020-07-08 03:41:08,036 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2020-07-08 03:41:08,042 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-08 03:41:08,537 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-08 03:41:08,664 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-08 03:41:08,668 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-07'], host='localhost') 2020-07-08 03:41:10,226 INFO [download_results] running in process 15351 2020-07-08 03:41:10,227 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:41:10,228 INFO [download_results] publishing log messages to tcp://*:5564 2020-07-08 03:41:10,229 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:41:10,230 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/07jul20 to /results/SalishSea/forecast2.201905 2020-07-08 03:41:12,061 INFO [make_ww3_wind_file] running in process 8639 2020-07-08 03:41:12,062 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:41:12,063 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2020-07-08 03:41:12,064 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:41:12,065 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-08 forecast2 run 2020-07-08 03:41:12,098 INFO [make_ww3_current_file] running in process 8641 2020-07-08 03:41:12,099 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:41:12,100 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2020-07-08 03:41:12,101 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:41:12,102 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-08 forecast2 run 2020-07-08 03:41:15,684 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-08 forecast2 run 2020-07-08 03:41:15,689 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2020-07-08 03:43:06,201 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-08 forecast2 run 2020-07-08 03:43:06,206 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2020-07-08 03:43:06,422 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-08'], host='arbutus.cloud-nowcast') 2020-07-08 03:43:08,533 INFO [run_ww3] running in process 8866 2020-07-08 03:43:08,534 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:43:08,535 INFO [run_ww3] publishing log messages to tcp://*:5572 2020-07-08 03:43:08,536 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:43:08,539 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-08T034308.531384-0700 2020-07-08 03:43:08,541 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-08T034308.531384-0700/SoGWW3.sh on arbutus.cloud-nowcast 2020-07-08 03:43:08,555 INFO [run_ww3] forecast2 WaveWatch3 run for 2020-07-08 on arbutus.cloud-nowcast started 2020-07-08 03:43:08,561 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2020-07-08 03:43:08,769 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-08 03:43:10,709 INFO [watch_ww3] running in process 8970 2020-07-08 03:43:10,710 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:43:10,711 INFO [watch_ww3] publishing log messages to tcp://*:5573 2020-07-08 03:43:10,713 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:43:10,747 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2020-07-08 03:46:10,541 INFO [download_results] forecast2 2020-07-07 results files from arbutus.cloud-nowcast downloaded 2020-07-08 03:46:10,548 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-08 03:46:10,757 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2020-07-07'], host='localhost') 2020-07-08 03:46:23,782 INFO [make_CHS_currents_file] running in process 15466 2020-07-08 03:46:23,783 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:46:23,784 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2020-07-08 03:46:23,785 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:48:10,848 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete 2020-07-08 03:48:47,770 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-07 for forecast2 2020-07-08 03:48:47,774 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2020-07-08 03:48:47,958 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2020-07-07'], host='localhost') 2020-07-08 03:48:49,363 INFO [update_forecast_datasets] running in process 15513 2020-07-08 03:48:49,364 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:48:49,365 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2020-07-08 03:48:49,366 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:48:49,367 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/07jul20 run 2020-07-08 03:53:00,524 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/07jul20 run: /results/SalishSea/rolling-forecasts/nemo 2020-07-08 03:53:00,525 INFO [update_forecast_datasets] nemo 2020-07-07 forecast2 rolling forecast datasets updated 2020-07-08 03:53:00,528 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-08 03:53:00,701 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2020-07-08 03:53:00,715 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2020-07-07'], host='localhost') 2020-07-08 03:53:00,848 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2020-07-07'], host='localhost') 2020-07-08 03:53:03,101 INFO [ping_erddap] running in process 15669 2020-07-08 03:53:03,102 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:53:03,103 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-08 03:53:03,107 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:53:03,121 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2020-07-08 03:53:03,123 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-08 03:53:04,917 INFO [make_surface_current_tiles] running in process 15671 2020-07-08 03:53:04,922 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:53:04,923 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564 2020-07-08 03:53:04,924 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:53:04,969 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-07 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/07jul20 2020-07-08 03:53:05,291 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07jul20 2020-07-08 03:53:06,167 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,205 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,239 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,275 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,322 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,359 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,411 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,460 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,515 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,762 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,811 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,852 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,887 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,919 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:06,984 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:07,050 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:07,108 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:07,138 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:07,165 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:07,191 WARNING [make_surface_current_tiles] PDF shrinking failed, proceeding with unshrunk PDF 2020-07-08 03:53:07,193 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07jul20 2020-07-08 03:53:07,193 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-07 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/07jul20 2020-07-08 03:53:07,194 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-07 forecast2 completed 2020-07-08 03:53:07,204 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2020-07-08 03:53:10,949 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete 2020-07-08 03:53:14,434 INFO [make_plots] running in process 15670 2020-07-08 03:53:14,436 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 03:53:14,438 INFO [make_plots] publishing log messages to tcp://*:5564 2020-07-08 03:53:14,439 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-08 03:53:15,870 INFO [make_plots] preparing render list for 2020-07-07 NEMO forecast2 publish figures 2020-07-08 03:53:15,946 INFO [make_plots] added Website_thumbnail figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,948 INFO [make_plots] added Threshold_website figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,949 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,951 INFO [make_plots] added Boundary Bay figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,952 INFO [make_plots] added Campbell River figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,953 INFO [make_plots] added Cherry Point figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,954 INFO [make_plots] added Friday Harbor figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,955 INFO [make_plots] added Halfmoon Bay figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,957 INFO [make_plots] added Nanaimo figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,958 INFO [make_plots] added Neah Bay figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,959 INFO [make_plots] added New Westminster figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,961 INFO [make_plots] added Patricia Bay figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,962 INFO [make_plots] added Point Atkinson figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,963 INFO [make_plots] added Port Renfrew figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,964 INFO [make_plots] added Sand Heads figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,966 INFO [make_plots] added Sandy Cove figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,967 INFO [make_plots] added Squamish figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,968 INFO [make_plots] added Victoria figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,969 INFO [make_plots] added Woodwards Landing figure to 2020-07-07 NEMO forecast2 publish render list 2020-07-08 03:53:15,970 INFO [make_plots] starting to render nemo forecast2 publish 07jul20 figures 2020-07-08 03:53:25,093 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-08 03:53:25,094 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-08 03:58:11,050 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete 2020-07-08 03:58:29,017 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:01:01,410 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:01:20,586 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:01:39,698 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:01:59,068 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:03:11,154 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 423, 78.3% complete 2020-07-08 04:04:32,585 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:04:49,082 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:07:14,167 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:08:11,252 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 531, 98.3% complete 2020-07-08 04:09:37,441 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:09:56,404 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:12:20,963 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:13:11,354 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2020-07-08 04:13:11,358 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2020-07-08 04:13:12,029 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-08'], host='localhost') 2020-07-08 04:13:16,971 INFO [download_wwatch3_results] running in process 16223 2020-07-08 04:13:16,972 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:13:16,974 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567 2020-07-08 04:13:16,976 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:14:54,210 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:14:58,547 INFO [download_wwatch3_results] forecast2 2020-07-08 results files from arbutus.cloud-nowcast downloaded 2020-07-08 04:14:58,553 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2020-07-08 04:14:58,774 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2020-07-08'], host='localhost') 2020-07-08 04:15:00,601 INFO [update_forecast_datasets] running in process 16255 2020-07-08 04:15:00,603 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:15:00,604 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567 2020-07-08 04:15:00,606 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:15:00,607 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/08jul20 run 2020-07-08 04:15:00,611 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/08jul20 run 2020-07-08 04:15:00,612 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/08jul20 run 2020-07-08 04:15:27,084 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/08jul20 run: /results/SalishSea/rolling-forecasts/wwatch3 2020-07-08 04:15:27,085 INFO [update_forecast_datasets] wwatch3 2020-07-08 forecast2 rolling forecast datasets updated 2020-07-08 04:15:27,090 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-08 04:15:27,277 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2020-07-08 04:15:28,818 INFO [ping_erddap] running in process 16283 2020-07-08 04:15:28,820 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:15:28,821 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-08 04:15:28,823 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:15:28,825 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2020-07-08 04:15:28,828 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-08 04:15:29,026 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-08'], host='localhost') 2020-07-08 04:15:45,149 INFO [make_plots] running in process 16287 2020-07-08 04:15:45,150 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:15:45,152 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-08 04:15:45,154 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:15:45,156 INFO [make_plots] preparing render list for 2020-07-08 WaveWatch3 forecast2 publish figures 2020-07-08 04:15:45,157 INFO [make_plots] added Halibut Bank figure to 2020-07-08 WaveWatch3 forecast2 publish render list 2020-07-08 04:15:45,159 INFO [make_plots] added Sentry Shoal figure to 2020-07-08 WaveWatch3 forecast2 publish render list 2020-07-08 04:15:45,160 INFO [make_plots] starting to render wwatch3 forecast2 publish 08jul20 figures 2020-07-08 04:16:31,031 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-08 04:16:31,033 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-08 04:16:31,044 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2020-07-08 04:17:16,564 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2020-07-08 04:17:17,639 INFO [make_plots] finished rendering wwatch3 forecast2 publish 08jul20 figures 2020-07-08 04:17:17,640 INFO [make_plots] wwatch3 publish plots for 2020-07-08 forecast2 completed 2020-07-08 04:17:17,645 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-08 04:17:30,709 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:17:49,733 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:20:31,976 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-03 00:00:00 to 2020-07-10 05:55:00 2020-07-08 04:23:00,465 INFO [make_plots] finished rendering nemo forecast2 publish 07jul20 figures 2020-07-08 04:23:00,467 INFO [make_plots] nemo publish plots for 2020-07-07 forecast2 completed 2020-07-08 04:23:00,478 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-08 04:23:00,625 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2020-07-07'], host='localhost') 2020-07-08 04:23:14,439 INFO [make_feeds] running in process 16457 2020-07-08 04:23:14,440 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:23:14,441 INFO [make_feeds] publishing log messages to tcp://*:5564 2020-07-08 04:23:14,656 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:23:23,562 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-08 04:23:23,564 INFO [make_feeds] NumExpr defaulting to 8 threads. 2020-07-08 04:23:30,381 INFO [make_feeds] ATOM feeds for 2020-07-07 forecast2 run completed 2020-07-08 04:23:30,385 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2020-07-08 04:23:30,628 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2020-07-08 04:23:32,650 INFO [clear_checklist] running in process 16471 2020-07-08 04:23:32,652 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:23:32,653 INFO [clear_checklist] publishing log messages to tcp://*:5564 2020-07-08 04:23:32,654 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:23:32,656 INFO [clear_checklist] requesting that manager clear system state checklist 2020-07-08 04:23:32,658 INFO [manager] writing checklist to log file 2020-07-08 04:23:32,697 INFO [manager] checklist cleared 2020-07-08 04:23:32,701 INFO [clear_checklist] nowcast system checklist cleared 2020-07-08 04:23:32,704 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2020-07-08 04:23:34,131 INFO [rotate_logs] running in process 16475 2020-07-08 04:23:34,132 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-08 04:23:34,133 INFO [rotate_logs] publishing log messages to tcp://*:5564 2020-07-08 04:23:34,134 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2020-07-08 04:23:34,135 INFO [rotate_logs] rotating log files