2024-05-04 05:11:06,864 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-05-04 05:11:06,865 DEBUG [rotate_logs] new /SalishSeaCast/logs/nowcast/nowcast.debug.log log file permissions set to rw-rw-r--
2024-05-04 05:11:06,870 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-05-04 05:11:06,871 DEBUG [rotate_logs] new /SalishSeaCast/logs/nowcast/checklist.log log file permissions set to rw-rw-r--
2024-05-04 05:11:06,871 INFO [rotate_logs] log files rotated
2024-05-04 05:11:06,873 DEBUG [rotate_logs] sent message: (success) log files rotated
2024-05-04 05:11:06,875 DEBUG [manager] received message from rotate_logs: (success) log files rotated
2024-05-04 05:11:06,877 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-05-04 05:11:06,879 DEBUG [manager] listening...
2024-05-04 05:11:06,880 DEBUG [rotate_logs] received message from manager: (ack) message acknowledged
2024-05-04 05:11:06,880 DEBUG [rotate_logs] shutting down
2024-05-04 05:24:57,394 DEBUG [manager] received message from watch_NEMO_hindcast: (success) NEMO hindcast run ended
2024-05-04 05:24:57,394 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-04 05:24:57,399 DEBUG [manager] Starting new HTTPS connection (1): hooks.slack.com:443
2024-05-04 05:24:57,581 DEBUG [manager] https://hooks.slack.com:443 "POST /services/TFR25L4LU/BN6BE538V/CoaynXHafLaNF0L2VBTSqXFT HTTP/1.1" 200 6
2024-05-04 05:24:57,586 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-26'], host='localhost')
2024-05-04 05:24:57,587 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.download_results', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', 'optimum-hindcast', 'hindcast', '--run-date', '2019-07-26']
2024-05-04 05:24:57,592 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-04 05:24:57,593 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.watch_NEMO_hindcast', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', 'optimum-hindcast']
2024-05-04 05:24:57,597 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-04 05:24:57,598 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.run_NEMO_hindcast', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', 'optimum-hindcast']
2024-05-04 05:24:57,601 DEBUG [manager] listening...
2024-05-04 05:24:58,522 INFO [download_results] running in process 2295729
2024-05-04 05:24:58,524 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-04 05:24:58,525 INFO [download_results] publishing log messages to tcp://*:5563
2024-05-04 05:24:58,526 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-04 05:24:58,527 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jul19 to /ocean/sallen/timbucktwo/oxygen
2024-05-04 05:24:58,529 DEBUG [download_results] running command in subprocess: ['scp', '-pr', 'optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jul19', '/ocean/sallen/timbucktwo/oxygen']
2024-05-04 05:25:02,793 DEBUG [manager] received message from run_NEMO_hindcast: (success) NEMO hindcast run queued
2024-05-04 05:25:02,793 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-04 05:25:02,797 DEBUG [manager] listening...
2024-05-04 05:25:43,578 INFO [download_results] hindcast 2019-07-26 results files from optimum-hindcast downloaded
2024-05-04 05:25:43,582 DEBUG [download_results] sent message: (success hindcast) hindcast results files downloaded
2024-05-04 05:25:43,588 DEBUG [manager] received message from download_results: (success hindcast) hindcast results files downloaded
2024-05-04 05:25:43,589 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-04 05:25:43,597 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-26'], host='localhost')
2024-05-04 05:25:43,598 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.split_results', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', 'hindcast', '2019-07-26']
2024-05-04 05:25:43,598 DEBUG [download_results] received message from manager: (ack) message acknowledged
2024-05-04 05:25:43,599 DEBUG [download_results] shutting down
2024-05-04 05:25:43,601 DEBUG [manager] listening...
2024-05-04 05:25:44,561 DEBUG [manager] received message from split_results: (success hindcast) hindcast results files split into daily directories
2024-05-04 05:25:44,561 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-04 05:25:44,571 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-jul', 'robot.graham'], host='localhost')
2024-05-04 05:25:44,572 DEBUG [manager] cmd = ['/SalishSeaCast/nowcast-env/bin/python3', '-m', 'nowcast.workers.archive_tarball', '/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml', 'hindcast', '2019-jul', 'robot.graham']
2024-05-04 05:25:44,575 DEBUG [manager] listening...
2024-05-04 05:25:45,360 INFO [archive_tarball] running in process 2295775
2024-05-04 05:25:45,362 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-04 05:25:45,363 INFO [archive_tarball] publishing log messages to tcp://*:5563
2024-05-04 05:25:45,364 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-05-04 05:25:45,366 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jul19.tar from /ocean/sallen/timbucktwo/oxygen/*jul19/
2024-05-04 05:25:45,482 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/01jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:25:54,976 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/02jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:25:57,600 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/03jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:00,135 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/04jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:02,719 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/05jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:12,228 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/06jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:18,397 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/07jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:22,070 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/08jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:48,172 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/09jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:26:55,585 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/10jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:27:05,405 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/11jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:27:08,970 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/12jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:27:11,518 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/13jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:27:14,056 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/14jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:27:16,548 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/15jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:27:57,491 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/16jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:01,015 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/17jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:03,521 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/18jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:06,107 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/19jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:08,714 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/20jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:48,330 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/21jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:55,231 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/22jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:28:59,146 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/23jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:07,071 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/24jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:15,082 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/25jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:40,630 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/26jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:49,071 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/27jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:51,716 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/28jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:54,346 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/29jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:56,922 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/30jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:29:59,445 DEBUG [archive_tarball] adding /ocean/sallen/timbucktwo/oxygen/31jul19/ to /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:30:28,645 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jul19.index from /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:30:28,729 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jul19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-05-04 05:30:28,730 DEBUG [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jul19.tar to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-05-04 05:48:30,786 DEBUG [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jul19.index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-05-04 05:48:59,743 DEBUG [archive_tarball] deleting /ocean/dlatorne/oxygen-jul19.tar
2024-05-04 05:49:00,779 DEBUG [archive_tarball] deleting /ocean/dlatorne/oxygen-jul19.index
2024-05-04 05:49:00,780 INFO [archive_tarball] hindcast *jul19 results files archived to robot.graham
2024-05-04 05:49:00,781 DEBUG [archive_tarball] sent message: (success hindcast) nowcast results tarball archived
2024-05-04 05:49:00,782 DEBUG [manager] received message from archive_tarball: (success hindcast) nowcast results tarball archived
2024-05-04 05:49:00,783 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-05-04 05:49:00,792 DEBUG [manager] listening...
2024-05-04 05:49:00,793 DEBUG [archive_tarball] received message from manager: (ack) message acknowledged
2024-05-04 05:49:00,794 DEBUG [archive_tarball] shutting down