Skip to content

Borg backup cannot connect to borgbase backup service

According to the borgbase.com web frontend, our new Borg backup job isn't working: The web frontend did not show any successful backups.

My research using an interactive SSH session on storage001.privatestorage-staging.com showed that Borg backup can't connect to the backup service since host key checking fails:

[root@storage001:~]# journalctl -u borgbackup-job-daily.service

Jun 08 11:22:26 storage001 systemd[1]: Started BorgBackup job daily.
Jun 08 11:22:27 storage001 borgbackup-job-daily-start[396754]: Remote: Host key verification failed.
Jun 08 11:22:27 storage001 borgbackup-job-daily-start[396754]: Connection closed by remote host. Is borg working on the server?

It seems the OpenSSH default behaviour of having StrictHostKeyChecking=ask won't let us connect non-interactively to a new host:

man ssh_config

       StrictHostKeyChecking
              If this flag is set to yes, ssh(1) will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to  connect
              to hosts whose host key has changed.  This provides maximum protection against man-in-the-middle (MITM) attacks, though it can be
              annoying when the /etc/ssh/ssh_known_hosts file is poorly maintained or when connections to new hosts are frequently made.   This
              option forces the user to manually add all new hosts.

              If this flag is set to accept-new then ssh will automatically add new host keys to the user's known_hosts file, but will not per‐
              mit connections to hosts with changed host keys.  If this flag is set to no or off, ssh will automatically add new host  keys  to
              the  user  known  hosts  files and allow connections to hosts with changed hostkeys to proceed, subject to some restrictions.  If
              this flag is set to ask (the default), new host keys will be added to the user known host files only after the user has confirmed
              that  is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed.  The host keys of known
              hosts will be verified automatically in all cases.

We could distribute ssh fingerprints to make the backup server hosts known to our storage nodes, or change the configuration to allow a new host and only check when the fingerprint diverges.

See also this NixOps issue where it is explained that, wich StrictHostKeyChecking=no, ...

My understanding was that it will refuse to connect whenever an existing host key is different from that in known hosts.

However, this turns out to be only true for keyboard interactive or password authentication and if we're using pubkey auth, OpenSSH will happily connect.

Now the real fix for this (already deploying with a pre-generated host key) is a bit more involved, but we can mitigate this for now, because since OpenSSH 7.5 there is the "accept-new" option to StrictHostKeyChecking, which does exactly what I thought "no" would do:

If this flag is set to ``accept-new'' then ssh will automatically add new host keys to the user known hosts files, but will not permit connections to hosts with changed host keys.

Edited by Florian Sesser