Server Backup With Rsync

Reading time ~4 minutes

Server Backup With Rsync

Server Backup With Rsync

1 Introduction

I manage the cloud server and the web applications of an organization. These applications are mostly dockerized and installed by docker-scripts. For robustness, I would like other people from the organization to be able to keep a backup of whatever is installed on the server, but without being able to restore or modify anything on the server (they can restore it on a new server, if needed). Using rsync is one of the obvious choices for making such a backup, but it also needs a couple of tricks for working properly, the way that I want it. In this blog I describe this setup.

First of all we need to make sure that rsync is installed both on the client and on the server:

apt install rsync

2 Create a backup user with ssh-key access

  1. Create a backup user, for example backup1 (in ubuntu there is already a user named backup):

    useradd backup1 -m
    ls -al /home/backup1/
    
  2. Create a ssh key-pair for this user:

    ssh-keygen -t ecdsa -P '' -q -f key1
    
    ls -l key1*
    cat key1
    cat key1.pub
    
  3. Add the public key to /home/backup1/.ssh/authorized_keys:

    mkdir -p /home/backup1/.ssh
    chown backup1: /home/backup1/.ssh
    chmod 700 /home/backup1/.ssh
    
    cat key1.pub >> /home/backup1/.ssh/authorized_keys
    chown backup1: /home/backup1/.ssh/authorized_keys
    chmod 600 /home/backup1/.ssh/authorized_keys
    
    ls -al /home/backup1/.ssh/
    cat /home/backup1/.ssh/authorized_keys
    
  4. Try to login with this key:

    ssh -p 22 -i key1 backup1@localhost
    

    You should be able to login without a password.

  5. Try to copy something:

    mkdir -p /home/backup1/test1
    touch /home/backup1/test1/file1.txt
    touch /home/backup1/test1/file2.txt
    ls -al /home/backup1/test1
    
    rsync -a -e "ssh -p 22 -i key1" backup1@localhost:~/test1 .
    
    ls -al test1
    

3 Restrict the ssh key of the backup user for using only rsync

  1. Let's find out the command that the client is sending to the server through SSH. Let's try the same rsync command again, with the added SSH switch -v (verbose):

    rsync -a -e "ssh -p 22 -i key1 -v" backup1@localhost:~/test1 .
    

    Then let's look for the debug line that says "Sending command":

    rsync -a -e "ssh -p 22 -i key1 -v" backup1@localhost:~/test1 . 2>&1 \
        | grep "Sending command"
    

    It should be something like this:

    rsync --server --sender -logDtpre.iLsfxC . ~/test1
    
  2. We can restrict the SSH key key1 to execute only this command and nothing else. For this we need to add something like this before the public key on /home/backup1/.ssh/authorized_keys:

    command="rsync --server --sender -logDtpre.iLsfxC . ~/test1" ecdsa-sha2-nistp256 AAAAE2Vj....
    

    To make it even more secure, we can also add the options no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding. The file /home/backup1/.ssh/authorized_keys now should look like this:

    command="rsync --server --sender -logDtpre.iLsfxC . ~/test1",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbMdR9uW4SMeinpVvr6UQZaFybkiVZxm2DRYxFlCuxHchpTMGR7U4gZGZwY4D5LQDDy1Py4TWSsEizda4LecgQ= root@server
    
  3. Let's check that now we cannot login with key1 anymore, but we can still use it to rsync:

    ssh -p 22 -i key1 backup1@localhost  # should fail
    
    rm -rf test1
    rsync -a -e "ssh -p 22 -i key1" backup1@localhost:~/test1 .
    ls -l test1
    
    rm -rf test1
    rsync -a -e "ssh -p 22 -i key1" backup1@localhost: .
    ls -l test1
    

4 Create a read-only view of the parts of the filesystem that need to be backed up

  1. Install bindfs:

    apt list bindfs
    apt show bindfs
    apt install bindfs
    
  2. Create mount directories:

    mkdir -p /mnt/backup-server/scripts
    mkdir -p /mnt/backup-server/apps
    
  3. Add these lines to /etc/fstab for mounting directories read-only:

    /opt/docker-scripts /mnt/backup-server/scripts fuse.bindfs perms=0000:u=rD,force-user=backup1,force-group=nogroup 0 0
    /var/ds             /mnt/backup-server/apps    fuse.bindfs perms=0000:u=rD,force-user=backup1,force-group=nogroup 0 0
    

    Since we are using docker-scripts for installing and managing apps, these two directories are what we need to backup: /opt/docker-scripts and /var/ds.

  4. Mount them:

    mount -a
    ls -al /mnt/backup-server/scripts
    ls -al /mnt/backup-server/apps
    
  5. Test that they are read-only:

    sudo -u backup1 ls -al /mnt/backup-server/scripts
    sudo -u backup1 touch /mnt/backup-server/scripts/test1.txt
    

5 Create and use a backup script

  1. For convenience, we can combine the command and the key in a bash script named backup-server.sh that looks like this:

    #!/bin/bash
    
    server=127.0.0.1
    port=22
    
    keyfile=$(mktemp)
    sed -n -e '/^----/,/^-----/p' $0 > $keyfile
    
    cd $(dirname $0)
    rsync -a -e "ssh -p $port -i $keyfile" backup1@${server}: .
    
    rm -f $keyfile
    exit 0
    
    -----BEGIN OPENSSH PRIVATE KEY-----
    MHcCAQEEIGHMv+mlPZg/V6TmLADsfZcpMRcfxykJGzXAF65tuQ4AoAoGCCqGSM49
    AwEHoUQDQgAExsx1H25bhIx6KelW+vpRBloXJuSJVnGbYNFjEWUK7EdyGlMwZHtT
    iBkZnBjgPktAMPLU/LhNZKwSLN1rgt5yBA==
    -----END OPENSSH PRIVATE KEY-----
    

    Let's try it:

    chmod 700 backup-server.sh
    rm -rf test1/
    ./backup-server.sh
    ls -l test1/
    
  2. Now we can move this script to the client (backup server), making sure to set the proper values for the variables server and port, and it should work.
  3. Let's also fix the directory on the server that is being backed up. We should edit /home/backup1/.ssh/authorized_keys and change ~/test1 to /mnt/backup-server
  4. On the client (computer that is receiving the backup), let's place the script backup-server.sh on a directory like /var/backup:

    mkdir -p /var/backup
    mv backup-server.sh /var/backup/
    
    cd /var/backup/
    ./backup-server.sh
    
  5. Let's also create a cron job that runs this script periodically each week:

    cat <<EOF > /etc/cron.d/backup-server
    # backup the server each tuesday
    0 0 * * TUE  root  /var/backup/backup-server.sh
    EOF
    

Date: 2020-08-10

Author: Dashamir Hoxha

Created: 2020-09-11 Fri 16:49

Validate

OpenPGP Web Key Directory

OpenPGP Web Key DirectoryOpenPGP Web Key DirectoryTable of Contents1. Introduction2. How WKD works3. Building a WKD3.1. Create the direct...… Continue reading

SMTP Server with LDAP Authentication

Published on April 17, 2021

Using WireGuard VPN

Published on November 09, 2020