Created: July 24th 2024
Last updated: August 13th 2024
Categories: Linux
Author: Marcus Fleuti

Ultimate Linux Full System Backup: Secure encrypted Live Image Backups with free OpenSource Software and Backup Monitoring

Donate with

82uymVXLkvVbB4c4JpTd1tYm1yj1cKPKR2wqmw3XF8YXKTmY7JrTriP4pVwp2EJYBnCFdXhLq4zfFA6ic7VAWCFX5wfQbCC

Secure Live Linux System backups without rebooting - Open Source Free Software - E-Mail Monitoring - Encrypted Backups

In the world of Linux system administration, ensuring data integrity and system availability is paramount. Our open-source backup script addresses these critical needs by offering a powerful combination of features that set it apart from traditional backup methods. And it's all free! 😉

At the heart of this solution is the ability to create full partition backups of your Linux system while it's running, eliminating the need for disruptive reboots. This live backup capability ensures that your services remain available and productive, even during the backup process.

Security is a top priority in our backup strategy. The script implements strong encryption for your backups, safeguarding your sensitive data from unauthorized access. This encryption, coupled with best practices for secure storage, provides a robust defense against data breaches and ensures the confidentiality of your backups.

To keep you informed about the status of your backups, we've integrated a comprehensive email monitoring system. This feature provides real-time notifications about backup successes, failures, and any potential issues that may require your attention. With this proactive monitoring, you can rest assured that your backup processes are functioning as expected and quickly address any problems that arise.

Furthermore, this script is capable of creating on-the-fly backups of LUKS (Linux Unified Key Setup) encrypted volumes, ensuring that even your most sensitive encrypted data can be backed up securely without compromising its protection.

One of the standout features of this backup tool is its ability to restore backups on volumes smaller than the original source volume. This flexibility allows for easier migration to smaller drives or partitions, making it an invaluable tool for system upgrades and storage management.

Best of all, this powerful backup solution is available as free, open-source software. This not only makes it accessible to all Linux users but also allows for community-driven improvements and customizations to meet specific backup needs.

Let's dive into the details of this robust, secure, and efficient Linux backup solution that keeps your systems protected without interrupting your operations.

For Linux system administrators and power users, finding a reliable and efficient full system backup solution can be challenging. Many seek a method to create complete system image backups while the system is running, avoiding the need for downtime or reboots. In this comprehensive guide, we'll explore a powerful FSArchiver script that allows you to perform live full system backups on Linux, addressing this common pain point.

Comparison with Other Backup Solutions

Feature	This Script	dd	Clonezilla
Live Backup	Yes	Limited	No
Compression	Yes	No	Yes
Encryption	Yes	No	Yes
Email Notifications	Yes	No	No
Selective Backup	Yes	No	Yes
LUKS Volume Backup	Yes	Limited	Yes
Restore to Smaller Volumes	Yes	No	Limited

Overview: Key Features of the FSArchiver Backup Script

Live system backups without rebooting
Full disk image creation while the computer is running
Open-source and free to use
Customizable for different partitions and drives
Email notifications for backup status
Secure encrypted backups for enhanced security

Understanding the FSArchiver Backup Script

Visit our GitHub repository for the latest version of the script:

Go to GitHub Repository

Here's the complete script:

#!/bin/bash

# User-defined backup configurations
# Format: BACKUP_PARAMETERS["Backup Name"]="Backup File Path:Mount Point or Device Path you want to backup"
declare -A BACKUP_PARAMETERS
BACKUP_PARAMETERS["EFI"]="/backup/taget/backup-efi.fsa:/boot/efi"
BACKUP_PARAMETERS["System"]="/backup/taget/backup-root.fsa:/"
BACKUP_PARAMETERS["DATA"]="/backup/taget/backup-DATA.fsa:/media/username/DATA"

# Backup log file location
BACKUP_LOG="/var/log/fsarchiver-bkp.log"

# Password file location
# In order to increase security, this file should be located on an encrypted volume with root-only access privileges
PASSWORD_FILE="/path/to/encrypted/volume/.bkp-password-file.sec"

# E-Mail configuration for notifications
# set the user used to send out the e-mail. The script uses ssmtp for mail sending. Ensure that ssmtp is installed (apt install ssmtp)
# Remember to edit /etc/ssmpt/ssmtp.conf and set the following options:
## mailhub=your-mailserver.tld:587
## hostname=your-desired-hostname
## FromLineOverride=YES                    ### This is important so that the script can set the proper sender name
## UseSTARTTLS=YES                         ### This is the default for mail dispatch via Port 587. SMTP/SSL via port 465 usually requires this to be disabled and UseTLS=yes to be set instead
## UseTLS=NO
## AuthUser=your-myusername@your-domain.tld
## AuthPass=your-email-account-password
MAIL_FROM="My e-mail sender name<myname@domain.tld>"
MAIL_TO="myname@domain.tld"
MAIL_SUBJECT_ERROR="[ERROR] FSARCHIVER Backup failed. Please check the logs." 		# Enter the default backup error message
MAIL_SUBJECT_SUCCESS="[SUCCESS] FSARCHIVER Backup Successful" 						# Enter the default backup success message

# Set paths to exclude from backup - specifically paths which contain other mountpoints (other volumes mounted in a directory of another volume - these should be backed up separately)
# If you are excluding paths with spaces you will need to escape those spaces. Example: "/opt/my\ exclude\ path"
EXCLUDE_PATHS=(
    "/home/username/Cryptomator-mountpoints"
    "/home/username/Steamlibrary"
    "/home/username/.steam"
    "/var/log/*"
    "/media"
    "/mnt"
    "/tmp/*"
    "/swapfile"
)

# ZSTD compression level (0-22)
# 0 = no compression, 1 = fastest/worst compression, 22 = slowest/best compression
# Default is 3. Values above 19 are considered "ultra" settings and should be used with caution.
ZSTD_COMPRESSION_VALUE=15

# Check if the script is run as root
if [[ $EUID -ne 0 ]]; then
   echo "The script must be executed as root. Exiting..."
   exit 1
fi

# Function to get the device path for a given mount point
get_device_path() {
    local mount_point="$1"
    
    # Check if findmnt command is available
    if ! command -v findmnt &> /dev/null
    then
        echo "Error: findmnt command not found. This script requires the findmnt command to function properly." &>2
        echo "You can install it by running: sudo apt update && sudo apt install util-linux" &>2
        return 1
    fi
    
    findmnt -no SOURCE "$mount_point"
}

ERROR=0
ERROR_MSG=""

# Process backup parameters
for name in "${!BACKUP_PARAMETERS[@]}"; do
    IFS=':' read -r backup_file source <<< "${BACKUP_PARAMETERS[$name]}"
    if [[ $source == /dev/* ]]; then
        device=$source
    else
        if [ ! -d "$source" ]; then
            ERROR=1
            ERROR_MSG+="Mount point $source does not exist or is not accessible\n"
            echo "Error: Mount point $source does not exist or is not accessible" &>2
            continue
        fi
        
        device=$(get_device_path "$source")
        if [ -z "$device" ]; then
            ERROR=1
            ERROR_MSG+="Failed to get device path for $source\n"
            echo "Error: Failed to get device path for $source" &>2
            continue
        fi
    fi
    
    BACKUP_PARAMETERS[$name]="$backup_file:$device"
    echo "Configured backup: $name - File: $backup_file, Device: $device"
done

if [ "$ERROR" -eq 1 ]; then
    echo "Errors occurred during configuration:" &>2
    echo -e "$ERROR_MSG" &>2
    exit 1
fi

# Get the archive password from an external file which can only be accessed by root. Cut eventual existing new lines.
if [ ! -f "$PASSWORD_FILE" ]; then
    echo "Error: Password file $PASSWORD_FILE not found." &>2
    ERROR=1
    ERROR_MSG+="Password file $PASSWORD_FILE not found.\n"
    exit 1
fi

if [ ! -r "$PASSWORD_FILE" ]; then
    echo "Error: Password file $PASSWORD_FILE is not readable." &>2
    ERROR=1
    ERROR_MSG+="Password file $PASSWORD_FILE is not readable.\n"
    exit 1
fi

FSPASS=$(cat "$PASSWORD_FILE" | tr -d '\n')

if [ -z "$FSPASS" ]; then
    echo "Error: Password file $PASSWORD_FILE is empty." &>2
    ERROR=1
    ERROR_MSG+="Password file $PASSWORD_FILE is empty.\n"
    exit 1
fi

export FSPASS

ERROR=0
ERROR_MSG=""

do_backup() {
    local backup_file="$1"
    local device="$2"
    
    echo "Backing up device: $device" >> $BACKUP_LOG
    ls -l "$device" >> $BACKUP_LOG
    lsblk "$device" >> $BACKUP_LOG
    
    fsarchiver ${EXCLUDE_STATEMENTS} -o -v -A -j$(nproc) -Z$ZSTD_COMPRESSION_VALUE -c "${FSPASS}" savefs "$backup_file" "$device" 2&>1 | tee -a $BACKUP_LOG
    
    check_backup_errors "$device"
}

check_backup_errors() {
    local BKP_SOURCE="$1"

    # Ensure the BACKUP_LOG variable is available - just to be 100% sure that the log was not removed in the meantime...
    if [ -z "$BACKUP_LOG" ]; then
        ERROR_MSG+="[ $BACKUP_LOG ] is empty after backing up [ $BKP_SOURCE ]. Something is not OK. Please check the logs and the backup process as a whole."
        return 1
    fi

    local LOG_OUTPUT
    LOG_OUTPUT=$(tail -n 5 "$BACKUP_LOG" | egrep -i "(files with errors)|\b(cannot|warning|error|errno|Errors detected)\b")

    if  [[ ${LOG_OUTPUT,,} =~ (^|[[:space:]])("cannot"|"warning"|"error"|"errno"|"Errors detected")([[:space:]]|$) ]]; then
        ERROR=1
        ERROR_MSG+="Errors detected in backup for [ $BKP_SOURCE ]:\n$LOG_OUTPUT\n"
    elif [[ $LOG_OUTPUT =~ regfiles=([0-9]+),\ directories=([0-9]+),\ symlinks=([0-9]+),\ hardlinks=([0-9]+),\ specials=([0-9]+) ]]; then
        for val in "${BASH_REMATCH[@]:1}"; do
            if [ "$val" -ne 0 ]; then
                ERROR=1
                ERROR_MSG+="Errors detected in backup for [ $BKP_SOURCE ]:\n$LOG_OUTPUT\n"
                break
            fi
        done
    fi
}

# Now generate fsarchiver exclude statements from the exclude paths..
EXCLUDE_STATEMENTS=""

for path in "${EXCLUDE_PATHS[@]}"; do
  EXCLUDE_STATEMENTS+="--exclude=$path "
done

TIME_START=$(date +"%s")
MAIL_BODY=""
MAIL_BODY="${MAIL_BODY}Backup start: $(date +%d.%B.%Y,%T)."

if [[ -e $BACKUP_LOG ]]; then
    rm -f $BACKUP_LOG
    touch $BACKUP_LOG
fi

# Execute the backup job by looping through the associative array
for KEY in $(echo "${!BACKUP_PARAMETERS[@]}" | tr ' ' '\n' | sort -n); do
    IFS=':' read -r BKP_IMAGE_FILE SOURCE_DEVICE <<< "${BACKUP_PARAMETERS[$KEY]}"
    do_backup "$BKP_IMAGE_FILE" "$SOURCE_DEVICE"
done

# Send E-Mail on error or success
TIME_DIFF=$(($(date +"%s")-${TIME_START}))
if [ "$ERROR" -eq 1 ]; then
    MAIL_BODY="${MAIL_BODY}\n\n"
    MAIL_BODY="${MAIL_BODY}Backup end: $(date +%d.%B.%Y,%T).\n"
    MAIL_BODY="${MAIL_BODY}Runtime: $((${TIME_DIFF} / 60)) Minutes and $((${TIME_DIFF} % 60)) Seconds.\n\n"
    MAIL_BODY="${MAIL_BODY}ERROR REPORT:\n"
    MAIL_BODY="${MAIL_BODY}$ERROR_MSG"

    # Send E-Mail...
    echo -e "From: $MAIL_FROM\nSubject: $MAIL_SUBJECT_ERROR\nTo: $MAIL_TO\n\n$MAIL_BODY" | ssmtp -t
else
    #Send E-Mail...
    echo -e "From: $MAIL_FROM\nSubject: $MAIL_SUBJECT_SUCCESS\nTo: $MAIL_TO\n\nBackup finished on: $(date +%d.%B.%Y,%T)\nRuntime: $((${TIME_DIFF} / 60)) Minutes and $((${TIME_DIFF} % 60)) Seconds." | ssmtp -t
fi

Key Components Explained

1. Root Privileges Check

The script ensures it's run with root privileges, which are necessary for accessing all system files and partitions.

2. Backup Configuration

An associative array is used to define backup parameters, allowing easy addition or modification of backup targets.

3. Encryption Password

For security, the script reads the encryption password from a separate file accessible only by root.

4. Email Notification Configuration

SSMTP is used to send email notifications about backup status.

5. Exclusion Paths

You can specify paths to exclude from the backup, such as temporary files or mounted volumes.

6. Backup Function

The core backup functionality uses FSArchiver with various options for compression, multithreading, and encryption.

7. Error Checking

A thorough error checking mechanism is implemented to detect and report any issues during the backup process.

Installing and Configuring SSMTP

To enable email notifications, you'll need to install and configure SSMTP:

1. Installation

sudo apt install ssmtp

2. Configuration

sudo nano /etc/ssmtp/ssmtp.conf

Add or modify the following lines:

mailhub=your-mailserver.tld:587
hostname=your-desired-hostname
FromLineOverride=YES
UseSTARTTLS=YES
UseTLS=NO
AuthUser=your-username@your-domain.tld
AuthPass=your-email-account-password

Note: If you're using SSL Port 465, you may need to use UseTLS=YES instead of UseSTARTTLS=YESAdjust these settings according to your email provider's requirements.

Test the SSMTP configuration

Use this code to quickly and easily test your SSMTP configuration on the console. If it does not work, check out your syslog.

SSMTP Test Syntax

Copy/paste the follwoing code on your terminal, change the MAIL_FROM and MAIL_TO e-mail addresses and execute it. SSMTP should send the e-mail:

MAIL_FROM="sender@example.com" MAIL_TO="recipient@example.com" && echo -e "From: $MAIL_FROM\nTo: $MAIL_TO\nSubject: SSMTP Test Email\n\nThis is a test email sent using ssmtp." | ssmtp -t

Monitor the Syslog

If you did not receive any e-mail, check the syslog. On most system it's either:

tail -F /var/log/syslog

tail -F /var/log/messages

Implementing the Backup Script

Copy the entire script into a new file, e.g.:fsarchiver_backup.sh
Make the script executable: chmod +x fsarchiver_backup.sh
Customize the backup parameters, email settings, and exclusion paths as needed.
Create the password file: sudo nano /root/bkp-password.sec and enter your desired encryption password.
Run the script as root: sudo ./fsarchiver_backup.sh

Scheduling Automated Backups

To schedule regular backups, you can use cron jobs. Here's how to set up a weekly backup:

Open the crontab file for editing: sudo crontab -e
Add the following line to run the backup every Sunday at 2 AM:
```
0 2 * * 0 /path/to/fsarchiver_backup.sh
```
Save and exit the editor.

Script Dependencies

This backup script relies on several system utilities to function correctly. One key dependency is the findmnt command, which is used to determine the device paths for mounted volumes. The script now includes a check for this command and will provide instructions if it's not found.

If you run the script and see an error message about findmnt not being found, you can install it using the following commands:

sudo apt update
sudo apt install util-linux

The util-linux package includes findmnt along with other useful system utilities. After installing, you should be able to run the backup script successfully.

Handling Encrypted LUKS Volumes

Our backup script is designed to work seamlessly with LUKS (Linux Unified Key Setup) encrypted volumes, providing an additional layer of security for your sensitive data. The script can back up LUKS volumes without requiring any special configuration, as long as the volumes are properly mounted. Here's how to ensure your LUKS volumes are backed up correctly:

LUKS Configuration: Ensure that your LUKS volumes are properly set up and mounted before running the backup script.
Mount Point: Make note of the mount point where your LUKS volume is mounted. This is typically something like /media/username/VolumeName or a custom mount point you've set up.
Backup Parameters: Add an entry to the BACKUP_PARAMETERS associative array for your LUKS volume, using the mount point:
```
BACKUP_PARAMETERS["LUKS_Volume"]="/path/to/luks_backup.fsa:/media/username/VolumeName"
```
Replace "/media/username/VolumeName" with the actual mount point of your LUKS volume, and adjust the backup file path as needed.

With these configurations, the script will automatically determine the correct device path for your LUKS volume using the get_device_path() function. This ensures that your encrypted data is backed up correctly without requiring manual device path specification.

The script will create backups of your LUKS encrypted volumes while they remain mounted and encrypted, ensuring that your data stays secure throughout the backup process. The resulting backup file will contain the encrypted data, maintaining the security of your sensitive information.

Troubleshooting Common Issues

Permission Denied Errors: Ensure the script is run with sudo or as root.
Email Notifications Not Working: Check SSMTP configuration and your email provider's security settings.
Backup Failing for Specific Partitions: Verify the partition paths in the BACKUP_PARAMETERS array.
Out of Space Errors: Ensure sufficient free space on the backup destination drive.

Known Security Issues

Password Visibility in Process List

There is a known security vulnerability in the current implementation of this backup script, which stems from a limitation in the FSArchiver tool:

The encryption password, stored in the FSPASS variable, is passed to FSArchiver using the -c parameter on the command line.
This method exposes the password in clear text to any user who can view the system's process list during a backup operation.

Impact: During a backup process, any user on the system could potentially view the encryption password by examining the running processes, compromising the confidentiality of the backup.

Mitigation and Future Resolution

We are aware of this issue and are taking the following steps:

Temporary Mitigation: We recommend running backups when fewer users have access to the system, and considering the use of more restrictive permissions on process list viewing.
Future Fix: The FSArchiver development team has acknowledged this issue and plans to implement a more secure method in a future release. This update will allow FSArchiver to read the password directly from a variable, eliminating the need to pass it via command line arguments.

Recommendation: Users should be aware of this limitation and take appropriate precautions. We advise closely monitoring system access during backup operations and considering the sensitivity of your data when using this script in multi-user environments.

We are actively monitoring the progress of FSArchiver's development and will update our script as soon as a more secure method becomes available. Please check our repository regularly for updates and security patches.

Restoring from Backups

To restore a backup, use the following command:

sudo fsarchiver restfs /path/to/backup.fsa id=0,dest=/dev/sdXY

Replace /path/to/backup.fsa with your backup file path and /dev/sdXY with the target partition.

For restoring the data the fsarchiver team recommends to use this rescue system:

https://www.system-rescue.org/

Performance Tuning

The script uses a high compression level (Z15) by default. If you experience slow backups, you can adjust this in the do_backup() function:

fsarchiver ${EXCLUDE_STATEMENTS} -o -v -A -j$(nproc) -Z10 -c "$FSPASS" savefs "$1" "$2" 2>&1 | tee -a $BACKUP_LOG

Lowering the -Z value (e.g., to 10 or 5) will reduce compression but speed up the backup process.

Cloud Integration

To integrate with cloud storage, you can use tools like rclone. After the backup is complete, add a command to sync the backup file to your cloud storage:

rclone copy /path/to/backup remote:backup-folder

FSArchiver Options Explained

-o: Overwrite the output file if it already exists
-v: Verbose mode, shows detailed progress
-A: Allow to save a filesystem which is mounted in read-write
-j$(nproc): Use multiple threads for compression (number of CPU cores)
-Z15: Set compression level (0-19, higher is more compression but slower)
-c: Create an encrypted archive

Handling Multiple Disks and Complex Configurations

To expand the script for multiple disks or complex storage configurations:

Add more entries to the BACKUP_PARAMETERS array:

BACKUP_PARAMETERS[2]="/path/to/backup3.fsa /dev/sdc1"
BACKUP_PARAMETERS[3]="/path/to/backup4.fsa /dev/sdd1"

For RAID or LVM configurations, use the appropriate device paths:

BACKUP_PARAMETERS[4]="/path/to/raid_backup.fsa /dev/md0"
BACKUP_PARAMETERS[5]="/path/to/lvm_backup.fsa /dev/mapper/vg0-lv0"

Security Best Practices

Encryption Key Management: Store the bkp-password.sec file in a secure location with restricted permissions.
Regular Password Rotation: Change the encryption password periodically.
Secure Backup Storage: Store backups in a location with restricted access.
Network Security: If using network storage, ensure the connection is encrypted.
Audit Logging: Implement additional logging for backup operations and access to backup files.

Disclaimer and Liability

Important: This script is provided as-is, without any warranty or guarantee. Users should understand that they are using this script at their own risk. LEXO does not take any responsibilities or liabilities for any data loss, system damage, or any other issues that may arise from the use of this script. It is strongly recommended to thoroughly test the script in a non-production environment before using it on critical systems. Always ensure you have multiple backups of your important data using various methods.

Conclusion

This FSArchiver-based backup script offers a powerful, flexible solution for Linux users seeking a reliable way to create full system backups without rebooting. By leveraging open-source tools and customizable configurations, you can ensure your critical data is protected while maintaining system uptime. Whether you're managing a single machine or a fleet of Linux systems, this script provides a solid foundation for your backup strategy.

Remember to regularly test your backups and restore processes to ensure the integrity of your data and the effectiveness of your backup solution. With proper implementation and maintenance, this script can significantly enhance your Linux system's data protection capabilities.

Further Resources

By following this guide and utilizing the provided script, you can implement a robust, automated, and secure backup solution for your Linux systems. Remember to adapt the script to your specific needs and always follow best practices for data backup and security.

November 18th 2024