Debian: Setting up a full Debian repository mirror

Creating a full mirror of the Debian repository is a task that requires significant resources and a reliable internet connection. A full mirror includes all packages, updates, backports, and security updates for a specific Debian release.
Debmirror is a specialized Perl utility designed for creating and maintaining Debian and Ubuntu repository mirrors. Unlike standard "rsync" methods, it offers a simpler and more reliable way to synchronize, especially for selective mirroring.

Below are the steps for setting up a Debian 13 ("trixie") repository mirror using "Debmirror":

1. Installing Debmirror and Apache2 web server:
sudo apt-get update
sudo apt-get install debmirror apache2 -y

The basic syntax of the debmirror command is as follows:

debmirror [OPTIONS] /path/to/local/mirror

Key debmirror parameters:
-a, --arch : Architectures (e.g., amd64,i386);
-s, --section : Repository sections (main, contrib, non-free for Debian; main, restricted, universe, multiverse for Ubuntu);
-d, --dist : Distributions (code names, e.g., bullseye, bookworm, trixie for Debian; focal, jammy for Ubuntu). Multiple can be specified separated by commas, including updates (bullseye-updates, bullseye-security);
-h, --host : Source mirror URL (e.g., ftp.debian.org, archive.ubuntu.com or a local Nexus repository);
-r, --root : Root path on the remote server (e.g., /debian for Debian, /ubuntu for Ubuntu);
-e, --method : Download protocol (http, https, ftp, rsync). http or https are often the most reliable.
--progress : Show download progress.
--nosource : Do not download package source code (significantly saves space).
--ignore-release-gpg : Ignore GPG signature verification of Release files. Use with caution as this reduces security.
--ignore-missing-release : Continue synchronization even if the Release file is missing on the server.


2. Test synchronization:

Create repository storage space on disk:
sudo mkdir -p /srv/mirror/debian
sudo mkdir -p /srv/mirror/debian-security
sudo chown -R nobody:nogroup /srv/mirror/debian
sudo chown -R nobody:nogroup /srv/mirror/debian-security

Then run synchronization commands as a test:

Synchronization of the main Debian repository (trixie,trixie-updates,trixie-backports):
debmirror \
-a amd64 \
--nosource \
-s main,contrib,non-free,non-free-firmware \
-d trixie,trixie-updates,trixie-backports \
-h ftp.debian.org \
-r /debian \
-e https \
--progress \
--ignore-release-gpg \
--no-check-gpg \
--rsync-extra=none \
/srv/mirror/debian

Synchronization of Debian security patches (trixie-security):
debmirror \
-a amd64 \
--nosource \
-s main,contrib,non-free,non-free-firmware \
-d trixie-security \
-h security.debian.org \
-r /debian-security \
-e https \
--progress \
--ignore-release-gpg \
--no-check-gpg \
--rsync-extra=none \
/srv/mirror/debian-security

If everything completed without errors, proceed to the next step.

3. Configuring the web server (Apache2) for access to the local repository:

To make the mirror available via HTTP create a file:
sudo nano /etc/apache2/sites-available/debian-mirror.conf

with the following content:

<VirtualHost *:80>
    ServerName your-mirror-domain.com #Replace with your server name or IP
    DocumentRoot /srv/mirror
    <Directory "/srv/mirror">
        Options +Indexes +FollowSymLinks
        IndexOptions NameWidth=* +SuppressDescription FancyIndexing
        Require all granted
    </Directory>
    ErrorLog ${APACHE_LOG_DIR}/debian-mirror_error.log
    CustomLog ${APACHE_LOG_DIR}/debian-mirror_access.log combined
</VirtualHost>

If HTTPS support is needed:

Create directories for the key pair (if they don't exist):
sudo mkdir -p /etc/ssl/private /etc/ssl/certs

Then generate a certificate:
sudo openssl req -x509 -nodes -days 730 -newkey rsa:2048 \ -keyout /etc/ssl/private/your-mirror.key \ -out /etc/ssl/certs/your-mirror.crt

During generation, specify Common Name: your-mirror-domain.com. Other fields can be filled as desired or left blank.

Set correct permissions for the key:
sudo chmod 600 /etc/ssl/private/your-mirror.key

And create a file:
sudo nano /etc/apache2/sites-available/debian-mirror-ssl.conf

with the following content:

<VirtualHost *:443>
    ServerName your-mirror-domain.com #Replace with your server name or IP
    DocumentRoot /srv/mirror
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/your-mirror.crt
    SSLCertificateKeyFile /etc/ssl/private/your-mirror.key
    <Directory "/srv/mirror">
        Options +Indexes +FollowSymLinks
        IndexOptions NameWidth=* +SuppressDescription FancyIndexing
        Require all granted
    </Directory>
    ErrorLog ${APACHE_LOG_DIR}/debian-mirror_ssl_error.log
    CustomLog ${APACHE_LOG_DIR}/debian-mirror_ssl_access.log combined
</VirtualHost>

Activate the configuration and restart Apache:
sudo a2ensite debian-mirror
sudo a2ensite debian-mirror-ssl #For HTTPS support
sudo apachectl configtest
sudo a2enmod rewrite
sudo a2enmod ssl #For HTTPS support
sudo systemctl restart apache2

Check accessibility: Open "http://your-server-ip/debian" in a browser. The repository directory structure should be displayed.


4. Creating a script to automate the process and monitoring:

The following script can not only perform synchronization of the required repository branches using the "debmirror" command but also log activities, check if the script was previously running, and send reports via email. Create a file:
sudo nano /usr/local/bin/update-mirror.sh

Add the script text to it and make it executable:
sudo chmod +x /usr/local/bin/update-mirror.sh

#!/bin/bash

# Script settings
LOG_FILE="/var/log/mirror_update.log"
LOCK_FILE="/var/run/mirror_update.lock"
MIRROR_BASE="/srv/mirror"

# Email notifications
EMAIL_RECIPIENTS="admin@example.com,devops@example.com"  # Replace with real addresses
EMAIL_SUBJECT_PREFIX="[Mirror Update] "
SMTP_SERVER="Input_IP_of_SMTP_Server"  # SMTP server address
FROM_EMAIL="mirror-update@$(hostname)"  # Sender address

# Mirror configurations
declare -A MIRRORS=(
    ["main"]="
        -a amd64
        --nosource
        -s main,contrib,non-free,non-free-firmware
        -d trixie,trixie-updates,trixie-backports
        -h ftp.by.debian.org
        -r /debian
        -e rsync
        --progress
        --ignore-release-gpg
        --no-check-gpg
        ${MIRROR_BASE}/debian
    "
    ["security"]="
        -a amd64
        --nosource
        -s main,contrib,non-free,non-free-firmware
        -d trixie-security
        -h ftp.by.debian.org
        -r /debian-security
        -e rsync
        --progress
        --ignore-release-gpg
        --no-check-gpg
        ${MIRROR_BASE}/debian-security
    "
)

# Logging function
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S'): $1" >> "$LOG_FILE"
}

# Email sending function using curl and SMTP
send_email() {
    local subject="$1"
    local body="$2"

    # Check if curl is installed
    if command -v curl &> /dev/null; then
        # Form email in MIME format
        local email_content=$(cat <<EOF
From: ${FROM_EMAIL}
To: ${EMAIL_RECIPIENTS}
Subject: ${EMAIL_SUBJECT_PREFIX}${subject}
Date: $(date -R)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"

${body}
EOF
)

        # Send via SMTP using curl
        echo "$email_content" | curl -s --url "smtp://${SMTP_SERVER}" \
            --mail-from "${FROM_EMAIL}" \
            --mail-rcpt "${EMAIL_RECIPIENTS}" \
            --upload-file - \
            --insecure

        if [ $? -eq 0 ]; then
            log "Email sent successfully: ${subject}"
        else
            log "ERROR: Failed to send email: ${subject}"
        fi
    elif command -v sendmail &> /dev/null; then
        # Alternative method using sendmail
        (
            echo "Subject: ${EMAIL_SUBJECT_PREFIX}${subject}"
            echo "From: ${FROM_EMAIL}"
            echo "To: ${EMAIL_RECIPIENTS}"
            echo
            echo "${body}"
        ) | sendmail -t
    else
        log "ERROR: No email client found. Cannot send notification: $subject"
    fi
}

# Lock check function
check_lock() {
    if [ -e "$LOCK_FILE" ]; then
        local pid=$(cat "$LOCK_FILE")
        if kill -0 "$pid" 2>/dev/null; then
            log "ERROR: Script is already running with PID $pid"
            send_email "Script already running" "Mirror update script is already running with PID $pid. Current execution blocked."
            exit 1
        else
            log "WARNING: Lock file found but process does not exist. Removing lock file."
            rm -f "$LOCK_FILE"
        fi
    fi
}

# Lock creation function
create_lock() {
    echo $$ > "$LOCK_FILE"
}

# Lock removal function
remove_lock() {
    rm -f "$LOCK_FILE"
}

# Debmirror execution function
run_mirror() {
    local name="$1"
    local options="$2"

    log "Starting mirror update: $name"
    log "Command: debmirror $options"

    # Command execution
    debmirror $options

    local exit_code=$?
    if [ $exit_code -eq 0 ]; then
        log "Update successful: $name"
        return 0
    else
        log "Update failed with exit code $exit_code: $name"
        return $exit_code
    fi
}

# Main logic
main() {
    log "=== Starting mirror update process ==="

    # Check lock
    check_lock

    # Create lock
    create_lock

    # Add cleanup handler on exit
    trap 'remove_lock; log "Script interrupted"; exit 1' INT TERM
    trap 'remove_lock' EXIT

    local overall_success=true

    for mirror_name in "${!MIRRORS[@]}"; do
        if ! run_mirror "$mirror_name" "${MIRRORS[$mirror_name]}"; then
            overall_success=false
            log "ERROR: Mirror $mirror_name failed with exit code $?"
            send_email "Update failed: $mirror_name" "Mirror update for $mirror_name failed. Check $LOG_FILE for details."
        fi
    done

    if [ "$overall_success" = true ]; then
        log "=== All mirror updates completed successfully ==="
        send_email "All updates completed" "All mirror updates completed successfully. Check $LOG_FILE for details."
    else
        log "=== Some mirror updates failed ==="
        # Error notifications were already sent for each specific failure
    fi
}

# Error handling
set -euo pipefail

# Launch main function
main "$@"

5. Setting up log rotation:

Create a file /etc/logrotate.d/update_mirror with the following content:

/var/log/mirror_update.log {
weekly
missingok
rotate 26
compress
delaycompress
notifempty
create 644 root root
dateext
dateformat -%Y-%m-%d
}

Explanation of directives:
weekly - rotation occurs once a week;
missingok - skip rotation if the file is missing;
rotate 26 - keep 26 archive copies (half a year ≈ 26 weeks);
compress - compress archive logs using gzip;
delaycompress - delay compression until the next rotation (convenient for debugging);
notifempty - do not rotate empty files;
create 644 root root - create a new log file with permissions 644 and owner root:root after rotation;
dateext - add date to the archive file name;
dateformat -%Y-%m-%d - date format in the file name (YYYY-MM-DD).

Configuration check:
sudo logrotate -d /etc/logrotate.d/update_mirror
The -d key enables debug mode, which will show potential problems without performing rotation.

Forced rotation for testing:
sudo logrotate -vf /etc/logrotate.d/update_mirror
The -v key enables verbose output, and -f forcibly starts rotation.


6. Setting up automatic synchronization via cron

To keep the mirror up to date, it needs to be regularly synchronized. It is recommended to do this at least 4 times a day.
Create a cron job (sudo crontab -e):
Synchronization every 6 hours (at random minutes for even load distribution):
0 */6 * * * /usr/local/bin/update-mirror.sh

or synchronization daily at 2 AM:
0 2 * * * /usr/local/bin/update-mirror.sh

Do not synchronize the mirror exactly at 3:00, 9:00, 15:00, and 21:00 UTC, as this is when the main mirrors are updated. Add a random offset of a few minutes.


7. Testing and using the mirror:

Test your mirror by configuring sources.list on another machine:
sudo nano /etc/apt/sources.list

Add the lines:
deb http://your-mirror-ip-or-domain/debian/ trixie main contrib non-free non-free-firmware
deb http://your-mirror-ip-or-domain/debian/ trixie-updates main contrib non-free non-free-firmware
deb http://your-mirror-ip-or-domain/debian/ trixie-backports main contrib non-free non-free-firmware
deb http://your-mirror-ip-or-domain/debian-security/ trixie-security main contrib non-free non-free-firmware

Update package indexes:
sudo apt update
sudo apt upgrade