Backup and Restore

Audience: Technical administration

Overview

oPBWeb’s data lives in two places that belong together: the database holds all metadata, the object storage holds the file contents. The database refers to objects in the storage via SHA-256 checksums and keys (blobs/…, artifacts/…). A backup is only usable if both parts come from the same point in time and the matching secrets are included.

What to back up

Also note the oPBWeb version (Git commit or the versions from the status bar). A database can only be used with the same or a newer oPBWeb version.

Creating a backup

To make sure database and object storage have the same state, first stop the services that change data, then back up the database, and then back up the object storage with the stack stopped.

1. Check under Jobs that no jobs are running or waiting.

2. Create a protected backup directory:

   mkdir -p -m 700 backup
   

3. Stop frontend, worker and API. Database and object storage keep running:

   docker stop opbweb-frontend opbweb-worker opbweb-api
   

4. Back up the database:

   docker exec opbweb-db sh -c 'mariadb-dump -uroot -p"$(cat /run/secrets/db_root_password)" --single-transaction opbweb' > backup/opbweb.sql
   

   The command reads the root password inside the container from the Docker secret; it does not appear on the host’s command line.

5. Stop the stack:

   ./stop.sh
   

6. Back up the two Garage volumes (example with an Alpine helper container):

   docker run --rm -v opbweb_garage-meta:/v:ro -v "$PWD/backup":/b alpine tar czf /b/garage-meta.tgz -C /v .
   docker run --rm -v opbweb_garage-data:/v:ro -v "$PWD/backup":/b alpine tar czf /b/garage-data.tgz -C /v .
   

7. Back up secrets, configuration and certificates:

   tar czf backup/config.tgz secrets .env certs
   

8. Start the stack again:

   ./start.sh
   

Then transfer the contents of backup/ to your backup target. The downtime depends mainly on the size of the object storage.

If a short interruption of the whole stack is acceptable anyway, you can back up the volume opbweb_mariadb-data with the stack stopped, just like the Garage volumes, instead of creating a dump. In that case, the passwords from secrets/db_password.txt and secrets/db_root_password.txt, which are stored in the volume, must be kept together with the database.

Restoring

The order matters: first secrets and configuration, then the object storage, then the database, and finally the start.

1. Stop the stack.

   ./stop.sh
   

2. Provide the program version. Check out the same oPBWeb version that belongs to the backup (or a newer one). If you use images from a registry, their versions are in the backed-up .env (BACKEND_IMAGE etc.).

3. Restore secrets, .env and certificates. Use the original files from the backup, not newly generated ones:

   tar xzf backup/config.tgz
   chmod 750 secrets && chmod 444 secrets/*
   

   The files need permissions 444 so that the processes in the containers can read them; start.sh sets the permissions via init-secrets.sh on every start anyway.

4. Restore the object storage. Remove the existing Garage volumes including the S3 credentials and restore the backup:

   docker volume rm opbweb_garage-meta opbweb_garage-data opbweb_garage-credentials
   docker volume create opbweb_garage-meta
   docker volume create opbweb_garage-data
   docker run --rm -v opbweb_garage-meta:/v -v "$PWD/backup":/b alpine tar xzf /b/garage-meta.tgz -C /v
   docker run --rm -v opbweb_garage-data:/v -v "$PWD/backup":/b alpine tar xzf /b/garage-data.tgz -C /v
   

   You must remove the volume garage-credentials so that garage-init rewrites the credentials from the restored key. If an old volume remains, the credentials do not match the object storage. On the next start, the garage-init log shows “Credential files missing, writing from existing key…”.

5. Restore the database. Remove the database volume, start only the database and import the dump:

   docker volume rm opbweb_mariadb-data
   docker compose up -d db
   docker compose ps db        # wait until the status is "healthy"
   docker exec -i opbweb-db sh -c 'mariadb -uroot -p"$(cat /run/secrets/db_root_password)" opbweb' < backup/opbweb.sql
   

   On the first start with the empty volume, MariaDB creates the database opbweb and the application user with the passwords from secrets/. If you backed up the volume instead of a dump, restore it like the Garage volumes.

6. Start the stack.

   ./start.sh
   

   The API runs any missing migrations if the program version is newer than the backup. It does not create a first administrator account, because the user table from the backup is populated.

7. Verify.

   • System status: all services OK. As an administrator, check under Object storage that no missing blobs are reported; missing blobs indicate that database and object storage come from different points in time.
   • Depots → config server: Test connection for each config server. If sign-in fails, opsi_key_private.pem does not match the database; enter the password of the technical user again.
   • Open a package and download a file or a built package.

Jobs that were waiting or running at the time of the backup are not resumed by oPBWeb after the restore. Start them again if needed.

Consistency

For regular backups, a fixed order in a script that runs all steps of the section Creating a backup one after another helps.