Skip to content
LabPod LabPod LabPod Docs

Backups & restore

LabPod backs up its SQLite database - workspaces, users, workspace templates, policies, audit logs, and settings. User files in each user’s work directory (their Linux home directory by default, or under LABPOD_WORK_BASE when that’s configured) are not included in the database backup; back those up with your host-level backup tool separately.

Daily backup timer

Section titled “Daily backup timer”

The install script registers labpod-backup.timer, which runs once a day with a randomized delay and calls:

Terminal window
labpod admin backup --retention 14

This creates a timestamped snapshot (labpod-YYYYMMDDTHHMMSSZ.db) and keeps the 14 most recent, pruning older ones. By default it uses /var/lib/labpod/backups; set LABPOD_BACKUP_DIR in /etc/labpod/labpod.env to move both scheduled backups and installer recovery artifacts together.

Terminal window
# Inspect the timer
sudo systemctl list-timers labpod-backup.timer


# Check recent backup logs
sudo journalctl -u labpod-backup.service --since today


# List existing backups
ls -lah /var/lib/labpod/backups/
Terminal window
# Run these as root
# Inspect the timer
systemctl list-timers labpod-backup.timer


# Check recent backup logs
journalctl -u labpod-backup.service --since today


# List existing backups
ls -lah /var/lib/labpod/backups/

Force a snapshot now:

Terminal window
sudo systemctl start labpod-backup.service
Terminal window
# Run these as root
systemctl start labpod-backup.service

Tune retention or directory

Section titled “Tune retention or directory”

Set a directory in the environment file and create it as root:

Terminal window
sudoedit /etc/labpod/labpod.env
# Add or change: LABPOD_BACKUP_DIR=/srv/labpod/backups
sudo install -d -m 0750 /srv/labpod/backups
Terminal window
# Run these as root
sudoedit /etc/labpod/labpod.env
# Add or change: LABPOD_BACKUP_DIR=/srv/labpod/backups
install -d -m 0750 /srv/labpod/backups

labpod-backup.service runs under a ProtectSystem=strict sandbox that only allows writes to the paths listed in its unit’s ReadWritePaths. Editing LABPOD_BACKUP_DIR alone does not update that list, so relocating the directory (or tuning retention) needs a drop-in:

Terminal window
sudo systemctl edit labpod-backup.service
Terminal window
# Run these as root
systemctl edit labpod-backup.service

In the editor, paste:

[Service]
ExecStart=
ExecStart=/usr/local/bin/labpod admin backup --retention 30
ReadWritePaths=/var/lib/labpod /srv/labpod/backups

Drop the ReadWritePaths line if you’re only tuning retention and keeping the default directory.

Terminal window
sudo systemctl daemon-reload
sudo systemctl restart labpod-backup.timer
Terminal window
# Run these as root
systemctl daemon-reload
systemctl restart labpod-backup.timer

Ad-hoc backup

Section titled “Ad-hoc backup”

Write a self-contained snapshot to an explicit path:

Terminal window
sudo labpod admin --db /var/lib/labpod/labpod.db backup --out /tmp/labpod-adhoc.db
Terminal window
# Run these as root
labpod admin --db /var/lib/labpod/labpod.db backup --out /tmp/labpod-adhoc.db

This fails fast if the database file doesn’t exist (no silent empty-DB creation).

Restore

Section titled “Restore”

Restore requires stopping the service because SQLite refuses a hot copy when the WAL sidecar (.db-wal, .db-shm) is in use.

Terminal window
sudo systemctl stop labpod


sudo labpod admin --db /var/lib/labpod/labpod.db restore \
  /var/lib/labpod/backups/labpod-20260517T030000Z.db


sudo systemctl start labpod
Terminal window
# Run these as root
systemctl stop labpod


labpod admin --db /var/lib/labpod/labpod.db restore \
  /var/lib/labpod/backups/labpod-20260517T030000Z.db


systemctl start labpod

restore will prompt for confirmation unless you pass --yes.

What the backup contains

Section titled “What the backup contains”

The backup is a complete snapshot of the LabPod database:

  • Users, roles, and hashed passwords
  • Workspace definitions and GPU allocations
  • Workspace templates and their port/env configuration
  • Resource policies and system settings
  • Session records and audit logs
  • Usage samples and rollups

Not included: workspace container layers, user home directories, uploaded files. The workspace container state (running kernels, training checkpoints) is also not captured - only the workspace metadata.