Upgrades

Upgrade from the release channel

Re-run the public install script on the host that already has LabPod installed:

Ubuntu / Debian Rocky / RHEL / AlmaLinux / Fedora

# Preview what the bundled installer would change
curl -fsSL https://labpod.ai/install.sh | sudo bash -s -- --check

# Upgrade to the latest release
curl -fsSL https://labpod.ai/install.sh | sudo bash

# Or pin a specific release
curl -fsSL https://labpod.ai/install.sh | sudo bash -s -- --version v0.4.0

# Run these as root
# Preview what the bundled installer would change
curl -fsSL https://labpod.ai/install.sh | bash -s -- --check

# Upgrade to the latest release
curl -fsSL https://labpod.ai/install.sh | bash

# Or pin a specific release
curl -fsSL https://labpod.ai/install.sh | bash -s -- --version v0.4.0

On an existing install, the installer does:

1. Release download - downloads the release tarball and verifies its checksum and signature.
2. Pre-upgrade recovery set - when the downloaded binary differs from the installed one, retains the executable, application support files, and a database snapshot under /var/lib/labpod/backups/pre-upgrade-<UTC>.* before replacing anything. Pass --skip-backup only if you intentionally want to suppress the database snapshot.
3. Pinned candidate and asset refresh - resolves latest to an immutable release tag, verifies the candidate binary reports that version, then replaces /usr/local/bin/labpod and the /opt/labpod/inject tree only when release contents changed.
4. Config preservation - keeps /etc/labpod/labpod.env, /var/lib/labpod/labpod.db, backups, licenses, TLS cache, Linux accounts, and user home directories.
5. Unit refresh - copies the latest systemd unit files to /etc/systemd/system/.
6. Preflight, migrate, then restart - validates the configured database, image-store, TLS, GPU-sharing and inject-tree paths before taking the API offline. A configuration error leaves the currently running service untouched so you can fix it and re-run the installer. For a valid upgrade, the installer then stops only the LabPod API while it refreshes files and applies durable database upgrades.
7. Post-restart health check and rollback - polls /api/version and verifies the restarted API reports the resolved release. If any upgrade step fails, the installer restores the previous executable, support files, database when migration was attempted, API service, and backup timer. It verifies the restored API before reporting rollback success. Manual recovery is needed only if this automatic rollback also fails.

The recovery set is automatic for a real upgrade. A no-op reinstall with an identical binary does not create one. The installer keeps the newest 10 complete recovery sets by default and prunes an older executable, application-file snapshot, and paired database snapshot together. Daily labpod-*.db backups follow their own retention policy.

If you configure a custom backup directory, keep it outside the updater/support directory and the separately located inject tree. A rollback must not delete or recursively copy the artifacts it is using to restore the installation. If you pass --skip-backup and a migration begins, automatic rollback has no database snapshot to restore.

For extra caution, you can still take a manual backup before upgrading:

Ubuntu / Debian Rocky / RHEL / AlmaLinux / Fedora

sudo labpod admin backup --out /var/lib/labpod/backups/pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ).db

# Run these as root
labpod admin backup --out /var/lib/labpod/backups/pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ).db

Running workspaces survive

Workspace containers keep running through the restart. LabPod’s service restart takes ~2 seconds; in-flight API requests that were processing at that moment may fail, but the containers themselves (Jupyter kernels and training runs) continue. Workspace pages and applications reload after the service returns. The LabPod Terminal reconnects automatically and keeps its tmux session and scrollback.

New API calls land on the new binary within ~2 seconds of restart.

Migration policy

LabPod schema migrations are forward-only. An upgraded database should run with the release that migrated it, or with a newer release. Do not run a restored older database with a newer upgraded binary, and do not run an already-migrated database with an older binary.

The installer uses the paired pre-upgrade-* recovery set for automatic rollback. Do not hand-edit the installed DB or its internal schema state.

If an upgrade fails

The installer exits nonzero even when automatic rollback succeeds, so the failed update remains visible. Confirm that /api/version reports the previous version, review the installer output and journalctl -u labpod, fix the cause, then retry.

If automatic rollback cannot complete, the installer prints the exact stopped-service recovery commands and names the retained executable, application files, and database snapshot for that attempt. Follow those commands from the host shell. Do not start a pinned older bootstrap or an older release’s installer before restoring the paired snapshot: that can run migrations against the wrong database generation. The database schema and executable must come from the same release generation.

Update from the admin UI

After this release has been installed once, the root administrator can use Admin → Software update to check the public release channel and start an upgrade. The page uses the same signed package and checksum verification as the public install command, records the installer log, and creates the same pre-upgrade recovery set.

Automatic checks are off by default. Enable them only when the host may contact the public release service; LabPod checks once every 24 hours but never installs a release without the administrator’s action. Air-gapped hosts continue to use the installer command above. The equivalent terminal commands are labpod admin update status, check, and apply.

Moving an account-home base

Changing LABPOD_FILEOP_HOME_BASE does not move existing account data. Before changing it on an installed server, stop each affected workspace and the LabPod service, then move each complete account home (including workspaces and non-relocated work) to <new-base>/<user>. Preserve ownership and update the account’s passwd home to that path before restarting LabPod.

Existing workspaces keep their recorded persistent home until the old path is removed. During that transition the Files view follows the existing path and LabPod rejects Delete with workspace_home_migration_required, so data cannot be orphaned. LabPod never performs this move for you; verify the copy before removing the old directory.