Upgrades

Upgrade from the release channel

Re-run the public install script on the host that already has LabPod installed:

Ubuntu / Debian Rocky / RHEL / AlmaLinux / Fedora

# Preview what the bundled installer would change
curl -fsSL https://labpod.ai/install.sh | sudo bash -s -- --check

# Upgrade to the latest release
curl -fsSL https://labpod.ai/install.sh | sudo bash

# Or pin a specific release
curl -fsSL https://labpod.ai/install.sh | sudo bash -s -- --version v0.4.0

# Run these as root
# Preview what the bundled installer would change
curl -fsSL https://labpod.ai/install.sh | bash -s -- --check

# Upgrade to the latest release
curl -fsSL https://labpod.ai/install.sh | bash

# Or pin a specific release
curl -fsSL https://labpod.ai/install.sh | bash -s -- --version v0.4.0

On an existing install, the installer does:

1. Release download - downloads the release tarball and verifies its checksum and signature.
2. Pre-upgrade recovery set - when the downloaded binary differs from the installed one, retains the executable, application support files, and a database snapshot under /var/lib/labpod/backups/pre-upgrade-<UTC>.* before replacing anything. Pass --skip-backup only if you intentionally want to suppress the database snapshot.
3. Pinned candidate and asset refresh - resolves latest to an immutable release tag, verifies the candidate binary reports that version, then replaces /usr/local/bin/labpod and the /opt/labpod/inject tree only when release contents changed.
4. Config preservation - keeps /etc/labpod/labpod.env, /var/lib/labpod/labpod.db, backups, licenses, TLS cache, Linux accounts, and user home directories.
5. Unit refresh - copies the latest systemd unit files to /etc/systemd/system/.
6. Preflight, migrate, then restart - validates the configured database, image-store, TLS, GPU-sharing and inject-tree paths before taking the API offline. A configuration error leaves the currently running service untouched so you can fix it and re-run the installer. For a valid upgrade, the installer then stops only the LabPod API while it refreshes files and applies durable database upgrades.
7. Post-restart health check and rollback - polls /api/version and verifies the restarted API reports the resolved release. If any upgrade step fails, the installer restores the previous executable, support files, database when migration was attempted, API service, and backup timer. It verifies the restored API before reporting rollback success. Manual recovery is needed only if this automatic rollback also fails.

The recovery set is automatic for a real upgrade. A no-op reinstall with an identical binary does not create one. The installer keeps the newest 10 complete recovery sets by default and prunes an older executable, application-file snapshot, and paired database snapshot together. Daily labpod-*.db backups follow their own retention policy.

If you configure a custom backup directory, keep it outside the updater/support directory and the separately located inject tree. A rollback must not delete or recursively copy the artifacts it is using to restore the installation. If you pass --skip-backup and a migration begins, automatic rollback has no database snapshot to restore.

For extra caution, you can still take a manual backup before upgrading:

Ubuntu / Debian Rocky / RHEL / AlmaLinux / Fedora

sudo labpod admin backup --out /var/lib/labpod/backups/pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ).db

# Run these as root
labpod admin backup --out /var/lib/labpod/backups/pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ).db

Running workspaces survive

Workspace containers keep running through the restart. LabPod’s service restart takes ~2 seconds; in-flight API requests that were processing at that moment may fail, but the containers themselves (Jupyter kernels and training runs) continue. Workspace pages and applications reload after the service returns. The LabPod Terminal reconnects automatically and keeps its tmux session and scrollback.

New API calls land on the new binary within ~2 seconds of restart.

Workspace template images after upgrading

Every built-in workspace template now pins an immutable image version instead of a moving tag. Expect one image download per workspace template your lab actually uses, on every upgrade that changes those pins - for example, PyTorch, TensorFlow, SciPy, Code Server, and PyTorch Demo moving from tags like :cu126, :py312, and :latest to v1-... versions. These are different images, not renames, so whatever was already pulled on this host does not satisfy the new pin.

Running workspaces are not affected - they keep using their already-running container. Starting a stopped workspace whose image changed reports that the image needs to be downloaded first; researchers can do that themselves from the workspace page, with no administrator build required. Each researcher has a separate rootless image store, so each account downloads the new pin when it first uses that workspace template. A network registry cache can reduce repeated network transfer.

Upgrading never enables a workspace template you had left disabled - your enable/disable choices are preserved. If you had already enabled and locally built one of the workspace templates that is now a published image (Miniforge, uv, R ML, RStudio, Hugging Face, ComfyUI, CUDA Composite, or Parallel Programming), it now points at the tested published image instead of your local build. Your previously built image is left in each user’s store; that user can remove it from Advanced tools → Workspace templates → My images once it is no longer needed. To keep using your own build instead, edit that workspace template’s image reference - that marks it customized and stops LabPod from re-applying the shipped default.

Upgrades remove the retired LABPOD_SHARED_IMAGE_STORE_MODE and LABPOD_SHARED_IMAGE_STORE settings and drop pending image-promotion requests. They do not delete files from an old root cache directory; inspect and remove that directory manually after confirming it is no longer needed.

Migration policy

LabPod schema migrations are forward-only. An upgraded database should run with the release that migrated it, or with a newer release. Do not run a restored older database with a newer upgraded binary, and do not run an already-migrated database with an older binary.

The installer uses the paired pre-upgrade-* recovery set for automatic rollback. Do not hand-edit the installed DB or its internal schema state.

If an upgrade fails

The installer exits nonzero even when automatic rollback succeeds, so the failed update remains visible. Confirm that /api/version reports the previous version, review the installer output and journalctl -u labpod, fix the cause, then retry.

If automatic rollback cannot complete, the installer prints the exact stopped-service recovery commands and names the retained executable, application files, and database snapshot for that attempt. Follow those commands from the host shell. Do not start a pinned older bootstrap or an older release’s installer before restoring the paired snapshot: that can run migrations against the wrong database generation. The database schema and executable must come from the same release generation.

Roll back to a retained release

The installer keeps the most recent pre-upgrade recovery sets, and a root administrator can restore one from the host shell when a completed upgrade turns out to be wrong:

Ubuntu / Debian Rocky / RHEL / AlmaLinux / Fedora

sudo labpod admin rollback list
sudo labpod admin rollback apply 20260812T120000Z --yes

# Run these as root
labpod admin rollback list
labpod admin rollback apply 20260812T120000Z --yes

list shows the recovery ID, the retained version, and whether each set is complete, compatible, and eligible; a set that cannot be applied explains why. Add --json for inventory tooling. apply accepts only a complete set whose retained binary is older than the installed one. It first captures the current installation as a new recovery set, stops the API, restores the retained binary, application files, and paired database, restarts the API, and verifies /api/version. If the restored server does not come back healthy, it puts the pre-rollback set back and still exits nonzero.

Rollback always restores that set’s database snapshot, so LabPod changes accepted after the recovery timestamp are discarded. Workspace files are not rewound and rootless workspace containers keep running. This is why --yes is required and why there is no button for it in the web interface.

Update from the admin UI

After this release has been installed once, the root administrator can use Admin → Software update to check the public release channel and start an upgrade. The page uses the same signed package and checksum verification as the public install command, records the installer log, and creates the same pre-upgrade recovery set.

Automatic checks are off by default. Enable them only when the host may contact the public release service; LabPod checks once every 24 hours but never installs a release without the administrator’s action. Air-gapped hosts continue to use the installer command above. The equivalent terminal commands are labpod admin update status, check, and apply.

While a scheduled update runs

A scheduled update announces itself with a banner. Apply now starts it ahead of its scheduled time, and that control disappears once the update is actually running - selecting it again could only report that an update was already in progress. Cancel update stays available until the installer starts.

After the server restarts, a tab left open still runs the interface from the previous version and can ask the server for parts of it that no longer exist. The Software update page offers Reload the interface for exactly that, and stops repeating the notice once the tab and the installed server match. A page that fails to load for the same reason names the cause and offers Reload rather than a bare “Internal Error”.

Moving a Linux account HOME

LabPod follows the HOME recorded in the host account database and has no separate account-home base. To move an account after installation, stop its workspaces and the LabPod service, then move the complete HOME, including workspaces and non-relocated work. Preserve ownership and update the account’s passwd HOME before restarting LabPod.

Existing workspaces keep their recorded persistent home until the old path is removed. During that transition the Files view follows the existing path and LabPod rejects Delete with workspace_home_migration_required, so data cannot be orphaned. LabPod never performs this move for you; verify the copy before removing the old directory.

Rootless Podman storage is separate. If it is still below the old HOME, migrate it independently; see Rootless Podman storage.