Troubleshooting
A user can’t log in
Section titled “A user can’t log in”LabPod passwords are separate from Linux passwords. Check the LabPod credential:
sudo labpod admin --db /var/lib/labpod/labpod.db list-users # confirm the account existssudo labpod admin --db /var/lib/labpod/labpod.db set-password alice# Run these as rootlabpod admin --db /var/lib/labpod/labpod.db list-users # confirm the account existslabpod admin --db /var/lib/labpod/labpod.db set-password aliceCheck journald for failed-login records:
sudo journalctl -u labpod --since "5 minutes ago" | grep login_failed# Run these as rootjournalctl -u labpod --since "5 minutes ago" | grep login_failedA workspace is stuck in starting
Section titled “A workspace is stuck in starting”LabPod waits up to LABPOD_WORKSPACE_START_TIMEOUT (default 15 minutes) for podman run -d
to complete. On timeout the workspace transitions to failed. To investigate while it’s
still starting:
sudo runuser -u <owner> -- podman ps -a --filter name=labpod-ws-<id>sudo runuser -u <owner> -- podman inspect labpod-ws-<id>sudo runuser -u <owner> -- podman logs labpod-ws-<id># Run these as rootrunuser -u <owner> -- podman ps -a --filter name=labpod-ws-<id>runuser -u <owner> -- podman inspect labpod-ws-<id>runuser -u <owner> -- podman logs labpod-ws-<id>Common causes:
- Missing image - select the template and choose the confirmed Pull action for a registry image. For an administrator-built image, Pull copies the shared master into the user’s store; ask the administrator to build it first. Dockerfile-backed personal templates need Build.
- Out of disk - check
/homeusage; the Resources page shows disk pressure. - Port conflict - rare; LabPod allocates from a managed pool (10000–19999 by default).
Workspace start fails on a /work container label
Section titled “Workspace start fails on a /work container label”The user may see:
Couldn’t start the workspace because LabPod couldn’t apply the container label to
/work/<path>. In Files, download it first if needed, then delete it from/work; or move it outside~/workfrom your host account.
On SELinux hosts, LabPod mounts each user’s ~/work directory into containers as /work
with a shared container label. Rootless Podman applies that label as the workspace owner.
If ~/work contains root-owned or otherwise protected files, Podman cannot relabel them and
the workspace fails before the container starts.
Ask the user to inspect the path shown in the error. From their host account, they can move
the item outside ~/work:
mkdir -p ~/labpod-unmountedmv ~/work/<path> ~/labpod-unmounted/If they do not need the item, they can delete it from LabPod’s Files page instead. To find
other files in ~/work that the user does not own:
find ~/work -not -user "$USER" -lsIf the file should stay in ~/work, fix ownership instead:
sudo chown -R <user>:<user> /home/<user>/work/<path># Run these as rootchown -R <user>:<user> /home/<user>/work/<path>Do not disable SELinux labeling for the workspace mount. That weakens the isolation LabPod relies on for user-owned bind mounts.
Jupyter shows the username jovyan
Section titled “Jupyter shows the username jovyan”Some Jupyter-based images show jovyan in terminals, file paths, prompts, or notebook UI
even when the LabPod user is named something else.
LabPod’s built-in templates (PyTorch, TensorFlow, Data Science, and the PyTorch Demo) use
LabPod-built ghcr.io/labpod/* images that have no jovyan account, so you will not see this
on them. It can still appear on a template pointing at an upstream Jupyter Docker Stacks image
(quay.io/jupyter/*-notebook), where the jovyan account is baked into the image and LabPod
does not rewrite the image username at startup.
LabPod runs the container with --userns=keep-id, so the numeric UID/GID inside the
container matches the LabPod Linux user on the host.
Users can ignore the displayed name. Files written under /work still belong to their host
LabPod account because ownership follows the UID, not the image’s display username. However, a
template image that already reserves the workspace owner’s numeric UID is rejected before Start,
because it would direct applications to a non-persistent home. Template authors must leave that
UID free.
To verify from inside the workspace:
idtouch /work/labpod-uid-check.txtls -ln /work/labpod-uid-check.txtOn the host, the same file should show the LabPod user’s UID:
ls -ln /home/<user>/work/labpod-uid-check.txtDo not rename the image account as a workaround. If a template needs a different visible username, build a custom image that creates that account consistently.
Admin Resources page shows untracked GPU usage
Section titled “Admin Resources page shows untracked GPU usage”The indicator fires when nvidia-smi sees GPU processes that don’t belong to any LabPod
workspace cgroup. Usually:
- A user is running training directly on the host outside any container.
- A workspace was deleted but left a stray GPU process.
Find and free the process:
nvidia-smi --query-compute-apps=pid,process_name,used_memory,gpu_uuid --format=csvsudo kill -TERM <pid># After 30 seconds if still alive:sudo kill -KILL <pid># Run these as rootnvidia-smi --query-compute-apps=pid,process_name,used_memory,gpu_uuid --format=csvkill -TERM <pid># After 30 seconds if still alive:kill -KILL <pid>labpod admin doctor reports a problem
Section titled “labpod admin doctor reports a problem”Read the message and fix accordingly. doctor checks:
| Check | Fix |
|---|---|
| Required command binaries | Install the missing package or re-run the public install script for LabPod-managed files |
| cgroup mode | Upgrade to cgroup v2 or accept degraded CPU/MEM enforcement |
| Fractional GPU library | Verify LABPOD_GPU_SHARING_LIB_PATH points to the correct file |
Workspace helper tree (/opt/labpod/inject) | Re-run the public install script to restore helper files |
| GPU process inspector | Install nvidia-smi or set LABPOD_GPU_PROCESS_INSPECTOR=none |
| Port guard | Confirm nft is installed and the service runs as root |
| TLS certificate | Check LABPOD_TLS_CERT/LABPOD_TLS_KEY paths or LABPOD_TLS_SELF_SIGNED |
Service won’t start after upgrade
Section titled “Service won’t start after upgrade”Check the journal first:
sudo journalctl -u labpod -n 200 --no-pagersudo labpod admin doctor# Run these as rootjournalctl -u labpod -n 200 --no-pagerlabpod admin doctorIf a migration is failing:
sudo labpod admin --db /var/lib/labpod/labpod.db migrate # idempotent; safe to retry# Run these as rootlabpod admin --db /var/lib/labpod/labpod.db migrate # idempotent; safe to retryIf the migration itself is broken, restore the pre-upgrade-*.db snapshot created by the
install script during the upgrade:
sudo systemctl stop labpodsudo labpod admin --db /var/lib/labpod/labpod.db restore \ /var/lib/labpod/backups/pre-upgrade-<stamp>.dbsudo systemctl start labpod# Run these as rootsystemctl stop labpodlabpod admin --db /var/lib/labpod/labpod.db restore \ /var/lib/labpod/backups/pre-upgrade-<stamp>.dbsystemctl start labpodIf the installer stops before downtime with an invalid configuration message, repair the named
setting and re-run it. The preflight checks configuration paths, TLS material, GPU-sharing library,
inject tree, and startup booleans before replacing the executable or stopping the API. Run
install.sh --check to report all configuration problems in one pass.
Disk pressure
Section titled “Disk pressure”Check what’s consuming space:
du -sh /var/lib/labpod/ # database + backups (usually small, <100 MB)du -sh /var/lib/labpod/backups/ # backup rotation directorysudo podman system df # root's image layerssudo -u <user> podman system df # per-user image layers# Run these as rootdu -sh /var/lib/labpod/ # database + backups (usually small, <100 MB)du -sh /var/lib/labpod/backups/ # backup rotation directorypodman system df # root's image layersrunuser -u <user> -- podman system df # per-user image layersLargest consumers:
- Per-user container images in
~/.local/share/containers/ - Hugging Face caches in
~/work/.hf-cache/
To reclaim unused image layers for a user:
sudo -u <user> podman image prune -f# Run these as rootrunuser -u <user> -- podman image prune -fOr use the admin Images page in LabPod to prune all unused layers at once.
Version mismatch
Section titled “Version mismatch”If curl http://127.0.0.1:24680/api/version returns a different SHA than
labpod --version, the binary was replaced but the service was not restarted:
sudo systemctl restart labpod# Run these as rootsystemctl restart labpod