Skip to content
LabPod LabPod LabPod Docs

GPU configuration

LabPod supports three GPU allocation modes. You configure each mode per physical GPU from the admin Resources page → GPU Policies.

Whole GPU (Container Device Interface, CDI)

Section titled “Whole GPU (Container Device Interface, CDI)”

Each workspace that requests a GPU gets one full physical GPU. No sharing - one workspace per GPU at a time.

Prerequisites:

  1. NVIDIA driver 535.54.03 or newer installed and nvidia-smi working.
  2. nvidia-container-toolkit installed (the install script does this).
  3. A Container Device Interface (CDI) spec at /etc/cdi/nvidia.yaml (the script generates it via nvidia-ctk cdi generate).

Admin policy: Set the GPU policy to whole on the Resources page, or leave it on auto (which allows whole-GPU requests by default).

When a workspace is started with GPU mode = Whole GPU, LabPod passes --device nvidia.com/gpu=<index> to Podman. The workspace sees the full GPU.

Fractional GPU sharing

Section titled “Fractional GPU sharing”

Multiple workspaces share a single physical GPU concurrently. Each workspace declares how much GPU memory and compute it needs; the server tracks allocations and refuses new workspaces that would exceed the physical GPU’s capacity.

Prerequisites:

Fractional GPU is off by default — the core install does not build the sharing library. A GPU sharing runtime library (libvgpu.so) must be present on the host. The supported way to build and install it is the public installer, which compiles HAMi-core inside a Podman container (no host build tools required) and wires LABPOD_GPU_SHARING_LIB_PATH for you:

Terminal window
curl -fsSL https://labpod.ai/install.sh | sudo bash -s -- --with-hami
Terminal window
# Run these as root
curl -fsSL https://labpod.ai/install.sh | bash -s -- --with-hami

The build pulls a ~3.3 GB CUDA image; it is a one-time, build-time cost you can reclaim with podman image rm afterward.

To use a library you built or obtained elsewhere, point LabPod at it directly:

Terminal window
# In /etc/labpod/labpod.env:
LABPOD_GPU_SHARING_LIB_PATH=/path/to/libvgpu.so

The server refuses to start with this variable set if the file doesn’t exist.

LabPod GPU hosts require NVIDIA driver 535.54.03 or newer.

Admin policy: Set the GPU policy for the target GPU to Fractional on the Resources page.

User workspace creation: The user selects GPU mode = Fractional GPU, then sets the memory (MB) and compute percentage they need. The server checks that the sum across all running workspaces on that GPU stays within physical limits.

Verify:

Terminal window
sudo labpod admin doctor
# Look for: GPU sharing runtime: OK  (or a "not found" error with the path)
Terminal window
# Run these as root
labpod admin doctor
# Look for: GPU sharing runtime: OK  (or a "not found" error with the path)

MIG slices (A100/H100)

Section titled “MIG slices (A100/H100)”

NVIDIA Multi-Instance GPU (MIG) partitions a physical GPU into independent slices at the hardware level. LabPod reads the current slice inventory and assigns one slice per workspace.

Prerequisites:

  • An NVIDIA A100, H100, H200, or similar MIG-capable GPU.
  • MIG mode enabled and slices pre-configured by the operator using nvidia-smi:
Terminal window
# Example: enable MIG mode and create slices on GPU 0
sudo nvidia-smi -i 0 -mig 1
sudo nvidia-smi mig -i 0 -cgi 1g.10gb,1g.10gb,2g.20gb -C
Terminal window
# Run these as root
# Example: enable MIG mode and create slices on GPU 0
nvidia-smi -i 0 -mig 1
nvidia-smi mig -i 0 -cgi 1g.10gb,1g.10gb,2g.20gb -C

Admin policy: Set the GPU policy for the target GPU to mig. LabPod reads slices at startup and on demand; no extra variable is needed.

MIG inventory source (optional override):

Terminal window
# In /etc/labpod/labpod.env:
LABPOD_MIG_INVENTORY=nvidia-smi     # default: parse nvidia-smi -L
LABPOD_MIG_INVENTORY=none           # no MIG on this host

Untracked GPU usage indicator

Section titled “Untracked GPU usage indicator”

The admin Resources page shows a warning when nvidia-smi detects GPU compute processes that don’t belong to any LabPod workspace. This usually means someone is running training directly on the host (outside any container), or a process remained after a workspace was deleted.

GPU process inspector:

Terminal window
# In /etc/labpod/labpod.env:
LABPOD_GPU_PROCESS_INSPECTOR=nvidia-smi    # default
LABPOD_GPU_PROCESS_INSPECTOR=none          # disable the check

Bypass at admission time (not recommended):

Terminal window
LABPOD_ALLOW_GPU_DRIFT=1    # skip the untracked-usage gate for new workspaces

To free a held GPU:

Terminal window
nvidia-smi --query-compute-apps=pid,process_name,gpu_uuid --format=csv
sudo kill -TERM <pid>
# If still alive after 30 seconds:
sudo kill -KILL <pid>
Terminal window
# Run these as root
nvidia-smi --query-compute-apps=pid,process_name,gpu_uuid --format=csv
kill -TERM <pid>
# If still alive after 30 seconds:
kill -KILL <pid>

Verifying GPU configuration

Section titled “Verifying GPU configuration”
Terminal window
sudo labpod admin doctor
Terminal window
# Run these as root
labpod admin doctor

doctor checks that:

  • The CDI spec exists at /etc/cdi/nvidia.yaml (whole-GPU mode).
  • The fractional GPU sharing library is readable (if LABPOD_GPU_SHARING_LIB_PATH is set).
  • The GPU process inspector (nvidia-smi) is reachable for each enumerated GPU UUID.

For the full workstation preflight, use the install script’s --check mode.

GPU inventory source

Section titled “GPU inventory source”
Terminal window
# In /etc/labpod/labpod.env:
LABPOD_GPU_INVENTORY=nvidia-smi     # default: enumerate GPUs via nvidia-smi
LABPOD_GPU_INVENTORY=none           # driverless host - no GPU workspaces

Use none on a CPU-only workstation to skip nvidia-smi calls and suppress spurious errors in the Resources page.