Skip to content
LabPod LabPod LabPod Docs

Monitoring & resources

The Resources page

Section titled “The Resources page”

Open /admin/resources to see a live snapshot of host utilization:

Admin Resources page - host-wide totals, per-user breakdown table, and GPU card with utilization and memory

  • Per-user breakdown - CPU cores, memory, GPU fraction, and disk used vs. its advisory alert threshold for every user with at least one running workspace.
  • GPU cards - current utilization percentage, temperature, and memory used per physical GPU.
  • Image storage - per-user image disk consumption.
  • Idle candidates - running workspaces that haven’t had recent CPU, GPU, or terminal activity (based on the idle_window_seconds threshold).

The page polls the server every few seconds; click Refresh to force an immediate sample.

Untracked GPU usage

Section titled “Untracked GPU usage”

If nvidia-smi sees GPU compute processes that don’t belong to any LabPod workspace, the Resources page shows a warning under the affected GPU. This usually means:

  • A user is running training directly on the host outside any container.
  • A workspace was deleted but a child process retained GPU memory.

To investigate:

Terminal window
nvidia-smi --query-compute-apps=pid,process_name,used_memory,gpu_uuid --format=csv

To free:

Terminal window
sudo kill -TERM <pid>
# If still running after 30 seconds:
sudo kill -KILL <pid>
Terminal window
# Run these as root
kill -TERM <pid>
# If still running after 30 seconds:
kill -KILL <pid>

See GPU configuration for how to tune or disable the inspector.

Disk-pressure banner

Section titled “Disk-pressure banner”

When the configured disk-capacity filesystem crosses 90% used, the admin dashboard shows a disk-pressure banner. When LABPOD_DISK_ROOT is unset, this is the filesystem containing LABPOD_FILEOP_HOME_BASE, not necessarily /home. Act before running workspaces start failing on checkpoint writes.

Disk hygiene commands:

Terminal window
du -sh /var/lib/labpod/              # database + backups
du -sh /var/lib/labpod/backups/      # backup files only
sudo podman system df                # root's image storage
sudo -u <user> podman system df      # per-user image storage (run once per user)
Terminal window
# Run these as root
du -sh /var/lib/labpod/              # database + backups
du -sh /var/lib/labpod/backups/      # backup files only
podman system df                # root's image storage
runuser -u <user> -- podman system df      # per-user image storage (run once per user)

Largest disk consumers in a typical deployment:

  • Container images in each user’s ~/.local/share/containers/storage/
  • Hugging Face model caches in ~/work/.hf-cache/ (LabPod sets HF_HOME there)
  • Datasets in shared mount directories

Usage reports

Section titled “Usage reports”

The admin Usage page (/admin/usage) shows per-user daily summaries and lets you export CSVs:

  • /api/usage/users/{serverUserID} - one user’s daily stats
  • /api/usage/users/{serverUserID}.csv - CSV download
  • /api/usage/all.csv - all users in one CSV

Reporting periods and CSV day values use UTC. Raw usage samples are retained for at least 25 hours (48 by default); daily rollups are kept indefinitely. The chart provides the most recent 12 months and the year/month selectors and chart bars load a month’s daily detail. Adjust raw retention via the runtime setting usage_sample_retention_hours.

Audit logs

Section titled “Audit logs”

The Audit page (/admin/audit) shows a chronological log of every mutation:

  • User creation and deletion
  • Workspace starts, stops, and owner-initiated deletions; administrator force-stops
  • Template and policy changes
  • Image pulls and builds
  • Login failures

Correlate login failures with the auth.login_failed journal entries:

Terminal window
sudo journalctl -u labpod --since "1 hour ago" | grep login_failed
Terminal window
# Run these as root
journalctl -u labpod --since "1 hour ago" | grep login_failed

Audit history is retained indefinitely by default. An administrator can set a positive audit-log retention window in Admin → Runtime settings when the installation’s retention policy permits permanent removal of older audit records.