When a workspace stops

Workspaces normally keep running until you or an admin stops them. But there are a few situations where a workspace stops or appears degraded on its own.

OOM kill - exceeded the memory limit

If the kernel kills the workspace for memory pressure, the workspace card and detail page show an OOM notice. The notice does not prove that the selected workspace limit was exceeded: a host-wide memory shortage can produce the same outcome.

OOM killed

What to do:

• Reduce memory usage: smaller batch size, fewer DataLoader workers, less data loaded at once.
• Or ask your admin to check both the workspace limit and host-wide memory pressure, then click Start again.

cgroup v1 hosts

On a server running the older cgroup v1 system, memory limits are not enforced by LabPod. If your workspace is OOM-killed there, it means the entire host ran out of memory - raising your workspace memory setting will not help. Coordinate with other users or wait for free memory, then Start again.

Unexpected exit

If the container’s main process ends on its own (not from a Stop action), the workspace transitions to stopped with:

Exited unexpectedly (code N)

This usually means the template’s primary process crashed or was shut down from inside the container. Check the Container logs on the workspace detail page, then Start again.

Process-level OOM (training killed, workspace still running)

A container can survive while a process inside it is OOM-killed. If your training script dies with Killed in the terminal but the workspace stays running, this is what happened - the kernel killed the Python process, not the container itself.

Check with dmesg | tail inside the terminal or look at the launcher logs. The fix is the same: reduce memory use.

Host reboot

Workspaces do not auto-restart when the server reboots. After a reboot:

• Every workspace that was running shows as stopped with the status stopped by host restart.
• A banner on the workspace card tells you it was stopped by a reboot.
• In-memory state is gone: running kernels, training processes, terminal scrollback.
• Click Start to bring the workspace back.

Save checkpoints to /work for anything you can’t afford to lose - /work is a host directory and survives a reboot (the data is on disk, not in the container).

Idle nudge badge

After a workspace has been inactive for several days (default is 5 continuous idle days, set by your admin), an idle nudge badge appears on the workspace card on the home page. This is display-only - nothing is automatically stopped or deleted. It’s a reminder that the GPU might be available for others if you’re done with the workspace.

Admin force-stop

An admin can force-stop (but not delete) any workspace at any time, for example to reclaim a GPU that has been idle for a long time. Deleting a workspace is owner-only. If your workspace stops unexpectedly, check with your admin.

Recovering a workspace that stopped

For any of the above, the recovery path is the same:

1. Read the stop reason on the workspace card or detail page.
2. Fix the underlying problem (reduce memory, save data, wait for the reboot to complete).
3. Click Start.

Your files in /work and your workspace home are intact; only in-memory state is lost.