VESSL Cloud Documentation

Every node in your cluster is health-checked continuously. Each GPU gets its own status, and node-level conditions (memory, disk, kernel modules, InfiniBand, observability) roll up into a single status for the node, refreshed about once a minute.

Node overview

The cluster detail page shows a Node overview strip with every node as a colored cell, so you can read the whole cluster’s health at a glance.

Statuses

Status	Color	Meaning
Healthy	🟢	No active alarms.
Warning	🟡	At least one warning-level alarm. The node is usable but something needs attention.
Critical	🔴	At least one critical alarm. The node may be unusable or producing bad results.
Unknown	⚪	We can’t judge the node right now — see Unknown below.

A node’s status is worst-wins: it takes the highest severity across all of its alarms and per-GPU statuses. Having more warnings does not escalate the node to Critical on its own.

Levels

Health is tracked at two levels — each GPU has its own status, and the node has an overall status that combines node-level checks with the worst per-GPU status.

Level	Where you see it	What’s checked
GPU (per device)	The 8-GPU grid on the node detail page	XID errors, ECC, temperature, throttling
Node	Node overview cell and the node row dot	Memory, disk, kernel modules, InfiniBand fabric, observability gaps

GPU checks

Each GPU has its own status — the worst condition among the checks below. The node picks up the worst GPU’s color using worst-wins.

Code	Condition	Status
`gpu_high_temp`	GPU temperature ≥ 85°C (sustained)	🟡 Warning
`gpu_high_temp`	GPU temperature ≥ 95°C	🔴 Critical
`gpu_memory_high_temp`	HBM (memory) temperature ≥ 85°C	🟡 Warning
`gpu_memory_high_temp`	HBM temperature ≥ 95°C	🔴 Critical
`gpu_hw_throttle`	Hardware throttling	🟡 Warning
`gpu_no_process_high_util`	High GPU utilization with no attached process	🟡 Warning
`row_remap_nearing_limit`	HBM row remaps approaching the safety limit	🟡 Warning
`gpu_ecc_dbe`	Uncorrectable (DBE) ECC error	🔴 Critical
`gpu_row_remap_failure`	HBM row remap failure	🔴 Critical
`gpu_remapped_rows_pending`	HBM row remaps pending (reboot required)	🔴 Critical
`gpu_count_mismatch`	Fewer than 8 GPUs detected by the driver	🔴 Critical
`gpu_smi_unhealthy`	GPU reported unhealthy by `nvidia-smi`	🔴 Critical
`gpu_driver_pri_bus_fault`	GPU driver / PCIe bus fault	🔴 Critical
`gpu_recovery_action_required`	GPU recovery action required	🔴 Critical

XID errors

An XID is an error report from the NVIDIA driver. The codes below signal a hardware, driver, or firmware fault serious enough for VESSL to mark the GPU Critical. Codes not listed here don’t change the GPU’s status.

XID	Code	Meaning	Status
38	`gpu_xid38_driver_firmware_mismatch`	Driver / firmware mismatch	🔴 Critical
48	`gpu_xid48_dbe_row_remap`	DBE row remap	🔴 Critical
62	`gpu_xid62_internal_fw_breakpoint`	Internal firmware breakpoint	🔴 Critical
64	`gpu_xid64_ecc_row_remap_failure`	ECC row remap failure	🔴 Critical
74	`gpu_xid74_nvlink_error`	NVLink error	🔴 Critical
79	`gpu_xid79_fallen_off_bus`	GPU fallen off the bus	🔴 Critical
95	`gpu_xid95_uncontained_ecc_reboot`	Uncontained ECC	🔴 Critical
110	`gpu_xid110_security_fault`	Security fault	🔴 Critical
119	`gpu_xid119_gsp_rpc_timeout`	GSP RPC timeout	🔴 Critical
120	`gpu_xid120_gsp_error`	GSP error	🔴 Critical
121	`gpu_xid121_c2c_link_error`	C2C link error	🔴 Critical
123	`gpu_xid123_spi_pmu_rpc_write_failure`	SPI PMU RPC write failure	🔴 Critical
136	`gpu_xid136_link_training_failed`	Link training failed	🔴 Critical
140	`gpu_xid140_unrecovered_ecc`	Unrecovered ECC error	🔴 Critical
143	`gpu_xid143_gpu_init_failure`	GPU initialization failure	🔴 Critical
154	`gpu_xid154_recovery_action_changed`	GPU recovery action changed	🔴 Critical
167	`gpu_xid167_pcie_fatal_timeout`	PCIe fatal timeout	🔴 Critical
169	`gpu_xid169_sec2_halt`	Micro-controller halt (SEC2)	🔴 Critical
171	`gpu_xid171_dram_uncorrectable`	Uncorrectable DRAM error	🔴 Critical
172	`gpu_xid172_sram_uncorrectable`	Uncorrectable SRAM error	🔴 Critical

These additional codes also mark the GPU 🔴 Critical (grouped under gpu_xid_critical_misc, shown with the specific XID number): 9, 12, 16, 18, 19, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 42, 44, 46, 47, 59, 60, 61, 65, 68, 69, 78. For the full list of XID codes, see the NVIDIA XID documentation. On the node detail page, each XID alert links to its catalog entry.

Node checks

These contribute to the node status. They affect the node dot and the Node overview cell — they don’t change any individual GPU’s status. Hover the node’s health indicator on the cluster detail page to see which specific alarm is firing.

System and kernel

Code	Condition	Status
`memory_low`	Low system memory	🟡 Warning
`disk_low`	Low root disk space	🟡 Warning
`peermem_not_loaded`	`peermem` kernel module not loaded	🔴 Critical

InfiniBand

Code	Condition	Status
`ib_symbol_errors`	Symbol errors on an InfiniBand HCA (physical-layer corruption)	🟡 Warning
`ib_link_flap` / `ib_storage_link_flap`	Link flap on an InfiniBand HCA	🟡 Warning
`ib_transport_retries_exceeded`	Transport retries exceeded on multiple InfiniBand HCAs	🟡 Warning
`ib_port_down`	One or more InfiniBand ports are down (multi-node paths fail)	🔴 Critical

Observability gaps

When part of a node’s health data can’t be collected, the node shows Warning with one of these labels:

Label	What it means	Status
Node unreachable	The node isn’t responding, so its health can’t be checked.	🟡 Warning, escalating to 🔴 Critical the longer it stays unreachable
GPU metrics unavailable	GPU metrics aren’t being collected, so GPU health can’t be checked.	🟡 Warning
InfiniBand metrics unavailable	InfiniBand metrics aren’t being collected, so link health can’t be checked.	🟡 Warning

Partial data loss does not turn the node Unknown — gray is reserved for the cases below.

Unknown (gray)

A node is Unknown (⚪) only when its health can’t be judged at all. Two cases:

The node is in Rebooting state. We don’t judge a node until it’s fully running. This is normal during reboot or initial provisioning.
The node isn’t set up for monitoring at all. Uncommon; if a running node stays this way, contact support.

In both cases, every cell on the node detail page is gray and the node dot is gray.

When a node is unhealthy

Reboot the node — often clears transient faults.
Wait for recovery — our engineers monitor nodes and recover them manually. During Beta there’s no recovery-time SLA.
Contact support if the problem persists.

If a node fails due to hardware, our engineers may replace it — replacement is a manual decision, not automatic. There’s no fixed maintenance schedule today; if maintenance is needed, we’ll notify you in advance.

Get started

Examples

vesslctl CLI

Use VESSL Cloud

Operate organization

VM Clusters

Glossary

Support

Health checks

Node overview

Statuses

Levels

GPU checks

XID errors

Node checks

System and kernel

InfiniBand

Observability gaps

Unknown (gray)

When a node is unhealthy

​Node overview

​Statuses

​Levels

​GPU checks

​XID errors

​Node checks

​System and kernel

​InfiniBand

​Observability gaps

​Unknown (gray)

​When a node is unhealthy

Node overview

Statuses

Levels

GPU checks

XID errors

Node checks

System and kernel

InfiniBand

Observability gaps

Unknown (gray)

When a node is unhealthy