> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloud.vessl.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Metrics
> GPU and system time-series metrics for every node in your cluster.
Each node exposes a per-GPU overview plus time-series charts for GPU, system, network, and InfiniBand metrics. Open a node's detail page from the **Active** cluster → **Node management** tab → click a node.
## GPU overview
Each node lists its 8 GPUs with utilization, VRAM, temperature, power, and a per-GPU health badge. See [Health checks](/vm-clusters/health-checks) for what each badge state means.
## Time range
Pick a time range from **1h, 6h, 12h, 1d, 7d, or 30d**. For deeper analysis, open the linked Grafana dashboard.
## Metric charts
Metrics are grouped into **6 sections with 25 charts total**.
### Utilization (2 charts)
Track whether GPUs are actively executing tasks and how much memory each one holds.
| Chart | What it shows |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **GPU Utilization** | Percentage of time the GPU is executing tasks. Low utilization may indicate over-provisioned GPU resources or that upstream vCPUs cannot keep the GPU fed. |
| **GPU Memory Used** | Amount of frame buffer (VRAM) currently in use on each GPU. A sudden drop to **0** indicates a process crash inside the VM; a steady upward trend may signal a memory leak. |
### System (7 charts)
Spot bottlenecks outside the GPU — vCPU saturation, memory pressure, disk, network, or the VM's overall health.
| Chart | What it shows |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **CPU Usage** | Percentage of time the VM's vCPUs spend executing tasks. Sustained high values indicate vCPUs are saturated; if GPUs are idle at the same time, vCPUs are likely the bottleneck. |
| **Load Average (5m)** | Average number of processes waiting on vCPUs inside the VM over the last **5 minutes**. Sustained high values may indicate VM overload. |
| **System Memory Usage** | Percentage of the VM's allocated RAM in use. Sustained high values may trigger the OOM (Out of Memory) killer inside the VM, and swap usage causes severe slowdowns. |
| **Root Disk Usage** | Percentage of the VM's root filesystem (`/`) capacity in use. Near capacity may cause write failures for any process needing disk space. |
| **Network RX** | Inbound throughput on the VM's primary ethernet interface. Sustained **0 B/s** during expected activity may indicate an ethernet outage, separate from the InfiniBand fabric. |
| **Network TX** | Outbound throughput on the VM's primary ethernet interface. Carries control-plane traffic such as API calls and logs, separate from the InfiniBand fabric used for high-bandwidth workloads. |
| **Node Uptime** | Tracks node uptime over time. **1** = up (no critical alerts firing), **0** = down (critical alert firing such as XID errors, ECC DBE, or IB link down). Gaps mean the node stopped reporting entirely. |
### Temperature & power (3 charts)
Watch for thermal throttling and power-related instability.
| Chart | What it shows |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **GPU Temperature** | GPU chip temperature in degrees Celsius. Sustained values approaching the GPU's thermal limit trigger thermal throttling. |
| **Memory Temperature** | HBM (High Bandwidth Memory) temperature in degrees Celsius. Cross-check with GPU temperature to identify the thermal hotspot. |
| **Power Usage** | Current power draw per GPU in watts. Active compute draws close to the GPU's rated TDP; **0 W** typically indicates the VM does not recognize the GPU, and sharp swings may signal unstable load. |
### Memory & clock detail (3 charts)
Identify throttling and HBM activity issues.
| Chart | What it shows |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Memory Utilization** | Percentage of time the GPU's memory engine is active. This measures activity time, not bandwidth (GB/s) — **100%** doesn't necessarily mean max throughput. |
| **Memory Clock** | Current frequency of the GPU's HBM memory clock. Values below the base clock may indicate throttling. Cross-check with **Temperature & power**. |
| **SM Clock** | Current frequency of the SM (Streaming Multiprocessor) clock. Values below the base clock may indicate power or thermal throttling. |
### ECC & errors (4 charts)
Catch GPU hardware faults early — before they take down a training run.
| Chart | What it shows |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **ECC SBE (Correctable)** | Single-bit memory errors corrected by hardware in the last **5 minutes**. Occasional events are normal, but a sharp upward trend may precede uncorrectable errors. |
| **ECC DBE (Uncorrectable)** | Uncorrectable memory errors in the last **5 minutes**. The healthy count is **0**; a single event may corrupt in-flight computation or crash the process using the GPU. |
| **Remapped Rows (Correctable Errors)** | Correctable HBM row remap events in the last **5 minutes**. An upward trend may signal HBM degradation. |
| **Remapped Rows (Uncorrectable Errors)** | Uncorrectable HBM row remap events in the last **5 minutes**. The healthy count is **0**; any occurrence is critical and may require GPU replacement. |
### InfiniBand detail (6 charts)
Monitor the node-to-node fabric used by multi-node distributed training — throughput, errors, and link stability. Each chart is plotted per HCA (Host Channel Adapter).
| Chart | What it shows |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **IB TX Throughput** | Outbound InfiniBand throughput per HCA attached to the VM. Significant imbalance between HCAs may indicate a cable or switch issue; sustained **0 Gbps** may indicate the link is down. |
| **IB RX Throughput** | Inbound InfiniBand throughput per HCA on the VM. Asymmetry against TX may indicate routing issues; per-HCA lines help isolate a faulty port. |
| **IB Receive Errors** | InfiniBand receive errors per HCA in the last **5 minutes**. Sustained values above **0** may indicate cable or port damage, disrupting multi-VM workloads. |
| **IB Symbol Errors** | InfiniBand symbol errors on the VM's HCAs in the last **5 minutes**. May indicate cable aging or optical module issues. |
| **IB Link Error Recovery** | InfiniBand link recovery events on the VM's HCAs in the last **5 minutes**. A rising trend signals link instability and may escalate to **Link Downed**. |
| **IB Link Downed** | InfiniBand link-down events per HCA in the last **5 minutes**. The healthy count is **0**; any occurrence disconnects the VM from the IB fabric. |