> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloud.vessl.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Fine-tune Gemma 4
> A complete walkthrough for fine-tuning Gemma 4 E4B with QLoRA and Unsloth on a single A100 GPU.
This guide walks you through fine-tuning **Gemma 4 E4B** using **QLoRA** and **Unsloth** on VESSL Cloud. By the end, you will have a fine-tuned adapter saved to shared storage, ready for inference or team collaboration.
## Prerequisites
Before starting, make sure you have:
* A VESSL Cloud account with credits ([sign up](https://cloud.vessl.ai/~/signup))
* An organization with access to **A100 SXM 80 GB** GPU instances
* Basic familiarity with Python and Hugging Face Transformers
New to VESSL Cloud? Complete the [Member quickstart](/guides/get-started/quickstart-user) first to set up your account, payment, and storage.
## Create a workspace
You need two types of storage for this workflow:
| Storage type | Mount path | Purpose |
| ------------------- | ---------- | -------------------------------------------------------------------------------------- |
| **Cluster storage** | `/root` | Home directory. Pip packages and conda environments persist across workspace restarts. |
| **Object storage** | `/shared` | Model checkpoints and outputs. Accessible from any cluster, shareable with teammates. |
**Why Cluster storage at `/root`?** Your home directory (`$HOME`) is where pip installs packages by default. Mounting Cluster storage here means you only run `pip install` once -- packages survive workspace pause/resume cycles.
**Why Object storage at `/shared`?** Fine-tuned model weights need to be accessible from other workspaces or clusters. Object storage is S3-backed and reachable from anywhere, making it easy to share results with your team or deploy from a different region.
Create both volumes before launching the workspace:
* **Cluster storage**: Go to **Cluster storage** in the sidebar and click **Create new volume**. See [Storage overview](/admin/storage/overview) for details.
* **Object storage**: Go to **Object storage** in the sidebar and click **Create new volume**. See [Create a volume](/member/volume/create) for details.
Do not mount Object storage at `/root`. Object storage is slower than Cluster storage and is not suitable as your primary workspace path. Use `/shared` or another separate mount point.
Create a new workspace with the following configuration:
| Setting | Value |
| ------------------- | --------------------------------------------- |
| **GPU** | A100 SXM 80 GB |
| **GPU count** | 1 |
| **Image** | `pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel` |
| **Cluster storage** | Your cluster volume mounted at `/root` |
| **Object storage** | Your object volume mounted at `/shared` |
See [Create a workspace](/member/workspace/create) for the full creation flow.
4-bit quantization (QLoRA) keeps VRAM usage around **18-22 GB** for the E4B model, well within the 80 GB available on an A100. This leaves headroom for larger batch sizes or longer sequences if needed.
Once the workspace shows **Running**, connect using JupyterLab or SSH. See [Connect to a workspace](/member/workspace/connect).
## Install packages
Open a terminal in your workspace and install the required libraries:
```bash theme={null}
pip install unsloth trl transformers datasets
```
If you mounted Cluster storage at `/root`, these packages persist across workspace pause/resume. You only need to run this once.
## Load the model
```python theme={null}
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
model_name="unsloth/gemma-4-E4B-it",
max_seq_length=2048,
load_in_4bit=True,
full_finetuning=False,
)
```
**What `load_in_4bit=True` does:** Instead of loading each parameter as a 16-bit float (the default), 4-bit quantization compresses the weights to 4 bits using the QLoRA (NF4) technique. This reduces the model's memory footprint by roughly 4x, allowing a model that would normally require \~32 GB of VRAM to fit in \~8-10 GB. The quality loss is minimal because only the frozen base weights are quantized -- the LoRA adapter trains in full precision.
Peak VRAM running this guide on an A100 80 GB — about 11 GB (13.7%). Plenty of headroom to grow the batch size or sequence length.
## Configure LoRA adapter
```python theme={null}
model = FastModel.get_peft_model(
model,
finetune_vision_layers=False,
finetune_language_layers=True,
finetune_attention_modules=True,
finetune_mlp_modules=True,
r=8,
lora_alpha=8,
lora_dropout=0,
bias="none",
random_state=3407,
)
```
### Hyperparameter reference
| Parameter | Value | Explanation |
| ---------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `r` | `8` | **LoRA rank** -- controls the capacity of the low-rank adapter. Higher values (16, 32) capture more complex patterns but use more memory and risk overfitting on small datasets. Start with 8 for general tasks. |
| `lora_alpha` | `8` | **Scaling factor** -- controls how much the adapter output is amplified. Typically set equal to `r`. The effective learning rate of the adapter scales as `lora_alpha / r`. |
| `lora_dropout` | `0` | **Dropout rate** -- probability of zeroing adapter outputs during training. Set to 0 for small datasets; increase to 0.05-0.1 if you see overfitting on larger datasets. |
| `finetune_vision_layers` | `False` | Gemma 4 is a multimodal model. Set to `False` for text-only training to skip vision encoder layers entirely. |
| `finetune_language_layers` | `True` | Train the language model layers. |
| `finetune_attention_modules` | `True` | Train the attention (Q, K, V, O) projection matrices. |
| `finetune_mlp_modules` | `True` | Train the feed-forward (MLP) layers. Together with attention modules, this covers the most impactful parts of the model. |
| `bias` | `"none"` | Do not train bias terms. Keeps the adapter small without measurable quality loss. |
| `random_state` | `3407` | Random seed for reproducibility. |
**When to increase `r`:**
* **Complex domain adaptation** (medical, legal, code): try `r=16`
* **Large, diverse datasets** (100k+ samples): try `r=16` or `r=32`
* **Simple style transfer or format following**: `r=8` is usually sufficient
## Prepare the dataset
This example uses the **FineTome-100k** dataset with 3,000 samples for a quick demo. For production use, train on the full dataset or substitute your own data.
```python theme={null}
from unsloth.chat_templates import get_chat_template, standardize_data_formats
from datasets import load_dataset
tokenizer = get_chat_template(tokenizer, chat_template="gemma-4")
dataset = load_dataset("mlabonne/FineTome-100k", split="train[:3000]")
dataset = standardize_data_formats(dataset)
def formatting_prompts_func(examples):
convos = examples["conversations"]
texts = [
tokenizer.apply_chat_template(
convo, tokenize=False, add_generation_prompt=False
).removeprefix("")
for convo in convos
]
return {"text": texts}
dataset = dataset.map(formatting_prompts_func, batched=True)
```
Your dataset should be a JSON file where each entry has a `conversations` field -- a list of message objects with `role` and `content`:
```json theme={null}
[
{
"conversations": [
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."}
]
},
{
"conversations": [
{"role": "user", "content": "Explain photosynthesis briefly."},
{"role": "assistant", "content": "Photosynthesis is the process by which plants convert sunlight, water, and CO2 into glucose and oxygen."}
]
}
]
```
Load it with:
```python theme={null}
dataset = load_dataset("json", data_files="/shared/my-data.json", split="train")
dataset = standardize_data_formats(dataset)
dataset = dataset.map(formatting_prompts_func, batched=True)
```
The 3,000-sample subset is for demonstration only. For meaningful quality improvements, use the full 100k dataset or at least 10k-20k high-quality samples of your own data.
## Train
```python theme={null}
from trl import SFTTrainer, SFTConfig
from unsloth.chat_templates import train_on_responses_only
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
args=SFTConfig(
dataset_text_field="text",
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
warmup_steps=5,
max_steps=60,
learning_rate=2e-4,
logging_steps=1,
optim="adamw_8bit",
weight_decay=0.001,
lr_scheduler_type="linear",
seed=3407,
report_to="none",
output_dir="/shared/gemma4-finetuned",
),
)
trainer = train_on_responses_only(
trainer,
instruction_part="<|turn>user\n",
response_part="<|turn>model\n",
)
trainer_stats = trainer.train()
```
### Training parameter reference
| Parameter | Value | Explanation |
| ----------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `per_device_train_batch_size` | `1` | Number of samples per GPU per forward pass. Kept at 1 to fit within VRAM. |
| `gradient_accumulation_steps` | `4` | Accumulate gradients over 4 steps before updating weights. **Effective batch size = 1 x 4 = 4.** |
| `warmup_steps` | `5` | Linearly ramp up learning rate for the first 5 steps to stabilize early training. |
| `max_steps` | `60` | Total training steps. This is for demo purposes -- for full training runs, remove `max_steps` and set `num_train_epochs=1` (or more). |
| `learning_rate` | `2e-4` | Standard learning rate for QLoRA fine-tuning. |
| `optim` | `"adamw_8bit"` | 8-bit AdamW optimizer. Uses quantized optimizer states to save \~2 GB of VRAM compared to standard AdamW. |
| `weight_decay` | `0.001` | L2 regularization to prevent overfitting. |
| `lr_scheduler_type` | `"linear"` | Linearly decay the learning rate to zero over training. |
| `report_to` | `"none"` | Disable external logging such as Weights & Biases. Set to `"wandb"` if you want experiment tracking. |
| `output_dir` | `"/shared/gemma4-finetuned"` | Save checkpoints to Object storage so they persist and are shareable. |
**What `train_on_responses_only` does:** By default, the loss is computed over the entire conversation (user + assistant turns). This option masks the user turns so the model only learns to generate the assistant responses. This improves training efficiency and prevents the model from memorizing user prompts.
## Evaluate
After training, check how the model performs qualitatively and quantitatively.
### Check training loss
The `trainer_stats` object contains the training log. A decreasing loss curve indicates the model is learning:
```python theme={null}
import json
# Print final training loss
print(f"Final training loss: {trainer_stats.training_loss:.4f}")
# Inspect per-step loss from the log history
for entry in trainer_stats.log_history[-5:]:
if "loss" in entry:
print(f" Step {entry['step']}: loss = {entry['loss']:.4f}")
```
A healthy loss curve starts high (2-3+) and decreases steadily. If loss plateaus very early, consider increasing `r` or using more data. If loss spikes or diverges, reduce the learning rate.
Actual 60-step training log from this guide. Starts around 2.37 and trends down — the textbook shape for a small SFT run.
### Compare before and after
Run the same prompt through the fine-tuned model to see the effect:
```python theme={null}
from transformers import TextStreamer
messages = [
{"role": "user", "content": [
{"type": "text", "text": "Explain quantum computing in simple terms."}
]}
]
_ = model.generate(
**tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to("cuda"),
max_new_tokens=256,
use_cache=True,
temperature=0.7,
streamer=TextStreamer(tokenizer, skip_prompt=True),
)
```
For a more rigorous evaluation, split your data upfront and compute loss on the held-out portion:
```python theme={null}
# Before training, split the dataset
split = dataset.train_test_split(test_size=0.1, seed=3407)
train_dataset = split["train"]
eval_dataset = split["test"]
# Pass eval_dataset to SFTTrainer
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
args=SFTConfig(
# ... same args as above, plus:
eval_strategy="steps",
eval_steps=20,
# ...
),
)
```
This reports evaluation loss at regular intervals, letting you detect overfitting (training loss decreases but eval loss increases).
## Save the model
Save the fine-tuned adapter and tokenizer to Object storage:
```python theme={null}
model.save_pretrained("/shared/gemma4-finetuned/final")
tokenizer.save_pretrained("/shared/gemma4-finetuned/final")
```
Since `/shared` is backed by Object storage, the saved model is:
* **Persistent** -- survives workspace termination
* **Cross-cluster** -- accessible from workspaces in any region
* **Team-shareable** -- any team member with access to the volume can load the adapter
Adapter and tokenizer files saved to `/shared/gemma4-finetuned/final/` after training. Backed by Object storage, so any teammate's workspace — in any region — can mount and load them as-is.
To load the adapter later from another workspace:
```python theme={null}
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
model_name="unsloth/gemma-4-E4B-it",
max_seq_length=2048,
load_in_4bit=True,
full_finetuning=False,
)
from peft import PeftModel
model = PeftModel.from_pretrained(model, "/shared/gemma4-finetuned/final")
```
## Gemma 4 model comparison
Choose the right Gemma 4 variant based on your GPU and task complexity:
| Model | Parameters | Recommended GPU | QLoRA VRAM estimate | Best for |
| --------------- | ---------- | -------------------------- | ------------------- | ----------------------------------------- |
| **Gemma 4 E2B** | 2B | T4 16 GB, L4 24 GB | \~6-8 GB | Quick experiments, edge deployment |
| **Gemma 4 E4B** | 4B | A100 40/80 GB, L4 24 GB | \~10-14 GB | Best balance of quality and efficiency |
| **Gemma 4 12B** | 12B | A100 80 GB | \~18-24 GB | Higher quality, single-GPU fine-tuning |
| **Gemma 4 27B** | 27B | A100 80 GB (tight), 2xA100 | \~32-40 GB | Near-frontier quality, requires more VRAM |
VRAM estimates assume 4-bit quantization with QLoRA, batch size 1, and sequence length 2048. Actual usage varies with batch size, sequence length, and gradient accumulation settings.
## Next steps
* **Get the runnable recipe** -- A packaged version of this workflow (an interactive notebook plus a `vesslctl` batch-job path, trained on a small domain QA dataset) lives in the [gemma4-finetuning cookbook recipe](https://github.com/vessl-ai/vessl-cloud-cookbook/tree/main/gemma4-finetuning), with measured cost and VRAM benchmarks.
* **Use your own data** -- Replace FineTome-100k with domain-specific conversation data for targeted improvements.
* **Try DPO or ORPO** -- After SFT, apply preference optimization (DPO/ORPO) to further align the model with desired behavior.
* **Scale up** -- Move to the 12B or 27B model for higher quality. Use `r=16` or `r=32` with larger datasets.
* **Export to GGUF** -- Convert the fine-tuned model to GGUF format for local inference with llama.cpp or Ollama.
* **Deploy as a batch job** -- Use [VESSL Cloud batch jobs](/member/job/overview) to run fine-tuning as a scheduled, reproducible pipeline.
* **Automate with vesslctl** -- Turn this interactive run into a one-line command with [`vesslctl job create`](/cli/quickstart). Swap the Jupyter cells for a `train.py` and submit it from your terminal or a CI pipeline.