Fine-Tuning LLMs on Custom Data: When, Why, and How

Fine-tuning is the process of taking a pre-trained language model and training it further on your specific dataset. While RAG retrieves external knowledge at inference time, fine-tuning embeds knowledge and behavior directly into the model's weights. The result is a model that speaks your domain's language natively, responds in your preferred style, and handles specialized tasks that general models struggle with.

When to Fine-Tune vs. When to RAG

Fine-tuning and RAG solve different problems. RAG excels when you need access to frequently updated information or when you need to cite specific sources. Fine-tuning excels when you need to change the model's behavior, style, or reasoning patterns. Need the model to always respond in a specific JSON format? Fine-tune. Need it to answer questions about yesterday's support tickets? RAG.

• Fine-tune for: consistent output format, domain-specific terminology, specialized reasoning, style adaptation
• Use RAG for: dynamic knowledge bases, source attribution, large document collections, frequently changing data
• Combine both for: the best of both worlds — fine-tuned behavior with retrieved context

Dataset Preparation

The quality of your fine-tuning dataset determines everything. You need high-quality examples of the exact input-output behavior you want. A curated dataset of 500 excellent examples often outperforms 50,000 mediocre ones. Each example should be in the conversation format the model expects, with clear system prompts, user messages, and ideal assistant responses.

{
  "messages": [
    {
      "role": "system",
      "content": "You are a cybersecurity analyst. Respond with structured threat assessments."
    },
    {
      "role": "user",
      "content": "Analyze this log entry: Failed SSH login from 192.168.1.105 — 47 attempts in 2 minutes"
    },
    {
      "role": "assistant",
      "content": "**Threat Assessment: SSH Brute Force Attack**\n\n- Severity: HIGH\n- Type: Credential Stuffing / Brute Force\n- Source: 192.168.1.105 (internal network)\n- Recommendation: Block IP immediately, investigate for lateral movement..."
    }
  ]
}

Garbage in, garbage out has never been more true than in fine-tuning. Spend 80% of your time on dataset quality and 20% on training hyperparameters.

LoRA: Fine-Tuning on a Budget

Full fine-tuning of a 7B+ parameter model requires serious GPU resources. LoRA (Low-Rank Adaptation) makes this accessible by training only a small number of additional parameters — typically less than 1% of the original model. The quality is remarkably close to full fine-tuning at a fraction of the compute cost. QLoRA goes further by quantizing the base model to 4-bit, enabling fine-tuning of 70B models on a single GPU.

Fine-tuning is a powerful tool when applied to the right problem. Start with a clear definition of what behavior you want to change, build a high-quality dataset, and use LoRA to keep costs manageable. And always evaluate against a held-out test set — the model should generalize, not memorize.