llm fine_tuning ai_generated partial

Fine-tuned model loses base capabilities after training on domain-specific data

ID: llm/fine-tune-catastrophic-forgetting

Also available as: JSON · Markdown
62%Fix Rate
78%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —
any active — — —

Root Cause

Fine-tuning on narrow domain data causes catastrophic forgetting of base model capabilities. The model becomes an expert in your domain but loses general reasoning, instruction following, or other skills.

generic

Workarounds

  1. 88% success Mix domain data with general instruction-following examples (80/20 ratio) 
    training_data = domain_examples + random.sample(general_examples, len(domain_examples) // 4)
  2. 92% success Use LoRA/QLoRA instead of full fine-tuning to preserve base capabilities 
    Fine-tune only adapter weights (r=16, alpha=32). Base model weights remain frozen. Merge on inference.
  3. 85% success Use RAG + few-shot prompting before resorting to fine-tuning 
    Often retrieval-augmented generation with examples achieves similar quality without forgetting risks

Dead Ends

Common approaches that don't work:

  1. Fine-tune on domain data only without including general examples 85% fail

    Training exclusively on domain data overwrites general capabilities. The model forgets how to follow instructions, do math, or reason about non-domain topics.

  2. Fine-tune for many epochs to maximize domain performance 82% fail

    More epochs = more forgetting. 3-5 epochs is usually optimal. Beyond that, validation loss increases while training loss decreases (overfitting).