---
title: "Recommended Open-Source Small LLMs, Fine-tuning Tools, and a Structured Learning Path for Aspiring LLM Developers"
created: 2026-04-15
updated: 2026-04-15
type: summary
tags: [finetuning, learning, llms, notes, telegram]
external: http://go/ui/posts/2408/
status: inbox
---

**user**: What are the popular open source small LLMs can be used for me to learn llm and fine tuning?  one of the examples is https://github.com/arman-bd/guppylm?tab=readme-ov-file

**claude**: Great question! Here are the popular small open-source LLMs well-suited for learning and fine-tuning:

---

## 🔬 From-Scratch / Educational Models (like GuppyLM)

These are great for *understanding internals* — architecture, tokenization, training loops:

- **[nanoGPT](https://github.com/karpathy/nanoGPT)** — Andrej Karpathy's legendary minimal GPT-2 reimplementation. The gold standard starting point for learning. Pairs well with his YouTube series.
- **[llm.c](https://github.com/karpathy/llm.c)** — Same author, GPT-2 in pure C/CUDA. Excellent for understanding low-level training.
- **[minGPT](https://github.com/karpathy/minGPT)** — Earlier, cleaner educational version of nanoGPT.
- **[micrograd](https://github.com/karpathy/micrograd)** — Even more fundamental: a tiny autograd engine. Good first step before LLMs.
- **[picoGPT](https://github.com/jaymody/picoGPT)** — GPT-2 in ~100 lines of NumPy. Fantastic for understanding attention mechanics.

---

## 🧠 Pre-trained Small Models (Good for Fine-tuning Practice)

These are real models small enough to run on consumer hardware:

- **Llama 3.2 1B / 3B** (Meta) — Best quality-to-size ratio right now. Great for QLoRA fine-tuning on a single GPU.
- **Qwen2.5 0.5B / 1.5B** (Alibaba) — Surprisingly capable at tiny sizes. Good for experimentation.
- **Phi-3.5 Mini** (Microsoft) — 3.8B, punches above its weight. Strong reasoning for its size.
- **Gemma 2 2B** (Google) — Clean, well-documented, good for fine-tuning tutorials.
- **SmolLM2 135M / 360M** (HuggingFace) — Extremely small, trains fast, ideal for laptop experiments.
- **GPT-2 (124M)** — Classic. Tons of tutorials, easy to fine-tune with HuggingFace Transformers.

---

## 🛠️ Recommended Fine-tuning Stack

Since you're already familiar with Python and local LLM tooling (Ollama, MLX-LM):

- **HuggingFace Transformers + PEFT** — The standard. Covers LoRA, QLoRA, full fine-tuning.
- **Unsloth** — 2x faster fine-tuning, less VRAM. Works great on consumer GPUs and even Apple Silicon.
- **MLX-LM** — Since you're on Mac, this is native and very fast for fine-tuning small models.
- **LitGPT** — Lightning AI's clean implementation, great for learning training loops with real models.
- **Axolotl** — Config-driven fine-tuning, good once you understand the basics.

---

## Suggested Learning Path

1. **Start**: Karpathy's `micrograd` → `nanoGPT` + his YouTube series ("Let's build GPT")
2. **Hands-on fine-tuning**: GPT-2 or SmolLM2 with HuggingFace + PEFT (LoRA)
3. **Real model fine-tuning**: Llama 3.2 1B with Unsloth or MLX-LM on your Mac
4. **Go deeper**: Implement attention, tokenizer, or training loop from scratch (GuppyLM style)

Karpathy's content is genuinely the best entry point — highly recommend starting there if you haven't already.