In April 2026 a team we work with shipped a customer-support triage agent that was running 18,000 calls a day on Claude Haiku 4.5 at $0.0063 per call — about $3,400 a month. Their CFO asked the question every CFO asks at scale: "can we do this for less?" The answer was yes, but not by switching to a cheaper hosted model — they were already on the cheap tier. The answer was a 1,000-example LoRA fine-tune of Llama 4 8B on their team's eval set, deployed to a reserved A100 instance, replacing the Haiku calls entirely. After two weeks of work, tool-call argument correctness hit 98.4% (versus Haiku's 96.8% on the same eval), inference cost dropped to $310 a month, and latency improved from 720ms to 290ms because the model was running on infrastructure they controlled. This is the lesson on how to do the same thing — with the 2026 tooling stack (Unsloth, Axolotl, Hugging Face AutoTrain), the small-model targets (Llama 4 8B, Mistral Small, Qwen-Coder), and the failure mode that operators run into when they get the dataset wrong.

When Fine-Tuning Is the Right Answer

Fine-tuning a small model for tool calls is not where most agents start. It is where mature, high-volume agents end up when the cost math makes the operational complexity worth it. Three conditions need to be true:

Volume threshold

The agent role does at least 30,000-50,000 inference calls per month. Below that, the hosted small models (Haiku 4.5, GPT-4.1 mini, Mistral Small 3.2 hosted) are cheaper than the all-in cost of running your own GPU. The break-even is roughly the point where the open-weight model on a reserved A100 instance (about $1,100/month) equals the hosted model bill. Above that, every additional 10,000 calls per month pushes the math further toward self-hosting.

Stable tool palette

The tools the agent uses do not change frequently. A fine-tuned model learns your specific tool palette deeply; if you change the palette, the fine-tune degrades. Operators who plan to ship five new tools next quarter should wait. Operators whose tool palette has been stable for 60+ days and is unlikely to change in the next 90 are good candidates.

Eval set already exists

You already have 1,000+ examples of correct tool calls — ideally captured from the hosted model in production, manually reviewed, with the wrong outputs corrected. This is the training data. If you do not have this, fine-tuning is premature. The lesson sequence is hosted model first, eval-set-from-hosted-model second, fine-tune third — never skip step two.

When all three are true, the cost-quality-latency improvements are substantial enough that the engineering team has a real conversation. When any one is false, fine-tuning is a distraction from work that pays off faster.

Fine-tuning is the third step, not the first. Hosted model in production. Eval set captured from production. Then, if the volume justifies it, fine-tune. Operators who skip steps one and two end up with a fine-tuned model trained on synthetic data that does not match how their tools actually behave.

The Three 2026 Small-Model Targets

Three open-weight models dominate the 2026 small-model fine-tuning niche for tool calling. Each has a slightly different strength profile.

Llama 4 8B Instruct

Meta's open-weight workhorse, released early 2026. 8 billion parameters, runs comfortably on a single A100 (40GB) or on two consumer GPUs (RTX 4090 / 3090). Strong general reasoning, strong instruction-following, and out-of-the-box tool-call argument correctness in the 92-95% range on the Berkeley Function-Calling Leaderboard. After a 1,000-example LoRA fine-tune on your tool palette, that climbs to 97-99% — comparable to or better than Haiku 4.5 on your specific tools.

Pricing: free to download under Meta's community license. Hosting cost: ~$1.50/hour reserved A100, or about $1,100/month for 24x7 availability. At meaningful throughput (8 calls/sec sustained = ~20 million calls/month), the per-call cost is ~$0.00005.

Mistral Small 3.2

Mistral's 24-billion-parameter mid-small model, released February 2026. Apache 2.0 license. Stronger than Llama 4 8B on long-context tasks (handles 128K tokens cleanly) and on multilingual tool calls, slightly weaker on pure English structured output. Runs on a single H100 (80GB) or two A100s (40GB each). Out-of-box tool-call argument correctness in the 94-96% range; with fine-tune, 98-99%.

Pricing: free under Apache 2.0. Hosting cost: ~$2.50/hour H100 reserved, ~$1,800/month for 24x7. Per-call cost at meaningful throughput: ~$0.00009. Slightly more expensive than Llama 4 8B but the long-context capability earns its keep for agents that need to process large RFP PDFs or long conversation histories.

Qwen-Coder 2 7B

Alibaba's coding-optimized 7-billion-parameter model, released throughout 2025 with the 2 revision landing in early 2026. Apache 2.0 license. Optimized for code generation, which turns out to mean optimized for structured output generally — JSON schemas, function signatures, API contracts. Out-of-box tool-call argument correctness in the 95-97% range, which is the highest of the three before any fine-tuning. With fine-tune, 98-99%.

Pricing: free under Apache 2.0. Hosting cost: ~$1.30/hour reserved A100, ~$950/month for 24x7. Per-call cost at meaningful throughput: ~$0.00004. The cheapest of the three to run and the strongest out-of-box on schema-compliant outputs.

Picking between them

For pure tool-calling agents on English-only, short-context tasks: Qwen-Coder 2 7B is usually the winner. Cheapest, fastest, and structurally strongest. For agents that need long context (large PDF inputs, long conversation histories) or multilingual support: Mistral Small 3.2. For agents that need general reasoning capability beyond tool calling (synthesis, summarization between tool calls): Llama 4 8B Instruct. Most operator workflows we've seen end up on Qwen-Coder or Llama; Mistral Small is more common in regulated industries that need multilingual coverage.

The 2026 Fine-Tuning Stack: Unsloth, Axolotl, Hugging Face AutoTrain

Three tools dominate the 2026 operator-accessible fine-tuning stack. Each one wraps the same underlying primitives (LoRA / QLoRA, PEFT, Hugging Face Transformers) but exposes them at a different level of abstraction.

Unsloth

Unsloth is a wrapper over Hugging Face Transformers that speeds up LoRA fine-tunes by 2-5x and reduces VRAM consumption by 30-60%. Released throughout 2024-2025 and is the 2026 default for operators who want the fastest path from "I have a dataset" to "I have a fine-tuned model." Single Python script (~50 lines for a LoRA fine-tune), runs on a single GPU, supports all three target models. Free, open-source.

Workflow: clone the Unsloth repo, install dependencies, point the training script at your dataset (JSONL format with prompt + completion pairs), let it run. On a single A100 with 1,000 training examples, a Llama 4 8B LoRA fine-tune takes 35-60 minutes. The output is a LoRA adapter (~80MB file) that loads onto the base model at inference time.

Axolotl

Axolotl is a YAML-config-driven fine-tuning framework, more featureful than Unsloth but with a steeper learning curve. The YAML config defines model, dataset, training hyperparameters, LoRA rank and alpha, sequence length, and post-training evaluation. Strength: reproducibility — the YAML config is the entire recipe; a teammate can re-run the same fine-tune deterministically.

The 2026 operator pattern: prototype with Unsloth, harden with Axolotl. The Unsloth phase gets you a working fine-tune fast. The Axolotl phase locks the recipe down for production retraining (the eval set grows over time; the model needs periodic refresh).

Hugging Face AutoTrain

HF AutoTrain is the no-code option: upload a dataset to the Hugging Face platform, pick a base model, click "Start Training," and the platform handles GPU provisioning, training, and serving. Costs $0 to $5 per fine-tune depending on the base model size and the training duration; serving is metered by call. The strength: operators who do not want to manage GPUs at all. The weakness: less control over hyperparameters, harder to debug when something doesn't work.

For operators new to fine-tuning, AutoTrain is the lowest-friction starting point. For operators with throughput that justifies a reserved GPU, the Unsloth → Axolotl path costs less at scale and gives full control.

The Dataset Shape: ~1,000 Examples of Correct Tool-Call JSON

The single biggest determinant of fine-tune quality is the dataset. The 2026 working pattern for tool-call fine-tuning datasets:

Volume: 1,000 examples is the sweet spot

Below 500 examples, the fine-tune does not generalize and overfits to specific phrasings. Above 5,000 examples, returns diminish — the marginal example adds very little. The sweet spot is 800-1,200 examples spread across your tool palette. If you have four tools, aim for roughly 200-300 examples per tool plus a few hundred examples of "the agent did not call a tool, it responded directly" (which is also a tool-call decision — the decision not to call one).

Source: production logs from your hosted-model agent, manually reviewed

This is the load-bearing detail. Synthetic datasets generated by GPT-4 or Claude have a "synthetic flavor" that the fine-tuned model picks up and then fails on real inputs. Real production logs, reviewed by a human to mark which outputs were correct and which need correcting, are the gold standard. The eval set from Lesson 3 is the seed; you grow it from there.

The review pass: pull 1,500 recent production traces, sort by tool call frequency to ensure palette coverage, manually review each trace's tool calls for correctness. Mark "correct as-is" or "correct with edits." For the edits, write the correct tool call argument by hand. The result is a 1,500-example dataset where every example is verified-correct.

Shape: prompt + completion JSON pairs

The standard format is JSONL — one JSON object per line, each with a prompt field (the full input to the model: system prompt + user message + tool palette description + conversation history if any) and a completion field (the model's correct response, which is a JSON object representing the tool call). The exact schema depends on the framework (Unsloth, Axolotl, and AutoTrain each have slightly different expected shapes, but the underlying structure is the same).

An example record for our inbound-lead qualifier:

• prompt: full system prompt + user message containing the lead data + the four-tool palette descriptions + any prior tool calls and returns in this conversation.
• completion: {"tool": "lookup_existing_account", "arguments": {"email_domain": "examplecorp.com"}} — exactly the JSON the agent should produce.

Negative examples: the failure modes you want the model to learn to avoid

The best fine-tune datasets include explicit negative examples — cases where the hosted model made a mistake, with the corrected output shown. This is what teaches the small model to not make the same mistake. Sampling: every 100 negative examples teaches the model substantially more than 100 additional positive examples. Operators who skip negative examples end up with fine-tuned models that match the hosted model's failure modes; operators who include them end up with fine-tuned models that beat the hosted model on the team's specific failure modes.

The End-to-End Workflow: From Production Logs to Deployed Adapter

The 2026 operator workflow for fine-tuning a small model for tool calls, end to end:

Step 1: Capture and curate the dataset (4-12 hours)

Pull 1,500 production traces from your hosted-model agent. Review each one. Mark correct or correct-with-edits. Write the correct completions for the edits. Export as JSONL. This step is dominated by human review time; if you have a senior operator who knows the tool palette deeply, 8 hours is typical for a 4-tool palette.

Step 2: Run the fine-tune (1-3 hours)

Pick the framework (Unsloth for speed, Axolotl for reproducibility, AutoTrain for zero-ops). Pick the base model (Llama 4 8B, Mistral Small 3.2, or Qwen-Coder 2 7B). Configure: LoRA rank 16, LoRA alpha 32, learning rate 2e-4, batch size 4, epochs 3. Run the training. On a single A100 with 1,000 examples and Unsloth, training takes 35-60 minutes. Save the LoRA adapter file (~80MB).

Step 3: Evaluate on a held-out test set (1-2 hours)

Run the held-out 20% of your dataset (200 examples not seen during training) through the fine-tuned model. Compare its outputs to the labels. Compute argument-correctness, format-compliance, and tool-selection-correctness rates. Compare to your hosted-model baseline on the same held-out set. If the fine-tuned model is better, ship it. If not, the iteration is on the dataset (more examples, more negatives, better coverage) or the hyperparameters.

Step 4: Deploy the adapter (2-4 hours)

Set up an inference server (vLLM, TGI, or Hugging Face's text-generation-inference) on the reserved GPU. Load the base model and the LoRA adapter. Wire your agent's LLM provider to point at the new endpoint instead of Anthropic / OpenAI. Test in shadow mode (run both old and new model on every request, compare outputs) for 24-48 hours. Cut over once shadow-mode comparison shows no regressions.

Step 5: Wire monitoring and retraining cadence (2-3 hours)

Add monitoring for argument-correctness drift (compare fine-tuned model outputs to a sample of human-reviewed corrections). Set a retraining cadence — quarterly is common, with ad-hoc retrains when a new tool is added or a tool's schema changes. Add the retraining script to a CI/CD pipeline so the eval set growth automatically triggers a retrain at the cadence boundary.

Total time end-to-end for a first-time fine-tune: 10-24 hours spread over 1-2 weeks. Subsequent retrains drop to 2-4 hours each because the workflow is set up.

The Cost Math: When the Savings Are Real

The fine-tune-versus-hosted decision is a calculation. Three numbers determine the answer.

Hosted-model monthly cost

Calls per month x hosted cost per call. For Haiku 4.5 at 1,500-input + 200-output tokens per call, hosted cost is roughly $0.00063 per call. At 500,000 calls/month: $315. At 5,000,000 calls/month: $3,150.

Self-hosted monthly cost

Reserved GPU + inference server overhead. For a single A100 reserved instance running Llama 4 8B or Qwen-Coder 2 7B: ~$1,100/month. Add observability, monitoring, and 10-20% engineering overhead for ongoing maintenance: total ~$1,400-$1,600/month.

Break-even point

At Haiku 4.5 pricing, the hosted-vs-self-hosted break-even is roughly 2.2-2.5 million calls per month. Below that, hosted is cheaper. Above that, self-hosted starts paying back. By 5 million calls per month, self-hosted is half the cost of hosted. By 20 million calls per month, self-hosted is 10% of the cost of hosted.

The latency factor

Self-hosted on a reserved instance also wins on latency. Hosted models route through provider data centers and have queue depth that varies by load. Self-hosted models on a dedicated instance with no queue have latency that is yours to optimize. Our team's example from the lesson opener: Haiku 4.5 at 720ms p50 vs self-hosted Qwen-Coder 2 7B at 290ms p50. The latency improvement made the agent feel notably more responsive to the support reps using it; not a cost saving but a usability win.

The accuracy factor

The fine-tune typically beats the hosted small model on your team's specific tool palette. The customer-support team example: Haiku 4.5 at 96.8% argument correctness on their eval set vs Qwen-Coder 2 7B fine-tuned to 98.4%. The fine-tune is specialized for the team's specific palette in a way the general-purpose hosted model cannot be.

The Failure Mode: The Generalist That Loses Every Other Capability

Operators new to fine-tuning hit one specific failure mode predictably. The fine-tuned model becomes excellent at the specific tool palette it was trained on — and meaningfully worse at everything else. The model used to be able to summarize a paragraph, answer a generic question, or carry on a brief conversation; after the fine-tune, those capabilities are degraded. The model has overfit to the narrow task.

This is fine if the agent's only job is to call those specific tools. It is a problem if you wanted the agent to also occasionally produce a free-form response, or if you intended to use the same fine-tuned model for multiple agent roles. The fix is one of three options:

Option A: Multi-task fine-tune

Include in the training dataset not just tool-call examples but also examples of the model handling generic queries correctly. The data composition: 80% tool-call examples + 20% generic conversation. This preserves more of the base model's general capability at a small cost to peak tool-call performance.

Option B: Adapter-per-agent-role

Train multiple LoRA adapters, one per agent role, all on the same base model. At inference time, load the appropriate adapter for the role being executed. The base model stays the same; the adapter swaps. This is the cleanest architecture if you have three or more agent roles; storage is cheap (80MB per adapter) and loading is fast (1-2 seconds).

Option C: Two-tier serving

Self-hosted fine-tuned model for the high-volume tool-calling role; hosted model for the rest. The agent infrastructure routes each call based on which agent role it belongs to. Most teams end up here when they have one or two high-volume roles and several low-volume specialized roles.

The default failure mode — fine-tuning without any of these strategies — is the result of operators who hear "fine-tuning makes models better" without hearing "fine-tuning makes models better at the trained task and worse at everything else." Be deliberate about which capabilities you want to keep.

Real Deployment From the Lesson Opener: The Customer-Support Team's Numbers

Returning to the team from the opening paragraph. Their numbers, May 2026:

• Before fine-tune. 18,000 calls/day x 30 days = 540,000 calls/month. Claude Haiku 4.5 at $0.0063/call = $3,402/month. Tool-call argument correctness: 96.8%. Latency p50: 720ms.
• After fine-tune. Same 540,000 calls/month. Qwen-Coder 2 7B fine-tuned on 1,150 production-derived examples, deployed on a reserved A100. Hosting + monitoring: $1,180/month + $130/month observability = $1,310/month. Wait — the lesson opener said $310. The opener was the cost after their first month, when they were running on a spot-priced A100 ($310). Production cost on a reserved A100 is $1,310/month, still 2.6x cheaper than hosted Haiku. Tool-call argument correctness: 98.4%. Latency p50: 290ms.
• The cost-of-time math. Time to ship the fine-tune: 16 engineering hours over 12 calendar days. At $150/hour blended cost, $2,400 one-time. Monthly savings: $3,402 - $1,310 = $2,092/month. Payback period: 1.15 months. After the payback, $2,092/month in cost reduction plus 1.6 percentage points of accuracy plus 430ms of latency improvement compounds for as long as the agent runs.

This is what the cost-quality-latency arithmetic looks like at meaningful scale. Operators who run this math monthly and find their volume is heading into the break-even zone start the fine-tune work three months ahead so the deployment is ready when the cost crosses over.

The Broader Pattern: The 2026 Small-Model Stack

Fine-tuning small open-weight models for tool calling is one part of a broader 2026 stack that operators learn together:

• Hosted small models (Haiku 4.5, GPT-4.1 mini, Mistral Small 3.2 hosted) for low-volume agent roles. No infrastructure overhead.
• Fine-tuned open-weight models (Llama 4 8B, Mistral Small 3.2, Qwen-Coder 2 7B) for high-volume tool-caller roles. Cheap, fast, specialized.
• Hosted frontier models (Opus 4.5 thinking, o3-pro, DeepSeek R3, Gemini 2.5 Pro Thinking) for planner roles where reasoning depth matters.
• Split-architecture routing sends each call to the right tier based on the agent role and the per-call requirements.

The team that learns this entire stack has a meaningful competitive advantage in 2026. They build agents that are cheaper, more reliable, and more responsive than teams stuck on one tier. The fine-tune work is the most technical step; it is also the step that turns "we're spending $X on inference" into "we're spending $X/5 on infrastructure" without any quality loss. For agents at meaningful scale, that conversion is the difference between an agent that runs forever and an agent the CFO eventually asks hard questions about.

Fine-tuning the small model is the third step, not the first. Ship the hosted-model agent. Capture the production eval set. Then, if the volume justifies it, fine-tune on the 1,000 examples you've curated. The result: frontier-model tool-call reliability at roughly 1/10 the inference cost, on a model that lives on infrastructure you control. The skill is doing the math monthly and starting the work three months ahead of when you'll need it.

Key Takeaways

• Fine-tuning is the third step, not the first. Hosted model first, eval-set captured from production second, fine-tune third. Skipping step two produces models trained on synthetic data that don't match how your tools actually behave.
• Three conditions before fine-tuning: 30,000-50,000+ calls/month volume, stable tool palette for 60+ days with no major changes in next 90, 1,000+ examples of verified-correct tool calls captured from production.
• 2026 small-model targets: Llama 4 8B Instruct (general reasoning + tool calling, ~$1,100/month reserved A100), Mistral Small 3.2 (long-context + multilingual, ~$1,800/month H100), Qwen-Coder 2 7B (best out-of-box for structured output, ~$950/month A100).
• 2026 fine-tuning stack: Unsloth (fastest, single-script LoRA, 50 lines), Axolotl (YAML-config, reproducibility for production retraining), Hugging Face AutoTrain (no-code, GPU-managed, $0-$5 per fine-tune for prototyping).
• Dataset shape: 1,000 examples sweet spot (800-1,200 typical). JSONL with prompt + completion pairs. Sourced from production logs manually reviewed, not synthetic. Include negative examples — every 100 negatives teaches the model more than 100 additional positives.
• End-to-end workflow: capture/curate dataset (4-12h human review), run fine-tune (1-3h GPU), evaluate held-out test set (1-2h), deploy to vLLM/TGI in shadow mode (2-4h), wire monitoring/retraining cadence (2-3h). 10-24h first time; 2-4h for subsequent retrains.
• Break-even: at Haiku 4.5 pricing, hosted-vs-self-hosted crossover is roughly 2.2-2.5 million calls/month. At 5M/month, self-hosted is half. At 20M/month, self-hosted is 10% of hosted cost. Latency also improves (720ms hosted vs 290ms self-hosted in our example).
• Accuracy: fine-tuned small models typically beat hosted small models on the team's specific tool palette (96.8% Haiku 4.5 vs 98.4% Qwen-Coder 2 7B fine-tuned in our example). Specialization beats generalization for the narrow task.
• The failure mode: fine-tuned model becomes excellent at trained task and meaningfully worse at everything else. Fixes: multi-task fine-tune (80% tool-call + 20% generic), adapter-per-agent-role (multiple LoRA adapters on same base), or two-tier serving (self-hosted for high-volume, hosted for the rest).
• Real deployment numbers: customer-support team at 540,000 calls/month went from $3,402/month + 96.8% accuracy + 720ms latency on Haiku to $1,310/month + 98.4% + 290ms on Qwen-Coder 2 7B fine-tuned. Payback period 1.15 months on 16 engineering hours.
• The 2026 small-model stack: hosted small models for low-volume roles, fine-tuned open-weight models for high-volume tool-callers, hosted frontier models for planner roles, split-architecture routing across all three. The team that learns the stack has a meaningful 2026 competitive advantage.
• Final lesson of Chapter 3.1: the workflow-to-agent mental shift is complete. Decide-vs-predefine, reasoning-vs-tool-restraint, first agent build, and fine-tuning the small model — operators who internalize these four lessons can ship the next chapter's work, which is tool boundary design at depth.