The first ticket on a lot of AI programs is a training run. Someone wants a model that knows our data. The Work card on 22atlas.com even names Custom model fine-tuning. That card is a capability we staff. It is a late ticket, not a kickoff.
A hosted API plus a prompt is the cheap experiment. You change the instructions, the few-shot examples, the tool list. You find out whether the verb belongs in the product at all. If the model already knows the domain and just needs a clearer job, you stay here. Most teams leave this layer half-finished and jump.
Three jobs, in order
Prompting changes the instructions. Retrieval changes what the model can see. Fine-tuning changes how it behaves. Mix them later if you want. Earn each one.
1. API and prompt
Call a strong general model through an interface you own. Write the verb. Put a handful of ugly examples in the prompt or beside it. Measure. If Tuesday is worse than Monday, you will know. If you cannot name the verb, you are shopping for weights to hide that.
2. Retrieval when knowledge is missing
When the model is guessing because it cannot see the record, fetch the record. Policy that changed on Thursday. Tenant documents. Catalog rows. Those belong in a store you can update without a training job. RAG, or a plain tool that loads the row, is how you give a model facts it was never trained on. Fine-tuning is a bad database.
3. Fine-tune after the behavior plateaus
Weights earn their keep when the target has stopped moving and the first two moves have a measured ceiling. Output schema that still drifts after you tightened the prompt. A refusal style operators need identical every time. A smaller model that must hold a format you already proved on a larger one. You need a frozen set and a plateau first. If you cannot point at the fixtures that still fail, you are funding a science project.
Hybrids are normal
Tune the form. Retrieve the facts. Plenty of production stacks in 2026 do both, after each piece paid rent. A LoRA that holds JSON plus a retriever that holds this week's policy is a system. A fine-tune stuffed with PDFs that change every sprint is a tax.
Why first-move fine-tunes fail
- The knowledge is still changing, so last week's weights are already wrong.
- The target tone is still being argued in Slack, so there is nothing stable to learn.
- Nobody kept traces, so the training set is a playground transcript.
- The team wanted the feeling of owning weights. Ownership is the interface, the evals, and who can roll the change back.
If the facts keep moving, change the context, not the weights.
How we gate the Work card
We will take Custom model fine-tuning under AI Development when prompt and retrieval have a ceiling you can show. Bring the fixtures and the date. Services is how you buy that sequence. A training run to decorate a roadmap waits.
Start your build if you want a pod that will hold the third move until the first two are honest.
Get our Articles and posts by email.
Up to one email a day with new Articles and posts from 22atlas.com and sean-weldon.com, plus occasional news about 22Atlas services.
Also on the desk
Technique
Golden cases from last week's failures
Build the first eval set from real traces. Cluster the errors before you add another judge.
Curated drop
Friday drop: production AI without the theater
Five practices from the week: the verb, golden cases, an override, traces, and consent before the model call.