Fewer than one in five US firms with four or fewer employees use AI at all (US Census Bureau, 2026) - yet the advice aimed at them keeps jumping straight to "train your own model." If you have been told your business needs a custom fine-tuned AI to compete, this guide is the reality check.
Short answer: For almost every small and mid-market business, the answer is prompting first, retrieval (RAG) second, and fine-tuning rarely or never. Among enterprises actually running AI in production, prompt design is the dominant customization technique, RAG is second, and fine-tuning is niche (Menlo Ventures, 2025). Start with the cheapest, fastest lever and only add complexity when you can prove the simpler one has hit its ceiling.
The three approaches in plain English
These three terms get thrown around as if they were competing products. They are not. They are three different ways to make a general-purpose AI model behave the way you want, and they sit on a ladder from cheapest to most expensive.
Prompting (prompt engineering) means giving the model clear, well-structured instructions and examples in the request itself. No training, no infrastructure - you are just writing better inputs. Changes take minutes.
RAG (retrieval-augmented generation) means connecting the model to your own documents - your price list, your policies, your knowledge base - so it can look up the right facts before it answers. You are not changing the model; you are handing it an open book at question time. Changes mean updating the documents, not retraining anything.
Fine-tuning means actually retraining the model's weights on hundreds or thousands of your own examples so it internalizes a specific style, format, or behavior. It is the only one of the three that changes the model itself, and it is the most expensive to build, evaluate, and maintain.
The critical point: these layer together, and the order matters. You do prompting first. You add RAG when the model needs facts it does not have. You reach for fine-tuning only when prompting and RAG still cannot get consistent behavior.
The data says fine-tuning is the exception, not the default
Here is the finding that should end the "you need a custom model" sales pitch. In Menlo Ventures' 2025 State of Generative AI in the Enterprise report - a survey of 495 US enterprise AI decision-makers conducted November 7-25, 2025 - the pecking order of customization techniques in production was clear: "Prompt design remains the dominant technique, followed by retrieval-augmented generation (RAG). More advanced approaches - fine-tuning, tool calling, context engineering, and reinforcement learning (RL) - are still niche" (Menlo Ventures, 2025).
Read that again. These are large, well-funded enterprises with dedicated ML teams and $37 billion in collective generative-AI spending in 2025 (Menlo Ventures, 2025). Even they mostly lead with prompting and retrieval. Fine-tuning is a specialist tool used at the frontier, not the default starting point.
The same report found that build-it-yourself is losing to buy-it: 76% of AI use cases are now purchased rather than built internally (Menlo Ventures, 2025). That is an enterprise statistic, not a small-business one - but the direction of travel is telling. If companies with real engineering budgets are increasingly buying rather than building, a five- or fifty-person business almost certainly should not be commissioning a custom-trained model as step one.
Start with prompting - and most of you can stop there
Prompting is underrated precisely because it is unglamorous. There is nothing to sell you, no monthly training bill, no ML engineer to hire. But it is fast, nearly free, and often good enough.
Hamel Husain, a widely-followed applied-AI engineer and educator, puts the sequencing plainly: "You should do as much prompt engineering as possible before you fine-tune" (Husain, hamel.dev). His broader point is that if prompt engineering works - and you are systematically checking your results - it is perfectly fine to stop there. Complexity is not a badge of honor.
Prompting has one more advantage that is easy to miss: it is cheap to run, and modern tooling makes it cheaper still. With prompt caching, a reused chunk of instructions or reference text is billed at a fraction of the normal rate. On Anthropic's current pricing, cache-read tokens cost 0.1 times the base input price, while writing to the 5-minute cache costs 1.25 times and the 1-hour cache costs 2 times the base rate (Anthropic, 2026). In plain terms: you pay a small premium the first time, then a 90% discount every time after - so a well-designed, stable prompt reused across many requests can be dramatically cheaper than it looks. (Model pricing changes often; confirm current rates before you budget.)
For most small-business use cases - answering FAQs, drafting emails, summarizing calls, categorizing tickets - a good prompt plus a capable off-the-shelf model is the entire solution. If you want a sense of what those off-the-shelf tools cost and do, our guide to the best AI tools for small business in 2026 is a practical starting point.
When RAG earns its place
RAG is the right next step when the model needs to know things that are specific to your business and change over time - facts it was never trained on and that a prompt cannot reasonably contain.
Classic triggers: a customer-service assistant that has to quote your current return policy, a booking bot that needs today's availability, an internal helper that answers from your operations manual. In each case the problem is not the model's writing ability; it is that the model does not have your information. RAG solves that by retrieving the relevant document and feeding it to the model at answer time. When your policy changes, you update the document - you do not retrain anything.
RAG is also the honest answer to hallucinations for fact-based tasks. A model asked about your pricing with no access to your pricing will guess. The same model handed your actual price sheet will quote it. This is why retrieval sits second on the ladder for a reason: it fixes a whole category of problems that neither prompting nor fine-tuning addresses well. If you are building a customer-facing assistant, our breakdown of AI chatbot costs for small businesses covers what this looks like in practice.
When fine-tuning is actually worth it
Fine-tuning is not useless - it is just narrow. It shines when you need the model to reliably produce a very specific style, tone, or output format, at scale, in a way that prompting cannot make consistent. Think of a high-volume task where the model must always respond in exactly the same structured way, or match a distinctive house voice across thousands of outputs a day.
But there is a hard prerequisite most people skip. As Husain notes, "It's impossible to fine-tune effectively without an eval system" (Husain, hamel.dev) - a way to systematically measure whether the model's answers are actually getting better. Without that, you are spending money to change the model blind. This is a large part of why fine-tuning stays niche even inside sophisticated companies: it demands evaluation infrastructure, labeled examples, and ongoing maintenance every time the base model updates.
A simple decision path
You do not need a committee to choose. Walk down this list and stop at the first "yes."
- Can a better prompt fix it? Clearer instructions, a couple of examples, a defined output format. Most tasks end here. This is hours of work, not months.
- Does it need your specific, changing facts? Prices, policies, inventory, internal docs. Add RAG. You are giving the model an open book, not a new brain.
- Does it still produce inconsistent style or format at high volume, and do you have a way to measure "better"? Only now consider fine-tuning - and budget for the evaluation system it requires.
| Approach | What it changes | Best for | Effort |
|---|---|---|---|
| Prompting | Your instructions | Most everyday tasks | Hours |
| RAG | The facts available | Answering from your own documents | Days to weeks |
| Fine-tuning | The model itself | Consistent style/format at scale | Weeks to months |
What This Means for Your Business
The fine-tuning-first pitch survives because it sounds impressive and it justifies a big invoice. The evidence points the other way. The companies with the most resources overwhelmingly lead with prompting and retrieval, and even they treat fine-tuning as a specialist's tool (Menlo Ventures, 2025). Meanwhile most US small businesses are still at the very start of AI adoption - under 20% of the smallest firms use it at all (US Census Bureau, 2026) - which means the highest-value move is almost never a custom model. It is a well-prompted, off-the-shelf tool wired into your real information.
Practically: pick one painful, repetitive task. Try to solve it with a strong prompt and a capable general model. If it needs your specific facts, add retrieval. Measure whether it actually works. Only escalate to fine-tuning if a real, measured ceiling forces you to - and even then, weigh buying a purpose-built product over building one. If you are also weighing automation against headcount, our guide on AI vs hiring an employee is a useful companion, and if agents are on your radar, start with what AI agents actually are for small businesses.
Not sure which rung of the ladder your problem sits on? That is exactly the kind of question worth an outside read before you spend. Talk to Semark's AI and machine learning team or get in touch - we will tell you honestly whether you need a custom model or just a better prompt.
