How to Actually Choose a Model
Benchmarks, price, latency, context: a working decision framework instead of hype.
Choosing a model is a multi-objective decision. You weigh accuracy, latency, cost, context length, and safety against the hard constraints of your specific task. No single benchmark or leaderboard ranking can make this call for you. The process that works in production is: define your task precisely, build a representative evaluation dataset, run structured offline experiments to prune candidates, then validate the survivors with real users.
Here is the surprising part: the largest, most expensive model is rarely the right choice. Small fine-tuned models often beat generalist giants on specific tasks while being two orders of magnitude cheaper. Most teams waste money chasing marginal accuracy gains that users never notice.
What Does "Choosing a Model" Actually Mean?
Choosing a model is not one decision. It is a chain of decisions that starts with your task and ends with a running system.
The first decision is architectural. Do you need retrieval augmented generation (RAG) to ground answers in documents? Do you need fine-tuning to enforce a specific format or style? Your task's requirements might force a particular architecture before you even look at specific models.
Remember our running example of asking a chatbot for a recipe. If that chatbot needs to pull ingredients from a frequently updated company database, RAG becomes nearly mandatory. You need fresh data and source citations. This architectural choice shapes every model decision that follows. You now need an embedding model, a retriever, and a generation model that plays nicely with retrieved context.
In classical machine learning, model selection meant picking between a linear model and a tree ensemble. In the LLM world, you might choose between a tiny fine-tuned encoder and a massive generalist model. You might decide to route simple queries to a fast cheap model and complex queries to a slow expensive one. The decision space is larger now, but the core principle is unchanged: narrow the space using constraints before you compute a single metric.
How Do You Define Your Task Precisely Enough to Choose?
A vague goal like "make a good chatbot" is useless for model selection. You need a specification concrete enough to measure against.
Write down your task type. Is it classification, retrieval, generation, or reasoning? List your evaluation objectives. Are you optimizing for accuracy, factuality, helpfulness, or some business metric like conversion rate? Then list your hard constraints. Maximum tail latency in milliseconds. Minimum throughput in requests per second. Budget per request. Data residency requirements.
Suppose your recipe chatbot has a strict service level objective: p99 latency under two seconds. Any model that cannot meet that constraint gets eliminated immediately. You do not need to run a single benchmark to cross it off the list. Constraints define the feasible region. Everything outside that region is irrelevant.
This matters because engineers often start by browsing leaderboards. That is backwards. Start with your own requirements. The leaderboard comes later, as a filter on the candidates that already fit your constraints.
How Do You Build Evaluation Data You Can Trust?
You need a dataset that looks like the actual traffic your model will handle. Not a random sample of tweets. Not a general benchmark. Your data.
Teams typically assemble a "golden dataset" of a few hundred to a few thousand real queries paired with ideal responses. These queries come from production logs, from synthetic generation reviewed by experts, or from both. The dataset must span real user intents, input lengths, difficulty levels, and edge cases. Happy paths alone will mislead you.
For tasks with clear right answers, evaluation is straightforward. You run the model on each query and check if the output matches the expected answer. Accuracy or exact match works here.
For open-ended generation, things get harder. There is no single correct recipe response to "what can I cook with chicken and rice." You need qualitative judgments: helpfulness, clarity, safety. You can build a rubric and have humans score outputs. Or you can use a strong model as a judge, giving it the query, the output, and a scoring rubric. This scales better but requires careful prompt design.
The crucial mechanical detail: keep your evaluation data separate from anything you use to tune the model. If you tweak prompts based on test set performance, you have contaminated your test set. You need a held-out set that sees zero exposure until final evaluation. Standard three-way splits (train, validation, test) still apply here. For LLM work where you are not training, reserve some evaluation data for hold-out testing of prompts, routing rules, or RAG parameters.
What Do Benchmarks Like MMLU and TruthfulQA Actually Tell You?
Static benchmarks are diagnostic tools. They are not final arbiters of model quality for your task.
MMLU (Massive Multitask Language Understanding) tests knowledge across dozens of subjects through multiple-choice questions source. A high MMLU score means broad factual knowledge. It does not mean the model writes good recipes or follows formatting instructions. TruthfulQA tests whether models avoid repeating common misconceptions source. A model that scores well here is less likely to tell you to put glue on pizza because it read that somewhere online. ARC tests science reasoning. HellaSwag tests commonsense continuation selection.
These benchmarks are attractive because they are automated. You download the dataset, run your model, and get a number. But they have limits. Models can be over-optimized for public benchmarks through training or prompt engineering. A two-point difference on MMLU between two models often means nothing for your specific task. And none of these benchmarks measure latency, cost, or how well a model handles your company's internal jargon.
Use benchmarks as sanity checks. If a model cannot handle basic commonsense reasoning on HellaSwag, it probably should not power your customer-facing chatbot. But do not use benchmark rankings as your primary selection criterion. Your golden dataset matters more.
How Do You Compare Models When There Are Multiple Objectives?
You cannot reduce model selection to a single number. Accuracy, latency, cost, and context length are all real constraints. The model that wins on accuracy might lose catastrophically on latency.
Pareto frontier analysis gives you a clear framework. Plot your candidates. A model is dominated if another model beats it on every metric. Eliminate all dominated models. What remains is the Pareto frontier: the set of models where improving one metric means sacrificing another. Decision makers then pick from this frontier based on their trade-off preferences.
You might formalize this with a composite metric like the Performance Efficiency Ratio (PER), which combines accuracy, throughput, memory usage, and latency into one normalized score source. Small models in the 0.5 to 3 billion parameter range often dominate on PER, especially in resource-constrained deployments. They give up a few points of accuracy and gain orders of magnitude in speed and cost.
Total cost of ownership per token is another composite worth computing. It combines hardware capital cost, operational cost (power, cloud), and engineering maintenance cost, divided by total lifetime tokens served. The cheapest GPU per hour is not always the cheapest deployment per token. Throughput and utilization matter more.
For our recipe chatbot, you might measure accuracy on your golden dataset of recipe requests, p99 latency at peak load, and cost per thousand requests. A small fine-tuned model might score 87% accuracy at 200ms p99 latency and cost $0.001 per request. A giant generalist model might score 91% at 800ms p99 latency and cost $0.05 per request. Users rarely notice the 4% accuracy gap. They absolutely notice a half-second delay. The frontier makes this trade-off explicit.
How Do You Validate a Model Before Full Deployment?
Offline evaluation on your golden dataset is necessary. It is not sufficient. Users behave differently than evaluation datasets predict.
Shadow deployment is the safest next step. Run the candidate model in parallel with your current production model. Send it the same real inputs. Log its outputs. Compare them against the incumbent. Users see no change. You collect real-world data on how the candidate behaves under live traffic.
When you are confident enough to expose users, run an A/B test. Route a small percentage of traffic to the candidate. Measure real outcomes: user satisfaction ratings, task completion rates, whether people immediately re-ask the question in a different way (a strong signal the first answer was bad). Statistical analysis tells you whether any observed differences are real or noise.
Monitor continuously after full deployment. User behavior drifts. Input distributions shift. The model that worked in January might degrade by June. Log inputs and outputs. Periodically score samples using your evaluation pipeline. Set alerts for metric degradation. This is not a "choose once" problem. It is an ongoing process.
Quick Reference: Key Decision Factors
| Factor | What to Measure | Why It Matters |
|---|---|---|
| Task fit | Accuracy on your golden dataset | General benchmarks do not predict domain performance |
| Latency | p50 and p99 response time | Users feel delays above 200ms. Strict SLOs may force smaller models |
| Cost | Total cost per request or per token | Marginal accuracy gains often cost 10-50x more |
| Context window | Tokens needed for your typical input plus retrieved documents | A model with a 4K window fails on tasks needing 8K tokens |
| Safety and governance | Hallucination rate, refusal rate, data residency | A model that makes things up confidently destroys user trust |
| Maintainability | Engineering effort to deploy, monitor, and update | Open models give control but add operational burden |
Frequently Asked Questions
Q: Should I just use the model at the top of the LMSYS leaderboard?
No. The LMSYS Chatbot Arena measures human preference across arbitrary prompts. It correlates weakly with performance on your specific task, tells you nothing about latency or cost, and reflects a generalist use case that may not match yours. Use it as a discovery tool to find candidates, not as a final decision criterion.
Q: When does a small fine-tuned model beat a large general model?
When your task is narrow and well-defined. Classification, entity extraction, structured output generation, and domain-specific Q&A on a stable corpus all favor small fine-tuned models. A 7B parameter model fine-tuned on your data can match or beat a generalist 70B model on your specific task while being faster and cheaper by orders of magnitude source.
Q: How many evaluation examples do I need?
Enough to cover your real input distribution and edge cases. For most production applications, 300 to 1,000 representative examples provide a reliable signal. Too few examples and your variance is high. Too many and you burn time and money on evaluation without gaining confidence. Start small, measure variance, and add examples where you see high uncertainty.
Q: Can I trust LLM-as-a-judge evaluations?
Partially. A strong judge model with a clear rubric correlates reasonably well with human judgments. But judge models carry biases. They tend to favor longer responses and their own model family's outputs. Validate your judge against human annotations on a subset of your data before relying on it. For high-stakes decisions, keep humans in the loop.
Q: How often should I re-evaluate my model choice?
Continuously monitor. Re-evaluate formally when you see metric degradation, when significantly better models become available, or when your task requirements change. For most teams, a quarterly structured re-evaluation alongside ongoing monitoring strikes the right balance between stability and improvement.
Test Yourself
Your team is building a customer support bot that answers questions from a 10,000-page product manual. The manual changes weekly. Your budget caps inference cost at $200 per month. Latency must stay under one second. A teammate proposes using the latest 405B parameter model because it tops the MMLU leaderboard. Another suggests fine-tuning a 7B model on a snapshot of the manual. Which approach is likely to fail and why?
Answer: The 405B model approach fails on cost and freshness. At typical API pricing, serving even a few thousand queries per day with a 405B model would blow past the $200 monthly budget within days. The fine-tuning approach fails on freshness. A model fine-tuned on a snapshot of the manual becomes stale the moment the manual updates. Weekly fine-tuning runs are engineering-intensive and test your ability to ship model updates reliably. The correct approach here is RAG with a small, fast model. You embed the manual chunks in a vector database, retrieve relevant sections for each query, and feed them as context to a 7B or even 3B model. This keeps answers grounded in the current manual version, stays within the latency budget (small models are fast), and costs far less than serving a giant model. You update the vector database weekly without retraining anything.
- High API cost
- High latency
- Cannot update weekly
- Low cost
- Low latency
- Weekly fine tuning possible
If you want this kind of breakdown every week — not hype, not leaderboard worship, but how real systems actually work under the hood — subscribe to Internals Decoded at internalsdecoded.com.
Sources
- MMLU: Measuring Massive Multitask Language Understanding
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Performance-Efficiency Ratio: Composite Metrics for Model Selection
- Fine-tuned Encoders vs Zero-shot LLMs for Classification
- Langfuse: LLM Evaluation Framework
- RAG Evaluation Metrics: Faithfulness and Retrieval Quality
- Fluid Benchmarking with Item Response Theory