How to Actually Choose a Model
Price, speed, thinking modes, privacy and your own tests: how to pick the AI that fits what you do, instead of following hype.
Choosing a model is easier than the headlines make it look. You do not need the "best" model in the world. You need one that does your tasks well, at a speed and price you are happy with, and that treats your data the way you need. No leaderboard can make that call for you, because no leaderboard has seen your tasks.
The largest, most expensive model is rarely the right choice. A smaller, faster one often does everyday work just as well, and the gap people pay for is one they often never notice.
This piece has two halves. First, how to choose the AI you use day to day. Then, for anyone building a product on top of a model, the framework teams use.
Which AI should you use day to day?
Start free, then pay for limits
ChatGPT, Claude and Gemini all have a free tier, and each has a main paid plan for individuals at about $20 a month: ChatGPT Plus, Claude Pro and Google AI Pro, in US prices as of October 2026 (ChatGPT Plus, Claude pricing, Google AI plans). Around those sit cheaper tiers, such as ChatGPT Go and Google AI Plus, and plans from about $100 to $500 a month for heavy use. Paid plans mostly buy you higher usage limits and fuller access to each company's strongest models. Start with the free tiers, try the apps on your real work, and pay only when you keep hitting a limit.
Fast answers or thinking?
As What an LLM Actually Is (and Why Predicting Words Works) explained, newer models can write out working notes before they answer. The apps let you choose how much of that you want:
- ChatGPT has a Thinking slider in its model menu, from Instant to higher effort levels, and free users can switch on Think for harder questions (OpenAI's help page).
- Claude has a Thinking toggle and an Effort level in its model menu; on the newest models, thinking is always on (Anthropic's help page).
- Gemini offers thinking levels of Low, Medium and High, with Low as the default, and can pick a model and level for you automatically (Google's help page).
A simple rule: use the fast setting for quick, everyday questions, rewording and simple lookups. Turn thinking up for problems with several steps, such as math, planning, code, comparing options or checking an argument. Thinking takes longer and uses more of your limits, so save it for when it pays.
Check the privacy setting
Each app has a setting that controls whether your conversations may be used to train future models. In ChatGPT it is "Improve the model for everyone" under Settings, then Data controls (OpenAI's data controls FAQ). In Claude it is the model improvement setting under Settings, then Privacy (Anthropic's privacy help). In Gemini it is "Keep Activity", under Settings and help, then Activity, and for adults it is on by default (Google's help page). Look before you paste anything sensitive. A work account may have different rules, so check with your employer.
Run your own ten prompt test
The best evidence is your own. Pick ten real tasks you actually do: an email to rewrite, a document to summarize, a question from your field, a small plan to make. Give the same ten prompts to two or three apps. If you can, hide which answer came from which app before you judge them, so a favorite name does not sway you. Note which one wins on each kind of task, and how long you waited. Ten minutes of this tells you more than any leaderboard, and you will often find that different tools win different jobs.
And whichever you choose, keep checking facts, names, numbers and quotes, as Why Models Make Things Up explained.
The landscape in two tables
Here are the main families as of October 2026. Model names and prices change often, so check the provider's page before relying on them. Prices are what developers pay per million tokens through the API, input then output; app users pay a monthly plan instead.
| Family (maker) | Current models | Window (tokens) | API price per million tokens |
|---|---|---|---|
| GPT (OpenAI) | GPT-6 Astra (flagship), GPT-6.1 Sol, GPT-6 Luna (smallest) | About 1.05 million | From $0.10 / $0.50 (Luna) to $10 / $50 (Astra) |
| Claude (Anthropic) | Claude Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5 | 1 million | From $0.10 / $0.50 (Haiku) to $10 / $50 (Fable) |
| Gemini (Google) | Gemini 4 Argon (flagship), 3.8 Flash, 3.5 Flash-Lite | About 1 million | From $0.30 / $2.50 (Flash-Lite) to $2 / $10 (Argon, rising to $4 / $20 on 1 December 2026) |
Sources: OpenAI's GPT-6 Astra page and pricing, Anthropic's models overview, Google's Gemini models and pricing.
Open-weight families you can download and run yourself, with the license that decides what you may do with them:
| Family (maker) | Recent models | License | Window (tokens) |
|---|---|---|---|
| Llama (Meta) | Llama 4 Scout and Maverick | Llama 4 Community License | Up to 10 million (Scout) |
| Qwen (Alibaba) | Qwen3.8 family | Apache 2.0 for some sizes, such as 27B; custom licenses for others | About 262,000, extendable to about 1 million |
| DeepSeek | DeepSeek-V4.1 | MIT | 1 million |
| Mistral | Mistral Small 4; Mistral Medium 3.5 | Apache 2.0 (Small 4); modified MIT with a revenue limit (Medium 3.5) | 256,000 |
| gpt-oss (OpenAI) | gpt-oss-120b and gpt-oss-20b | Apache 2.0 | 128,000 |
| OLMo (AI2) | Olmo 3 | Apache 2.0, with training data and code released | See the model card |
Sources: Llama 4 model card, Qwen3.8-27B, DeepSeek-V4.1-Flash, Mistral Medium 3.5, Introducing gpt-oss, Olmo 3.
If you are building a product
The rest of this piece is for people putting a model inside an app or a service. If you just want to pick a chatbot, you can stop here.
What does choosing a model actually mean?
For a product, choosing a model is not one decision. It is a chain that starts with your task and ends with a running system.
The first decision is about the shape of the system. Do you need retrieval augmented generation (RAG), searching your documents and putting the relevant passages in the prompt, so answers stay current and can cite sources? Do you need fine-tuning to enforce a format or style? Remember the recipe chatbot from earlier in this series: if it has to pull ingredients from a company database that changes every day, RAG is close to essential, and you now need a search system as well as the model that writes the answers.
You can also mix models. Many systems send simple questions to a small, fast, cheap model and hard ones to a slower, stronger one. The space of choices is large, but the principle is simple: rule out what cannot work before you measure anything.
How do you define your task precisely enough to choose?
"Make a good chatbot" is too vague to choose against. Write down the task type: sorting messages into categories, finding information, writing, or reasoning through problems. Write down what success means: accuracy, helpfulness, or a business number such as fewer support tickets. Then write down your hard limits: the slowest response you will accept, the most you can pay per request, and where your data is allowed to go.
Suppose your recipe chatbot must answer within two seconds for 99 out of 100 requests, what engineers call p99 latency. Any model that cannot do that is out, before you run a single test. Hard limits define what is possible, and everything outside them is irrelevant. That is why browsing leaderboards first is backwards: start from your own requirements, and use leaderboards only to find candidates that already fit them.
How do you build evaluation data you can trust?
You need test data that looks like the real traffic your model will handle. Teams usually assemble a "golden dataset" of real questions paired with ideal answers, drawn from real logs, from experts, or both. It must cover the real range of requests, lengths, difficulty and odd cases. Easy examples alone will mislead you.
For tasks with a clear right answer, scoring is simple: check whether the output matches. For open ended writing there is no single right answer to "what can I cook with chicken and rice?", so you need judgments of helpfulness, clarity and safety. People can score answers against a written rubric, or a strong model can act as the judge. Model judges scale well but have known biases: they can favor the first answer shown, longer answers, and answers from their own model family (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena). Check a judge against human scores before you trust it.
One rule matters most: keep part of your test data hidden from everything you tune. If you keep adjusting prompts until a test set looks good, that set no longer tells you the truth. Keep a held out set that nobody looks at until the final check.
What do benchmarks actually tell you?
Public benchmarks are useful for spotting candidates. They are not a verdict on your task.
MMLU tests knowledge with multiple choice questions across 57 subjects (Measuring Massive Multitask Language Understanding). TruthfulQA tests whether a model repeats popular misconceptions, such as the myth that we only use 10% of our brains (TruthfulQA: Measuring How Models Mimic Human Falsehoods). Both helped once, but the top models have outgrown MMLU: they now score above 90% on it, so a small gap between two leading models means almost nothing (Humanity's Last Exam).
Model announcements in 2026 quote harder tests instead. GPQA Diamond asks graduate level science questions that experts answer correctly but most non-experts get wrong even with web search (GPQA: A Graduate-Level Google-Proof Q&A Benchmark). SWE-bench Verified checks whether a model can fix real bugs in real code, judged by hidden tests. Humanity's Last Exam has 2,500 expert questions across dozens of subjects. Arena ranks models by which answers people prefer when shown two anonymous answers side by side (LMArena is now Arena). AI Benchmarks, Explained: What MMLU and Friends Actually Measure explains how to read all of them.
Benchmarks have limits. Models can be tuned toward public tests, small differences are often noise, and none of them measures your latency, your costs or your company's jargon. Use them to shortlist. Let your golden dataset decide.
How do you compare models when there are several goals?
You cannot reduce the choice to one number. Accuracy, speed, cost and window size all matter, and the model that wins on accuracy may lose badly on speed.
A simple way to see the trade-offs is to rule out any model that another candidate beats on every measure. What is left are the real options, each better on something and worse on something else, and you pick among them by what matters most to you. Engineers call this set the Pareto frontier. A 2026 study that folded accuracy, speed, memory and delay into one efficiency score found that small models of 0.5 to 3 billion parameters scored best on all five tasks it tested (Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models).
Cost is more than the price per token. If you run a model yourself, count the hardware, power and the people who keep it running, spread over everything it serves. The cheapest machine per hour is not always the cheapest per answer: how much work you keep it busy with matters more.
For the recipe chatbot, you might measure accuracy on your golden dataset, p99 latency at peak load, and cost per 1,000 requests. Say a small specialized model scores 87% at 200 milliseconds and costs $1 per 1,000 requests, while a giant general model scores 91% at 800 milliseconds and costs $50 per 1,000 requests. Users rarely notice a 4 point accuracy gap. They do notice waiting four times as long, and you would notice a bill fifty times higher.
- 87% accuracy on the golden dataset
- 200 ms p99 latency
- $1 per 1,000 requests
- 91% accuracy on the golden dataset
- 800 ms p99 latency
- $50 per 1,000 requests
How do you validate a model before full deployment?
Testing on your golden dataset is necessary, but not enough. Real users behave differently from test sets.
The safest next step is a shadow test: run the candidate quietly beside your current model, send it the same real requests, and log its answers without showing them to anyone. You see how it behaves on live traffic, and users see no change.
When you are confident, run an A/B test: send a small share of real users to the candidate and measure real outcomes, such as satisfaction ratings, finished tasks, and whether people immediately rephrase their question, a strong sign the first answer missed. Statistics then tell you whether a difference is real or noise.
After launch, keep monitoring. What people ask changes over time, and providers update and retire models. Log requests and answers, score a sample regularly with your evaluation set, and set alerts for drops. Choosing a model is not a one time decision.
Quick Reference: Key Decision Factors
| Factor | What to measure | Why it matters |
|---|---|---|
| Task fit | Accuracy on your own tasks or golden dataset | Public benchmarks do not predict your results |
| Speed | Time to the first words, and the slowest responses (p99) | People notice a slow start; streaming the answer helps |
| Cost | Price per request or per million tokens, input and output | Small gains in accuracy can cost many times more |
| Window | Tokens your typical input plus retrieved documents needs | A model with a 128,000 token window cannot take a 300,000 token task, such as a whole codebase |
| Thinking | Whether a reasoning mode improves your hard tasks | It is slower and costs more, so use it where it pays |
| Privacy | Where data goes and whether it can be used for training | Decides whether you can use a hosted model at all |
| Upkeep | Effort to run, monitor and update | Open weights give control but add work |
Frequently Asked Questions
Q: Should I just use the model at the top of the Arena leaderboard? No. Arena, which began as Chatbot Arena, was then called LMArena and has been simply Arena since January 2026, measures which answers people prefer across all kinds of prompts. That is useful for finding candidates, but it says little about your specific task, and nothing about speed or cost. Use it to build a shortlist, not to decide.
Q: When does a small fine-tuned model beat a large general model? When the task is narrow and well defined, especially sorting text into categories. One study found that smaller fine-tuned models consistently and significantly beat much larger models used without fine-tuning, including ChatGPT and Claude, on a range of text classification tasks (Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification). They are usually far cheaper and faster to run, too. Fine-Tuning, Explained: Teaching an Old Model New Tricks covers when fine-tuning is worth it.
Q: How many evaluation examples do I need? Enough to cover your real requests and the odd cases you care about. Start with as many as you can review carefully, run your candidates a few times to see how much the results move, and add examples where the results are close or unstable.
Q: Can I trust a model acting as the judge? Partly. With a clear rubric, a strong judge agrees reasonably well with people, but it can favor answers by position, by length or from its own model family. Check it against human scores on part of your data first, and keep people involved for decisions that matter.
Q: How often should I revisit my choice? Keep monitoring all the time, and look again properly when results drop, when your needs change, when a clearly better or cheaper model appears, or when your provider announces it will retire the model you use.
Test yourself
Your team is building a customer support bot that answers questions from a 10,000 page product manual. The manual changes every week. Your budget is $200 a month for the model, you expect about 2,000 questions a day, and answers must arrive within a second or two. One teammate wants the biggest model at the top of the leaderboards. Another wants to fine-tune a small model on a copy of the manual. Which approach is likely to fail, and what would you do instead?
Answer: Both are likely to fail. The biggest model fails on cost and speed. With search results in the prompt, a question might use 3,000 input tokens and 300 output tokens; at a flagship price of $10 and $50 per million, as of October 2026, that is about 4.5 cents a question, or roughly $2,700 a month at 2,000 questions a day. Fine-tuning a small model fails on freshness: it learns last week's manual, and retraining every week is slow and risky. Instead, use RAG with a small, fast model. Store the manual in a search index, find the relevant passages for each question, and give them to the model. At a small model's $0.10 and $0.50 per million, the same questions cost about $27 a month, the answers come quickly, and updating the index each week keeps every answer current without retraining.
- About $2,700 a month
- Slower answers
- Only knows what it learned in training
- About $27 a month
- Fast answers
- Always current: update the search index weekly, no retraining
That completes AI, From Zero. You can now follow the main ideas behind the AI tools you use, from tokens and attention to training, context windows, hallucinations, openness and fine-tuning.
If you want this kind of breakdown every week, the real internals behind the tools you use, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- What is ChatGPT Plus?, GPT-6 and other models in ChatGPT and Data controls FAQ (OpenAI, read October 2026)
- Claude pricing, Change the model, effort and thinking settings and Model improvement privacy settings (Anthropic, read October 2026)
- Google AI plans, Gemini thinking levels and Keep Activity (Google, read October 2026)
- GPT-6 Astra model page and API pricing (OpenAI, read October 2026)
- Models overview (Anthropic, read October 2026)
- Gemini models and Gemini API pricing (Google, read October 2026)
- Llama 4 model card, Qwen3.8-27B, DeepSeek-V4.1-Flash, Mistral Medium 3.5, Introducing gpt-oss and Olmo 3 (model makers, read October 2026)
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al. 2023)
- Measuring Massive Multitask Language Understanding (Hendrycks et al. 2020)
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al. 2021)
- Humanity's Last Exam (Phan et al. 2025)
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Rein et al. 2023)
- LMArena is now Arena (Arena, January 2026)
- Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models (Cao et al. 2026)
- Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification (Bucher and Martini 2024)
