AI token cost comes down to one blunt fact. Every single token a model reads or writes gets recomputed, every time, across the entire conversation so far.
That is not a pricing gimmick. It is how these models actually work.
It is the real reason the bill can balloon so fast, especially with coding agents.
I watched a Computerphile breakdown of this recently. It finally explained something I had been paying for without fully understanding.
Here is what a token actually is, why the cost scales the way it does, and why so many AI tools are quietly changing how they charge for it.
I run my own AI operating system on top of this exact cost structure every day, so this is not theoretical for me.
What is a token in AI
A token is a small chunk of text, not a whole word. It can be a full word, part of a word, a punctuation mark, or even a space.
A rough rule of thumb is that one token is about four characters, or roughly three quarters of a word in English. The exact split depends on the tokenizer a given provider uses.
A short sentence like “the cat sat on the mat” does not split cleanly into six word-tokens. Spaces, punctuation, and common word fragments all count.
Longer or rarer words often get broken into two or more pieces. Short, common words usually stay whole.
Models do not read text the way people do. Everything gets broken into these chunks before the model ever sees it.
This matters for cost because pricing is not based on words, or on how hard a question feels. It is based on how many tokens flow in and out of the model.
How tokenization turns text into something a model can use
Before a model can do anything with your text, a tokenizer splits it into chunks based on frequency. Common fragments like “the” get their own token because they show up constantly.
Rarer words get split into smaller pieces instead.
A modern model typically has access to something like 100,000 possible tokens, as NVIDIA explains in its own breakdown of AI tokens. That set covers standard English, other languages, code symbols, and special characters.
Most of the actual traffic runs through a much smaller subset that gets reused constantly, even though the full vocabulary is far larger.
Each token then gets converted into a numerical vector called an embedding, a representation the model can actually compute with. That embedding step is learned during training and is specific to each model, even when providers reuse similar tokenizers.
What actually drives AI token cost
Every AI provider charges by the token, split into input tokens and output tokens.
Input tokens are what you send in. Output tokens are what the model generates back.
Rates vary by model and provider, but a common range is somewhere around two to three dollars per million input tokens, and closer to fifteen dollars per million output tokens.
That sounds cheap for a single question, and for a simple one it usually is. A short prompt and a short reply might cost a small fraction of a cent.
The catch is that models do not remember anything between turns the way a person does.
When I answer a follow-up question, I do not silently reread this entire exchange first. A language model does exactly that, every single time, because it has no memory outside of what gets fed back into it as input.
That is also why a model’s context window, its maximum token capacity per session, is not just a technical spec. It is a direct cap on how much history can keep getting reprocessed before something has to drop out.
Why AI token cost compounds the longer a conversation runs
Language models generate text one token at a time, in a loop. Each new token gets predicted, then added back into the input for predicting the next one.
A 500 token question that produces a 500 token answer is not really one small pass through the model.
If the model also produces, say, 1,000 tokens of internal reasoning before answering, the full input effectively gets reprocessed for every one of those output tokens along the way.
Providers use a technique called KV caching to avoid completely redoing that work from scratch on every single token. It genuinely helps with both speed and cost.
But the underlying cost still climbs as a conversation, or a single agent session, runs longer. More and more history has to stay in play with every new step.
Why AI token cost scales so fast in coding agents
A basic chatbot conversation barely dents this. A coding agent is a different situation entirely.
A coding agent does not answer once and stop. It thinks, reads a file, thinks again, reads another file, writes a patch, and checks its own work.
Every one of those steps gets added back into the running conversation for the next step. It works the same way a follow-up question does, just automated and repeated many times in a row without a person in between.
One real example from that Computerphile video stuck with me. A single bug fix request, reading only two files, ran up to somewhere around 55,000 to 60,000 tokens by the time the fix was actually done.
A separate real session building a small screensaver project, just six prompts total, used something like 2 million input tokens and 47,000 output tokens.
That is not a bug or an inefficiency someone forgot to fix. It is how autoregressive models work by design.
Every step an agent takes on its own adds to the pile the next step has to carry. That is exactly the tradeoff I weigh when I decide how much autonomy to give an agent inside my own workflow.
The same math applies to the multi-pass review process I run for my own site. Loop Engineering deliberately runs several separate critic passes over the same work instead of one.
That catches real problems a single pass misses. It also means paying the token cost of rereading the same material multiple times, on purpose, because the tradeoff is worth it for anything high stakes.
Why is Claude so expensive
People ask this a lot. Usually what they are actually reacting to is token cost, not the sticker price printed on a plan.
A flat monthly subscription can feel expensive fast if you are running agentic work through it.
Agentic work is exactly the kind of usage that burns tokens fastest, for all the reasons above.
That is also why plans like Claude’s come with usage tiers instead of one flat unlimited price. I broke down what those tiers actually include in Claude Cowork pricing, including where the free tier stops and where Pro and Max plans start making sense.
The subscription is not charging a flat rate per question asked. It is capping how many of these expensive, compounding token passes a person can run before hitting a limit and having to wait or upgrade.
Why the GitHub Copilot pricing change happened
GitHub Copilot is a clear real-world example of this playing out in public. It used to charge a flat monthly fee based on request count.
It moved to a token-based credit system instead. For a lot of users that meant noticeably less usage per month than before, for the same subscription price.
That was not an arbitrary price hike dressed up as a feature change.
Flat-fee agentic coding tools were effectively subsidizing heavy users, because a flat fee has no real relationship to how many tokens an agent actually burns in a given session.
Once the true cost became visible internally, the pricing model had to catch up to it.
The same pressure sits underneath every AI tool offering agentic features on a flat fee right now. Token cost is the real constraint, whether the plan shows it to you directly or hides it behind a simple monthly number.
Why some companies measure AI adoption by token usage, and why that is a bad idea
There is a newer, odder wrinkle worth knowing about. Some companies have started measuring internal AI adoption by counting how many tokens employees burn through.
That treats token volume as a proxy for value delivered, which does not hold up. It is closer to judging a driver’s skill by how fast they wear through a set of tires.
An incentive like that quietly rewards long, rambling prompts and agents left to loop for longer than necessary, not better outcomes.
If token cost is the real constraint behind almost every AI pricing model right now, optimizing for more tokens instead of fewer is optimizing for the wrong number entirely.
AI token cost, quick answers
Do AI tokens cost money? Yes. Every major AI provider charges per token, split into input and output tokens, whether you see that cost directly through an API or indirectly through a subscription tier.
How many tokens is 1,000 words? Roughly 1,300 tokens for typical English text, using the common estimate of one token per 0.75 words.
Why do coding agents cost more than chat? Every tool call, file read, and internal thinking step gets added back into the model’s input for the next step. The token count compounds fast across a single session.
Does starting a new conversation save on token cost? Sometimes. A cached conversation that sits idle too long gets dropped, so picking it back up means the model has to reprocess everything from scratch again, a step called pre-filling. A short new conversation can occasionally cost less than reviving a long stale one.
Can I reduce my own AI token cost? Shorter, more specific requests help. So does avoiding situations where an agent has to reread large files repeatedly across a long session instead of working in smaller, targeted steps.
Where this actually runs
None of this is abstract for me. Every post on this blog and every workflow behind softDev23 runs on tokens I am actually paying for, plan tiers and all.
Understanding what actually drives the cost is part of what let me build a system that uses agentic work deliberately instead of by accident.
That system is my own AI operating system, running on the same token economics covered here, not a hypothetical.
If you want to see what that actually looks like day to day, join the AIOS waitlist.
