LLM Token Counter

Each vendor splits text with its own tokenizer, and none of them publish how it works. OpenAI uses byte-pair encoding over a vocabulary built from their training data, which is why a common English word is one token and a rare one is split into three or four pieces. Anthropic and Google use different vocabularies with different training corpora. The same sentence can be 18 tokens on one and 21 on another, and no amount of cleverness in a third-party tool closes that gap, because the rule that decides it is private.

What is possible is to approximate the same kind of counting. This tool splits on word boundaries, treats leading whitespace as its own token, attaches punctuation to the word before it, and splits numbers into short runs of about three digits — all of which are behaviours real BPE tokenizers exhibit, and all of which are visible in their published examples. That lands within a few percent on ordinary English prose often enough to plan a budget with, and it is honest about being an approximation rather than presenting a guess as a measurement.

Four rules do most of the work, and they are worth stating because they are why the number differs from a naive word count. Leading spaces are their own token. Real tokenizers spend a token on the space before a word, which is why a 100-word paragraph is more like 130 tokens — this is the single biggest reason word count understates cost. Punctuation attaches to the preceding word. A comma does not get a token to itself. Long words split. Anything past about seven characters is broken into roughly four-character pieces, which is how subword tokenizers keep rare words from needing thousands of vocabulary entries. Numbers split into threes. A long number costs several tokens, which is why IDs and timestamps are expensive to paste.

Beyond that, accented Latin text is counted per character rather than per word, which is deliberately conservative and tends to over-count, and CJK is counted at close to one token per character, which is close to how Chinese and Japanese tokenizers actually behave. Code is counted conservatively too, because punctuation density makes it expensive in practice and the estimate should not flatter it.

Costs are shown per call at the input rate, and separately for a 1,000-token answer, so you can see which half of the bill a long conversation is driven by. The model list is priced per million tokens and kept in one configuration file, because vendors change prices more often than this page changes. Self-hosted models are shown as free per token, which is not quite true — there is a GPU somewhere — but per-token pricing is genuinely the wrong model for them, since the cost is per hour of hardware rather than per unit of text.

The comparison table shows what the same text would cost across every model in the list, along with each context window and how much of it your text uses. It is the fastest way to notice that a document which is unremarkable on a million-token model is over half the window of a 128k one.

Everything happens locally: the segmentation, the arithmetic, the cost table. Nothing is uploaded, which for the kind of text people paste into a token counter — unreleased documentation, customer records, anything under an NDA — removes the question entirely rather than asking you to trust a privacy policy.

The browser implementation and the server implementation are held to the same output by a test that compares every field of the result for a range of inputs. Two copies of the same logic drifting apart is the standard way a tool like this starts lying, so the test exists to stop it, and it covers the awkward cases: tiny costs that would print in scientific notation, empty input, and text with no spaces in it at all.

A request that exceeds a model's context window does not degrade politely. It fails, or the framework quietly truncates the middle of your prompt and hands the model a document with a hole in it, and the resulting answer looks confident. Cost is the limit people budget for; context is the one that breaks first, because the failure arrives as a bad answer rather than an error.

That is why the count and the window are shown together. When the estimate is close to the limit, the useful move is to cut the prompt before you spend anything on a request that cannot succeed — and the optimizer next door will tell you which lines in it are padding rather than instruction.

Leave headroom rather than aiming at the ceiling. Providers reserve part of the window for the reply, and a prompt counted at exactly the limit leaves no room for the answer the model is supposed to give.