Inference is a model doing its job: taking an input it has never seen and producing an output. In plain terms, it is prediction on demand. It is the counterpart to training, the expensive, one-time process of building the model. Training happens once; inference happens every single time a user hits your feature, which makes it the cost that scales with success.
Each inference call consumes computation, usually on a GPU, and that shows up as either a per-token API charge or a server bill if you host the model yourself. The larger the model and the longer the output, the more each call costs and the longer users wait for a response.
Latency is the side of inference users feel. A model that takes several seconds to respond changes what you can build; it rules out instant autocomplete but is fine for a report generated in the background. Streaming tokens as they are produced is a common trick to make the wait feel shorter.
Levers to manage inference cost include picking a smaller model where accuracy allows, caching answers to repeated questions, batching requests, and shortening prompts and outputs. For most startups, renting inference through an API beats running your own hardware until volume is high and steady enough to justify the operational load. That break-even point arrives later than most founders expect.