noun · AI performance
Latency
also called response time
How long you wait. It is the gap between pressing send and seeing the answer, and it depends on how much there is to read and write.
The time a request takes, usually measured as time to first token and total time to complete the response.
Total time is the wait for the first token, which includes processing the whole input, plus the time to generate every later token one at a time. Longer inputs raise the first wait, longer outputs raise the total, and queuing and network add to both.
Real-life example
Your chat feature feels slow. You find that each request sends a huge pile of instructions and files before the question. The model spends most of the wait just reading. Trimming the input, caching the repeated part and streaming the answer make it feel quick without changing the model.
Where the analogy breaks
-
There are two waits, not one. The time before the first word and the time to finish are different, and different fixes help each.
-
Streaming improves how fast it feels, not how long it takes. The same text still has to be generated.
-
Bigger models and longer prompts are slower. Choosing a smaller model or sending less can be the biggest speed-up.