As it largely depends on the prompt, no. I’ve seen prompts that ate kilotokens and took ages, and others that barely scratched the budget and returned immediately.
You simply cannot give a generalized answer here.
It’s like asking “How much does a visit to the ER cost?” It depends on if they have to put on a band aid, or put you in the ICU for a few weeks.
It’s like asking “How much does a visit to the ER cost?”
Easy answer: it’s free. :D
Yes, here in civilized countries it is, but this was an example for Americans.
As you can tell from the previous answers: It depends.
The bigger an LLM is, the more power it uses. AI models can be quantized or distilled to yield smaller but less capable models. Providers may try to route you to the cheapest model that can handle your prompt.
Another question is the length of the output. The length of the input matters less but might be relevant for processing long texts.
The energy used for training is relatively insignificant once you average it over its lifetime. The energy efficiency of a particular data center will certainly matter more.
Providers like OpenAI claim that the typical query uses about 0.3Wh. That’s about the same as an idling phone charger uses in an hour; ie charger plugged into the outlet but not into the phone.
It will vary based on the llm, tokens and tool calls
Most of the power consumption comes from training and optimising models. You only interact with the finished product, so power per query is very low compared to that required to develop the LLM.
As far as I know it is still much more expensive compare to alternatives like grammar checker and web search. Especially give that model already searches the web on its own in many queries.
Just because inference is more efficient than an training, which consumes energy on the scale of nation states, doesn’t mean inference itself is econonical.
while this is true in isolation, the amount of users means that inference now uses more power than training for the large actors.
The question is about per-prompt, so number of users is not relevant. What may be more relevant is number of tokens in and out.
If anything, number of users will decrease power use per prompt due to economy of scale.
sure, hardware wattage × time taken per prompt. which model specifically are you referring to and on what hardware?
Edit:
say, for example, that i’m running a model that takes ten seconds to respond on my Radeon 7900 XTX. it’s power limited to 300W, but the rest of the system also pulls power during runtime so let’s call it 400.
to get watt-hours we take watts times hours. one second is 1/3600th of an hour.
that comes out to 400 × 10 ÷ 3600 ≈ 1.11Wh. so that’s equivalent of leaving a 6W LED light on for about 11 minutes, or an old-fashioned incandescent bulb on for 80 seconds.
For additional reference, boiling a liter of water takes just shy of 100Wh (not counting heat absorbed by kettles or heat distributed into the air): https://www.quora.com/How-many-watts-do-I-need-to-heat-1-liter-of-water
You’re looking for tokens. Prompts are broken down into tokens, which then are used to generate tokens in response. All are represented by large integers. The common metric is tokens/second, and if utilized correctly the GPU should pin at 100% usage while generating tokens. Calculate how many tokens per second it’s generating and how many tokens you’re using, times the wattage per second and you’re good.
You can use a wattage meter between your outlet and computer. I’ve tried that, and the usage is around the same as a graphically intensive videogame while it is generating.
How does a wattage meter on my computer measure power used by an LLM server someplace else per prompt?
I have a LLM server on my computer, so I can tell how much electricity it is using this way. It’s not somewhere else is how
deleted by creator
You asked if there’s a way to tell how much power LLMs use, you didn’t specify LLMs on a server you don’t have physical access to.
deleted by creator
There’s a huge difference between a model you can run locally and a chatgpt model.
Sure, but without actually knowing what kind of hardware the servers are running, what kind of software too, and what their service backend looks like we can’t say whether it is going to be higher or lower.
I think we can assume it’s nvidia H200 which peaks at 700W what what I saw on Google. Multiply that by the turnaround time from your prompt to full response and you have a ceiling value. There’s probably some queueing and other delays so in reality the time GPU spends on your query will be much less. If you use the API, it may include the timing information.
