You're telling me your datacentre is a 1GW generator now?
Sup nerds,
Just imagine I look this cool
wanna learn about how the datacentres are being sold as a grid asset, rather than a liability?
Oh man, not more AI stuff. I just want to go back to-
Grabbing you roughly, and in the voice of a man having recently seen Cthulhu, or having built a flux-capacitor recently:
Listen Morty… er… Marty… Whatever: It’s too late! They’re beyond control now! Even if the infrastructure bubble collapses and the big labs die, the cheap GPUs will be gobbled up by the next wave of neoclouds! The next wave of model companies will sell models wholesale to companies with inference, who will then sell arbitrage! The existing open models will be the floor of the market and will enable it to set sustainable prices! They’re already planning for it! — and that assumes the US Govt doesn’t bail out the AI capability to continue their fruitless competition with China! The colour has stained the very world we live in! We will never be rid of them! The madness will never end!!!
uh… Ahem. Anyway. The good news is we can still have fun.
This is more about the wild world of datacentre investments anyway! The front line of the bubble itself. Shockingly the companies building DCs (at least in Australia) don’t have the kind of capital lying around to just up and build them, and so are going out to investment firms (and just really rich people) to get the dollerydoos to stand these things up. I’m trying to keep this post short so that I actually get it done, so let’s get straight into it:
The Returns Are Infinite Finite But Real Good*
Aight so what are investors currently being sold on? Changing the world? Being part of the permanent overclass? Infinite power and such?
It may shock you to learn it is not the usual AI booster garbage, it’s literally standard “if you put in dollars, you will get out dollars” investment math. I’ve talked to people at the cutting edge of this fundraising whirlwind, and the basic rules are something like the following:
Graphics cards are expensive, but once you have them, even with cost of purchase & running, over the lifetime of the GPU you make something like 3-8 USD per user per day, depending on their usage. Given that each (good, Nvidia) GPU can serve something like 10 users at a time with most of the good models, that’s a lotta money very quickly over a DC. The returns are very, very good…
Typical investors seeing these numbers
… assuming that you actually have stable, high usage across the whole DC. Given that something like 1% of all users generate 80% of all revenue, and that we’ve decided to build terawatts of capacity already… I mean… those numbers don’t turn up in the investment pack is what I’m saying.
The point is, investment looks very good from the outset. The problem is Social License. Basically, us non-giga-chud-capitalists for some reason hate people putting DCs in across the road from us. Somehow they’ve gotta convince us that it’s a good thing. They try to tell us this will produce jobs (and I mean, even if the number is one janitor dude that’s still a job produced so take that with a huge grain of salt), they make sure they have to build out energy and water supplies (though this is more of a “lol Mr Billionaire has to pay for Australia’s future green energy roflmao” play by the federal govt here)… I mean there’s not really much more they can do, but they’re trying.
The nice thing (for the DC builder/operators at the very least) is that they don’t have to deal with us. They just have to deal with the government (local, state, fed… whichever). Governments are way easier to negotiate with, because they look for things that are wins for their constituents. They’re pragmatic (usually), so you can negotiate with them. So… I mean what do governments really want?
What If I Told You This Was A Giant Warehouse Shaped Battery?
For reasons that will shortly become apparent, they’re calling these things AI Factories. The reason is that they tend to use hundreds of megawatts into the low gigawatts of power, and before this all started the only things that used that kind of power consistently were steel foundries and car factories and stuff like that.
Your average datacentre without cooling, allegedly
Factories use power pretty consistently. Blast furnaces suck to restart, so you just kind of keep it all running, which uses a lotta power. Plus, if your car factory is mostly automated anyway, there’s no real reason to stop at any given time. And if we were getting like 80% usage across a neocloud DC, well you’d expect to see very similar power utilisation.
This is one of the Big Problems of DC deployment. It uses the same kind of resources as a factory! Which uhhh… kinda makes putting it anywhere that isn’t an industrial zone pretty freaking terrible for the general infrastructure of the area. If a local government was being asked to allow a DC in, well they’re looking at some chonky amount of water usage and an actually insane amount of power usage that might very well piss off their constituents.
But, aha, there is a difference! Datacentres are just a bunch of computers, they can be started and stopped at will! Plus, GPUs are the bulk of the power usage, and they’re only using power when they’re actively doing their thing (crunching ungodly huge matrices at speeds that boggle the imagination). So if you just… reduce inference somehow… you can suddenly create headroom on the power grid! Normal factories can’t do that.
So, assuming you’re a 3GW DC, and you rock up and go “Hey look it’s gonna be a big 3GW of data draw, and we’re gonna fund the power supply but it will require upgrades to the local grid… Howwwwwever, should you need it, we can just drop usage down and give you something like 1.21GW of headroom”… that’s a pretty good deal!
Your average local pro-industry councillor hearing that
But, how?
Well this part’s easy: They just pick up the inference, and move it someplace else!
Literally the strategy. I am not kidding.
Context Window? More Like Context IMAX Screen.
If you don’t know much about the LLM inference process, this might seem like something that sounds particularly easy: Surely you just send the prompt to another server?
Well, yes. But also no. You see there’s more to it than that: Prompts and the text the LLM is working on are not operated on directly. They are first embedded and converted into a series of high-dimensional vectors that are then stored in a huge stash called the KV store. This is, surprisingly, not a Key-Value system, as the letters would imply.
To simplify some math, every new token in the system produces three vectors, the Q vector, the K vector, and the V vector. You do not need to know what each vector represents, you are not trying to become the omnissiah. If you want to learn this to become the omnissiah, please go elsewhere as we only stan explainable AI systems in this house.
Each new token is produced in a process that requires every K and V vector in the entire sum total of text that is sent to the LLM. That includes the system prompt, every prompt you sent, the responses the AI sent, any injected memories, every tool call and all file contents that have been read/written if you’re using an agent… and so on. Basically everything in the whole bloody conversation up to this very next token. And that’s gotta be sent through every. Single. Time. You infer a new token. It’s not like the system guesses each token from the last one, it has to re-check the entire fucking conversation up to this point every time it does one step.
You may now be getting some idea of why this is so damn inefficient and needs so many damn graphics cards.
Still, that’s just text right? Context window of 256k tokens like the main big boy models have? What’s that… couple tens of megs? Nothing! But aha! It’s not text, it’s high dimensional K and V vectors, made of floating point numbers. Plus, each token has different vectors for each layer of the transformer network. And transformers have some number of attention heads where they make contact with the state of the input, which each have their own set of layers and therefore vectors. This complicates things significantly.
Let’s say we have a small, at home model… ahhh… Qwen 3.8 27B. That’s a 27 billion parameter transformer released by Alibaba. It’s capable of being squashed into a consumer grade graphics card, but a really big consumer grade graphics card. Or an Apple Silicon Mac with Beeg Memory. This is small fry stuff, is what I’m saying. The reason we’re picking this thing is because I know the numbers (as it’s open weight), rather than guessing at what Fable 5.1 actually has.
Okay so Qwen 3.8 27B has 4 heads, 16 layers per head, and each vector has 256 dimensions. Using half-precision floating point (BF16 values, two bytes, half the size of your average float, and a quarter of the size of a double — no this isn’t cheating, these are standard stuff here) numbers, a full 256k context window (which consists of actually 262144 tokens, fun fact) can be calculated thusly:
Which is 16 gigabytes. Which is an awful lot more than a few megabytes of text. The big models use way, wayyyyyyy more.
This, by the way, is the actual context window. When you compress a context window to continue on going, you’re getting an LLM to summarise the content of the context window into text, then converting the text back into a new KV store, and then continuing on as normal. Nearly all of the time (and energy) in this process is generating the summary, KV conversion of existing text is orders of magnitude faster than using it to generate new text. Like in my at home experiments it averages at about 20-30x faster. Remember that for later.
Now we actually have some neat tricks up our sleeve to make this less impactful on the poor GPU, as massive context windows rapidly become the limiting factor on GPU usage. Firstly, we can make it more expensive! Usually users are charged (ostensibly anyway) based on their usage of a GPU. Most users have a relatively small slice of the H200 they find themselves on, but they can have more of a GPU if they pay more of a share. If you’ve ever wondered why the “Fast” option costs more, or why there’s a price break once your context window gets over a certain number of tokens in big providers, that’s why! You’re using more of a GPU than usual, so you get charged more of a share than usual. If it’s more expensive, people will use it less, GPU can do more work in parallel, provider make more money, etc.
We can also shrink the physical size of the context window by just using less bytes. Quantisation is all the rage in the LLM space, most LLMs you’ll use are in fact quantized in one way or another. If you don’t know what quantisation is, it’s the process of reducing the bytes you use in a thing without losing (too much) information. Ever wonder why GIFs look crunchy? They only allow 256 distinct colours in each frame! By reducing the number of options to just 256, you can reduce the number of bytes you need to encode that information. And you don’t mind, it still gets the point across! That’s quantisation. Why’s the FLAC 20 megs and the MP3 2 and yet you can’t tell the difference? Really smart quantisation! Why is it that a Blu-ray rip won’t fit on my 16GB thumbdrive yet I can stream them all day across Netflix? Quantisation strikes again! It’s not the same method, but the goal is always to reduce the number of bits you need to represent a thing by just dropping options where you don’t need them. Some Star Wars frames have just black and white and some grey in them. So why bother wasting bits for representing other possible colours in that frame, you know? Honestly video compression techniques in this vein are a massive rabbit hole and if you don’t know you don’t want to know how much of the actual video they are cutting out to push it down the internet pipe at you.
Anyway where was I? Ah yes, Quantising things. Turns out, you can quantise the whole Transformer most of the time, sometimes all the way down to 2 bits in some places! That’s exactly 4 numbers you can represent! But if you can shuffle the math one uses to represent continuous real numbers on discrete binary bits the right way, those four numbers are just the numbers you need. Warping the number line in Lovecraftian ways has let us cram huge models into tiny, tiny places. For example, there’s a quantised variant of the Qwen 3.8 27B model that fits in just 4 gigabytes of RAM! Why? Whole chunks of it are just 2 byte values! And it retains something like 87% of it’s usefulness, which is frankly bananas.
We usually quantise KV stores to 8 bits, which halves the amount of RAM you need. If we’re real stingy, we go down to 4 bits, but that starts having a pronounced effect on accuracy. Free tiers tend to use 4 bit quantisation of KV caches pretty aggressively, because the answer being shittier is fine if you’re not paying money. Remember that next time someone tells you the model you tried for free and realised was kinda garbage works super good for them on their 200 buck a month plan — there might actually be a reason for that.
This is still in the gigabytes though, which… I mean if you’re trying to move a substantial portion of sessions off a given DC’s GPUs, you’re still jamming hundreds of gigabytes down an at best couple of hundred gigabit pipes. Not ideal! So even with the best we can manage in squeezing information into fewer and fewer bits… we’re kind of screwed.
So given that’s the problem… how are they doing it?
Continuity Of Consciousness? More Like Continuity Of — What Were We Talking About?
Fun fact: The AI doesn’t have continuity of consciousness. This has blown people who aren’t into this shit’s mind, and I’m sorry if I’m about to blow your mind too, but the bot does not actually have any continuity of being when talking to you.
See, every time you do a turn, it generates some text, and then that response goes back to you. You then spend seconds to minutes thinking at human speed (real thinking btw, you aren’t producing a slurry of intermediary vector states to hopefully direct your final response to the LLM in a more accurate way), and send a response back. That response? Totally different graphics card sometimes. Your AI girlfriend exists in instant snapshots, she can’t think of you between the times she generates text because she literally doesn’t exist. Also she doesn’t think and also you should probably get psychological help.
Every time the model gets your text, it has the text of what was said in this conversation for every instance of the model before it, the system prompt, your responses, and all the tokens that came out of tool operations. It then takes up the mantle of your sweet assistant bobot and craps out a response. Then, it is summarily banished into the aether. It’s not a continuous experience for the AI, it’s not there between turns.
But wait, it gets weirder!
Aaaaactually, the only time the model exists in any meaningful way… is during the time it is guessing the next token. That means it’s even worse than the per-response thing. It actually gets brought into existence, gets every token up until now, then takes up the mantle of your sweet assistant bobot and generates exactly one token, part of a word, a character, punctuation mark, something… and then is snapped out of existence Thanos-style, only to be replaced by another model instance which does the same thing for the next token.
Trying to make sense of this when anthropomorphising the LLM is really hard. Which is great! Because none of this happens because it’s not alive and has no sense of self or sense of anything. It’s math in motion, about as intelligent as the graphics output of CounterStrike. You don’t ask CounterStrike for help with your love life — at least I hope you don’t — because it’s not smart, or full of particularly nice people who won’t teabag you after they bean you with a PP Bison. That said, CounterStrike doesn’t give you the feeling of something that knows you and cares either, so fair enough.
To make things more bizzaro, they aren’t random either. It’s really hard to do random on computers, because random isn’t procedural (duh), and can’t be calculated (also duh), and all computers are big calculators. So technically if you use the same starting parameters and the same prompt you’ll get the exact same output (same thinking and everything)… which is really weird — unless of course you remember that it’s just math and not alive or thinking.
Now, this actually presents us with a solution to our moving context windows around problem!
Language Is Lossily Compressed Thunks
One of my favourite thoughts to have in a sort of I-don’t-need-drugs-to-come-up-with-some-wild-shit way is that language itself is a lossy compression format of thought. We quantise our actual brain state into sounds that come out of our face holes, go into someone else’s head holes, and they then decode that series of sounds into thoughts too. And if you have similar backgrounds, knowledge, concepts, and just the same sound-protocol… you will extract very similar brain states to the sender. This is probably a stupid series of thoughts that seem profound when it’s 2am and you’re out around a campfire with a bunch of friends (even more so if alcohol is involved), but, it is true at least in the LLM sense.
You see, if you retain the several megabyte sized text representation of the context window, and the initial stating parameters, and the same model… you can recalculate the entire KV store from scratch, token by token! And, because this is at least an order of magnitude faster than generating tokens, you can do this relatively cheaply if you need to!
This results in an incredibly stupid, but obvious, way of sorting this data-transfer thing out. You yeet the text and the starting parameters from one datacentre to the next, and just route user data to the new instance once it’s done calculating the KV store! It’s so incredibly simple… except… when?
Users tend to get all squirrely if the agent takes too long once you hit enter. Generally they want the AI to start Doin Stuff right the fricc now. If it mysteriously paused for 30 seconds half way through a turn, they might decide that your service is stinky and switch somewhere else. Not what you want as a DC operator! So, say you’ve made a deal to drop 1.21GW of usage real fast when the grid asks for it… how do you drop that usage without harshing the customer’s vibes?
Humans Are Terminally Slow
The solution is absolutely hilarious: You just move people across in the breaks between turns. Humans take seconds to minutes to think about what the hell they’re doing next, or to ponder on what the LLM output… or to copy and paste that shit into LinkedIn or their email or whatever cursed and unholy act they think is acceptable. Computers operate on the scale of microseconds and nanoseconds. Even though calculating like a million tokens worth of KV cache is an enormous operation, that time the user spends thinking is more than enough to do the bulk of a transfer-terminate-recalculate process, and the user will likely have no idea their best buddy software dev in their computer just astral walked from Vancouver to Sydney in the time it took for them to formulate their next question about NextJS or whatever. This means that the transition to low power usage isn’t instant, but it’s still fast. Way faster than if you asked a real factory to power down (assuming you could power it down). And still easily fast enough to be extremely useful in a pinch.
Being able to reduce usage as a massive industrial grade consumer of resources looks a lot like adding power, so you can legitimately get away with calling yourself an asset to the power grid rather than a liability, as insane as that may sound. And “Asset to the power grid” sounds an awful lot like a win some local politician can sell to their constituency. The wacky bit though is that the power problem they are dealing with is likely almost entirely caused by the DC in the first place, the only other problem we’ve got down here in Aus is the big spike in energy usage whenever the ACs are all turned on when people get home from work… and frankly we’re doing a pretty good job resolving that with solar+battery rollouts. So I mean, humans are terminally slow both literally and figuratively it seems.
Was There A Point To This?
Terence Tao has recently posted about AI having the problem of being too solution-minded. If you ask for an answer, and get an answer, you lose all of the little discoveries and skills and serendipitious occurrences that happen while you’re trying to solve the problem. This is why there are so many math problems that don’t seem to actually solve anything beyond some purely mathematical interest: It’s not always about solving the problem, it’s about what you learn along the way. The journey is always more valuable than the destination.
You may now know a bunch more about the mechanics of LLM technology, and may also have a more concrete feeling about the inhuman-ness, or even false-ness of modern AI. You may also now have an idea of why people keep building datacentres everywhere, it sounds like really good money! All of these are good things to have in your memory somewhere, and none of them you would have learnt if you hadn’t read this. Journey > Destination, etc etc. Doing this kind of meandering stuff is extremely human, and you should totally feel positive about it.
Plus, and this is really important, I like writing and sharing dumb things I learn. AI stealing all the written text in existence hasn’t stopped me, so I doubt the perceived dumb-ness of my blog post will either. So there!
Until next time!