After a few years of rumors, Microsoft has lastly confirmed that it’s following rivals Amazon Internet Companies and Google into the design of customized processors and accelerators for his or her clouds. That affirmation got here right now as Satya Nadella, Microsoft’s chief government officer, introduced the Cobalt 100 Arm server CPU and the Maia 100 AI accelerator chip.
The transfer is a shock to exactly nobody, as a result of even when Microsoft doesn’t deploy very a lot of its personal chips, the actual fact that they exist means it may well negotiate for higher pricing from chip makers Intel, AMD, and Nvidia. It’s like spending a whole lot of thousands and thousands of {dollars} to avoid wasting billions, which could be reinvested again into the infrastructure, together with additional growth. Notably on the relative excessive price of X86 server CPUs and the outrageous pricing for Nvidia “Hopper” H100 and H200 GPU accelerators and, we presume, for the forthcoming AMD “Antares” Intuition MI300X and MI300A GPOU accelerators. With provides restricted and demand far in extra of provide, there isn’t a incentive in any respect for AMD to undercut Nvidia on value with datacenter GPUs until the hyperscalers and cloud builders give them one.
Which is why each hyperscaler and cloud builder is engaged on some type of homegrown CPU and AI accelerator at this level. As we’re keen on reminding individuals, that is exactly just like the $1 million Amdahl espresso cup within the late Nineteen Eighties and the Nineteen Nineties when IBM nonetheless had a monopoly on mainframes. Gene Amdahl, the architect of the System/360 and System/370 mainframes at IBM based an organization bearing his title that made clone mainframe {hardware} and that will run IBM’s techniques software program, and simply having that cup in your desk when the IBM gross sales rep came around despatched the message that you weren’t messing round anymore.
This is likely one of the causes, however not the one one, {that a} decade in the past, Amazon Internet Companies got here to the conclusion that it wanted to do its personal chip designs as a result of finally – and it absolutely has not occurred but – a server motherboard, together with its CPU, reminiscence, accelerators, and I/O – will finally be compressed right down to a system on chip. As legendary engineer James Hamilton put it so properly, what occurs in cell finally occurs in servers. (We’d observe that typically the converse can be true.) Having another at all times brings aggressive value strain to bear. However greater than that, by having its personal compute engines – Nitro, Graviton, Trainium, and Inferentia – AWS can take a fill stack co-design strategy and finally co-optimize its {hardware} and software program, boosting efficiency whereas hopefully decreasing prices to, thus pushing the value/efficiency envelope and stuffing it filled with working earnings money.
Microsoft acquired a later begin with customized servers, storage, and datacenters, however with the addition of the Cobalt and Maia compute engines, it’s changing into a quick follower behind AWS and Google in addition to others in he Tremendous 8 who’re making their very own chips for exactly the identical cause.
The transfer by Microsoft to design its personal compute engines and have them fabbed was a very long time coming, and admittedly, we’re stunned it didn’t occur a number of years in the past. It in all probability comes right down to constructing a great group when everybody else – together with a number of CPU and a complete bunch of AI chip startups – can be attempting to construct a great design group and get in line on the factories run by Taiwan Semiconductor Manufacturing Co.
“Being the world’s pc implies that we must be even the world’s finest techniques firm throughout heterogeneous infrastructure,” Nadella defined in his opening keynote on the Microsoft Ignite 2023 convention. “We work intently with our companions throughout the trade to include the most effective innovation from energy to the datacenter to the rack to the community to the core compute, in addition to the AI accelerators. And on this new age of AI, we’re redefining all the pieces throughout the fleet within the datacenter.”
Microsoft has needed a substitute for the X86 structure in its fleet for a very long time, and method again in 2017 it stated its aim was for Arm servers to be 50 p.c of its server compute capability. A number of years again, Microsoft was an early buyer of Cavium/Marvell with its “Vulcan” ThunderX2 Arm server CPUs and was on observe to be an enormous purchaser of the “Triton” ThunderX3 follow-on CPUs when Marvell determined in late 2020 or early 2021 to mothball ThunderX3. In 2022, Microsoft embraced the Altra line of Arm CPUs from Ampere Computing, and began placing them in its server fleet in considerable numbers, however all that point there have been persistent rumors that the corporate was working by itself Arm server CPU.
And so it was, and so right here it’s in Nadella’s hand:
We don’t know what Microsoft has been doing all of those years on the CPU entrance, however we do know {that a} group on the Azure {Hardware} Techniques and Infrastructure (ASHI) group designed the chips. This is similar group that developed Microsoft’s “Cerberus” safety chip for its server fleet and its “Azure Enhance” DPU.
The corporate supplied little or no in the way in which of particulars in regards to the internals of the Cobalt server chip, however the phrase on the road is that the Cobalt 100 relies on the “Genesis” Neoverse Compute Subsystems N2 mental property package deal from Arm Ltd, which was introduced again on the finish of August. If that’s the case, then Microsoft is taking two 64-core Generis tiles with the “Perseus” N2 cores with six DDR5 reminiscence controllers every and lashing them collectively in a single socket. In order that’s 128 cores and a dozen reminiscence controllers, which within reason beefy for 2023.
The “Perseus” N2 core meshes scale from 24 cores to 64 cores on a single chiplet, and 4 of those could be ganged up in a CSS N2 package deal to scale to most of 256 cores in a socket utilizing UCI-Categorical (not CCIX) or proprietary interconnects between the chiplets as prospects want. The clock speeds of the Perseus cores can vary from 2.1 GHz to three.6 GHz, and Arm Ltd has optimized this design bundle of cores, mesh, I/O, and reminiscence controllers to be teched in 5 nanometer processes from TSMC. Microsoft did verify that the Cobalt 100 chip is certainly utilizing these manufacturing processes. Microsoft stated that the Cobalt N2 core would supply 40 p.c extra efficiency per core over earlier Arm server CPUs out there within the Azure cloud, and Nadella stated that slices of Microsoft’s Groups, Azure Communication Companies, and Azure SQL providers had been already operating atop the Cobalt 100 CPUs.
Here’s a shot of some racks of servers in Microsoft’s Quincy, Washington datacenter utilizing the Cobalt 100 CPUs:
Nadella stated that subsequent 12 months, slices of servers based mostly on the Cobalt 100 will likely be out there for purchasers to run their very own purposes on.
The Maia 100 AI chip might be the one developed beneath the code-name “Athena” that we now have been listening to about for greater than a 12 months and that we introduced up just lately as OpenAI, Microsoft’s massive language mannequin companion, was rumored to be taking a look at creating its personal AI accelerator, tuned particularly for its GPT generative AI fashions. This may occasionally have all been crossed wires and Athena is the chip the rumors about OpenAI had been referring too, or perhaps OpenAI is hedging its bets whereas additionally getting Microsoft to tune up an AI engine for GPT. Microsoft has been engaged on an AI accelerator for about 4 years, if the scuttlebutt is appropriate, and this may occasionally or is probably not the one it meant to do again then.
Right here is the Maia 100 AI accelerator chip that Nadella held up:
What we will inform you is that the Maia 100 chip relies on the identical 5 nanometer processes from TSMC and features a complete of 105 billion transistors, in line with Nadella. So it’s no light-weight in the case of transistors or clock velocity. The Maia 100 chip is direct liquid cooled and has been operating GPT 3.5 and is powering the AI copilot that’s a part of GitHub proper now. Microsoft is increase racks with the Maia 100 accelerators and will likely be allowed to energy outdoors workloads by way of the Azure cloud subsequent 12 months.
One of many neat issues in regards to the Maia effort is that Microsoft has designed an Open Compute suitable server, which holds 4 of the Maia accelerators, that slides into the racks it has donated to OCP and has a companion sidekick rack that has all the liquid cooling pumps and compressors to maintain these gadgets from overheating and permitting them to run hotter than they in any other case would possibly with solely air cooling. Have a look:
The Maia 100 is designed to do each AI coaching and AI inference, and is optimized for giant language fashions – and presumably, given how briskly this a part of the IT trade is altering, goes to have the ability to help different fashions apart from flavors of OpenAI’s GPT.
The opposite attention-grabbing factor is that Microsoft goes to be utilizing Ethernet interconnects to lash collectively the Maia accelerators, not Nvidia’s InfiniBand.
We will likely be poking round to get extra particulars on the Cobalt and Maia compute engines.
