Hope y’all enjoy!

The transcript below has been edited for conciseness and readability.

George Cozma: At AMD’s Advancing AI was last week, you guys made some announcements there about 3DIC, and I would just like to learn a little bit more about that and hear a bit more about how you guys are thinking about thermals, but not just thermals, things like current such how to carry current from package to the compute dies?

Ravi Subramanian: So maybe a good starting point is the two main trends going on in computing from a pure silicon perspective, chips are getting bigger and bigger to the point where we have some reticle-limited chips. So we see that companies are using 2.5D and 3D types of integration techniques to have multi-die solutions to handle the ever-increasing compute complexity.

Thermals have always been a problem in chip design, so it’s not new. The first wave of big thermal innovation really had to do with the early eras of CPUs. As CPUs were getting faster and faster, how did they need to allow the device to operate in a package that’s sitting inside a laptop or sitting inside a desktop. Then came the mobile communications world brought some severe limits, because it was the first time the whole chip operation was determined by a battery-powered device.

George Cozma: Holds up phone.

Ravi Subramanian: Bingo. Exactly. All of a sudden, what was needed in doing low-power design, in terms of the EDA tools to do low-power design, and what was needed in terms of managing thermal effects not just from the compute die, but also from the power amplifier, which is sitting not so far away from the compute die in that chip, and then the battery, which is also creating heat. So all of a sudden, we now have a system where it’s not so simple to put a fan in a chassis and control things. The mobile industry really created the first step-function in how you need to think about thermal from a system perspective.

Then fast-forward, we go to automotive chips, and then we see regulations or rules really determining how the chip should operate because of the environment that it’s in.

George Cozma: So for automotive, is it less the actual thermals of the chip and more the environment of where the chip is being placed?

Ravi Subramanian: So the environment is a bigger factor, but if you look at a typical mobile SoC, it’s about 2 to 2.5 billion gates. A typical automotive ECU chip now is about 7 billion gates. So it is bigger, 3x bigger, and there are big power challenges. As a simple example, the way that’s really measured now is, “Oh, it translates into a much smaller range for the car,” because of how much energy is consumed by the compute die. So Mercedes, BMW, the US car makers, and others have talked about what’s the range restriction because of how much [power] is consumed by the chip.

Then we get to the data center, which is now another type of system with another set of constraints around the chip, the package, the rack, and then the whole building that everything is sitting in. So that’s the scale of how the thermal problem has changed, right? With different system complexity and the importance of not only the chip power dissipation and the way packaging is handled, but also the environment and what the environment brings in.

So now, if we look at 3D-IC and chip-in-package, and maybe more specifically about AMD…

George Cozma: If I remember correctly, AMD announced at their Advancing AI event which was last week and while there Synopsys announced the release of 3DIC which came with AMD as the co-partner for it.

Ravi Subramanian: Yes, yes. So AMD has been a leader since their presentation at ISSCC, I believe it was in 2019, when they disaggregated the die, and then they showed what the advantage is. That was the first seminal paper in this field that really set the stage for what’s the thinking behind going multi-die, and obviously evolving that to 2.5D and 3D.

Now, the minute you go multi-die, you get a number of advantages in terms of the total cost of the solution because of the fact that you’ve really thought about which logic node should which processing be in. But then what you’ve also done is you created a problem in that you have to move so much data around, and the energy consumed by moving the data.

So to get higher performance, Moore’s Law only gave you so much. You pick an advanced node, you can get the highest performance there, but then within a package now, you have multiple dies, each one having power dissipation during operation, which is very much related to the workload running on it.

So the first step is understanding, for a given architecture with a workload running on it, exactly how much power is consumed and where in the workload the power is actually consumed. So that’s one capability needed early on pre-silicon, that’s a verification type of problem.

And then the second piece is, once you actually have a structure or an architecture, what happens is, because of the package and because of the die, there can be warpage, and there are mechanical effects that used to be second and third-order effects and those effects now [are first-order effects], because the chips are getting bigger, you actually have mechanical effects creating a strain in the silicon, and that changes the electrical properties.

George Cozma: Okay. I know the whole joke is: if you want the best thermal dissipation, make it the size of a wafer; if you want the cheapest cost, make it the size of your fingernail.

Ravi Subramanian: Exactly! So what we have then is, in order to be able to analyze all the effects, sign-off used to be, “I’m going to make sure the functional performance of the chip is okay, I’m going to make sure the electrical performance is okay.” But now, you need to look at the mechanical-electrical pairing. And then similarly, you need to look at the thermal, and that is, as the chip is running, you have IR drop, you have electromigration, and you need to be able to analyze the performance of the chip with those effects incorporated.

And then finally, the last one is electromagnetic effects. Electromigration has to do with electrons; electromagnetics with Maxwell’s equations. So a good way to think about it is there’s Moore’s Law coming one way, and then there are all the notions of material stiff equations, partial differential equations for looking at materials and then Maxwell’s equations for electromagnetics, right? So, you know, even though people say, “Well, Moore’s Law was doubling every 18 months and you had to deal with that complexity,” 10 years ago, many of the effects we’re talking about were second and third-order effects, and didn’t really affect the performance.

George Cozma: And now it’s first-order.

Ravi Subramanian: Now it’s first-order, exactly. So the definition of what you do to sign off a chip before you give it to TSMC means you not only do the classical electrical sign-off, but you now have to co-sign-off on mechanical-electrical, thermal-electrical, electromigration, and electromagnetic. Because as you get to smaller and smaller wires, these actually behave as antennas as the signal speeds increase.

George Cozma: Yeah, effectively everything starts to become RF.

Ravi Subramanian: Exactly. So even within a package, if you have very high-speed connections, essentially you have radiating wires, right? And they could create interference because one wire is next to another where it could create a crosstalk effect, etc.

So that’s the scope of the nature of the problems AMD had to solve in successfully getting out some of the chips they announced at the Advancing AI event. We worked very closely together with them—R&D teams working with chip designers to understand what do they want to analyze, and how do we make that happen pre-silicon. Because as you may read, some of these chips now, they’re very complex multi-die or 3D-IC, and they have complex packaging, and sometimes they find a problem only after it comes out.

George Cozma: I mean, I think Jensen famously talked about the issues that they had with Blackwell.

Ravi Subramanian: Exactly. And so now people want to very much say, “How can I bring the view of those problems upfront? How can I look at those earlier and be more confident about handling them?”

George Cozma: So, I guess, going to that second part, the whole way of powering a chip via basically through TSVs and through all that, how has that changed the way that EDA tools work?

Ravi Subramanian: Sure. So a good way to think of that is maybe I’m going to start with chip and then system, right? On the chip side, today, if you look at a typical blade that’s going into many data centers, on the board you have the chips, but you also have a lot of power supplies, LDOs and other types of chips, that are managing how current is distributed across the chip.

For a very long time, the standard architecture has been 54V. As you try to push more and more signal across, the current in these wires starts getting higher and higher. So there’s a trend, an initiative that’s happening that I’ll talk about briefly, but right now, what that means is you have to carefully manage the currents on the board and have these LDOs managing how these currents are fed into these packages and then ultimately fed into the chips.

If you have I/O chips that are primarily SERDES or driving signals from chip package off the package, they have a driver strength that’s a certain amount. Whereas what you have in terms of a specific digital processing chip, you have power domains even within that chip where you’re powering up and powering down, and you want to carefully manage how current is fed to that chip and regulated to that chip. So you literally have a world of regulators regulating how currents are fed to chips.

Now, a big key trend that’s happened recently is a view to change the data center architecture and the rack standard to 800V. That’s going to dramatically reduce the magnitude of currents, steering towards 800V as a standard. Nvidia and others are driving towards that, and companies such as Bosch and others are supporting standardized components for that.

So the architecture of a board, you can expect, is going to start looking very different. Because now you have 800V, much lower current, and what you have with respect to trace lengths and how much energy is lost, because you have a much lower current, so the $I^2R$ combination of how much energy is being lost, it completely changes that equation in terms of how effective we can be in the management of the delivery of power to every packaged device.

George Cozma: Okay. And so on these 54V systems, was there a lot more thought having to be put into where certain like I/O wires. For example, where you have these big NVLink racks, you have these switches, and you need to connect the GPUs to the switches was there a lot more consideration in 54V systems and racks to how you would lay out those wires [compared to 800V systems]?

Ravi Subramanian: Absolutely. So the two main things there would be: first, you want to have high signal fidelity, so signal integrity, analysis of signal integrity with various routing topologies for those wires. That’s one piece, and that’s really artists at work using tools to see that.

And then the second one is the rate at which data rates are increasing. With each new generation of chip, you have to feed the beast, which means feeding the data from memory to the chip, and then from the chip into a switch to somewhere else, right? Those speeds are growing dramatically.

What those interfaces look like, and the introduction of new types of links between chip-to-chip, Nvidia introduced NVLink to be able to have heterogeneous systems connecting; there’s the UCIe standard, the UAL standard, all of these about chip-to-chip. But with UCIe, it’s not one standard, every company has their own view of “We want 8 lanes, we want 16 lanes, we want 12 lanes,” right?

George Cozma: If I had a nickel for every time I’ve heard that this morning, I would have two nickels—which isn’t a lot, but it’s odd that it’s happened twice! [Laughs]

Ravi Subramanian: [Laughs] Yeah! So that’s where understanding how you’re going to deal with much higher speeds comes in. Ultimately, you have to do signal integrity, but the architecture of how you’re going to connect, what standard are you going to use chip-to-chip, package-to-memory, package-to-package, package-to-switch, all come into play there.

George Cozma: Moving sort of more from just the EDA side to the IP side, right? We’ve been talking a lot about 200G networking or 200G SerDes. Moving forward, 400G is going to become the standard, although Nvidia has said that they have 400G. What is the difference between the industry standard 400G and what Nvidia is using, which I believe is called simultaneous bidirectional 200G? What exactly are the differences?

Ravi Subramanian: Sure. So today, if we look at 224G SerDes or 224G Ethernet, that’s a standard that’s well-established. That’s where a lot of IP is getting sold and put into systems and chips.

400G is not yet standardized. That standards committee is still going, and we’ve got many companies making contributions, trying to steer the standard one way or another. And there, you have what’s called PAM signaling, right? And there are a lot of different proposals with variations of that.

George Cozma: I believe the current proposals are for PAM6 or PAM8?

Ravi Subramanian: Exactly. So that’s going on there, and there’s a lot of jostling to get to a 400G standard, but there’s also a lot of jostling to get something out in 400G, though that has to essentially get adopted and become a de facto standard. In the meantime, what we’re seeing with 224G is 224G plus optical is going to extend the life of 224G. So that’s at least the standards landscape.

Now, with Nvidia and bidirectional 224, that’s really trying to pull forward a higher speed capability, not quite 400 as a standard, but very high performance, greater than 220. And by having bidirectional capability, you are able to have much greater throughput into and out of systems. So that’s the current standards landscape.

In terms of when will 400G be a standard? The way that’s going, it’s probably going to be at least a year and a half to two years towards that, just looking at how the history of standards has been. What we’re also seeing is hyperscalers are trying to create a pseudo-custom version of the standard. The standard is relatively mature, but there are specific changes that they want to make to really support the systems that they are building for their workloads and their data center architectures.

George Cozma: Well, thank you for sitting down with me.

Ravi Subramanian: Thank you!

NVIDIA has published a 45-page whitepaper explaining Vera, its first server CPU built around the company’s own Olympus core. On paper, Vera is a fascinating chip with an 88-core monolithic compute die, Olympus being a 10-wide Arm v9.2 core that has value prediction, a graph prefetcher, 2 MB of private L2 per core, 164 MB of shared last-level cache, and eight LPDDR5X memory interfaces promising 1.2 TB/s.

Unfortunately, NVIDIA also spends a good part of the paper trying to turn those interesting design choices into a morality play about x86. Traditional simultaneous multithreading is drawn as time-slicing, a configurable NUMA topology is presented as an unavoidable 32-node maze, four SPEC components become “agentic benchmarks,” undefined performance-counter ratios are promoted as causal proof, and an unlabeled pictogram becomes a 1.8x reinforcement-learning result.

The frustrating part is that Vera does not need this help, with early independent testing suggesting Olympus is genuinely formidable. The whitepaper’s strongest case is the hardware; its weakest case is the story wrapped around it, so let’s pull that story apart.

Olympus Deserves Better Than This Marketing

Before getting out the cheese grater, let’s talk about the good stuff. Olympus is a very wide out-of-order Arm core.

Its front end can decode ten instructions per cycle and handle up to two taken branches per cycle. NVIDIA describes a neural branch predictor, value prediction, memory renaming, a large instruction window, six 128-bit SVE pipes, four load pipes, two store pipes, a 96 KB L1 data cache, and roughly 10-cycle access to a 2 MB private L2. Eighty-eight of those cores sit behind a 3.4 TB/s coherency fabric and a distributed 164 MB system-level cache.

Looking closer at the core, the value prediction is one of the more unique additions that Olympus has. This has been a research area for a long time and what value prediction allows Olympus to do is if the core correctly predicts a result, dependent instructions can keep moving instead of piling up behind a long-latency operation. Researchers have discovered that Apple uses value prediction in their cores and AMD talked about how in Family 17h (Zen 1 and 2) they could predict the value of some floating point instructions. However, AMD’s Family 17h implementation was quite limited, while Olympus appears to have a broader value-prediction implementation closer to Apple’s.

However, the graph prefetcher is not unique to NVIDIA. Intel has a similar mechanism called Data-Dependent Prefetcher that has been in shipping silicon since at least 2022. Intel’s newest datacenter CPU, Granite Rapids, also has an Array of Pointers prefetcher which “treats the data prefetched for a constant stride load as a pointer and may issue prefetch requests to the memory addresses corresponding to the pointer’s value.” This is fundamentally the same producer-consumer idea that NVIDIA describes for its graph prefetcher. Intel’s implementation is fairly constrained, so NVIDIA’s implementation may be able to deal with more complex chains than Intel’s implementation. So while Vera’s Graph Prefetcher may be an implementation that can deal with more workloads, producer-consumer prefetching is not a new idea.

Nor is a “neural branch predictor” a new idea. Back in 2012, AMD implemented a perceptron branch predictor in the Piledriver microarchitecture and continued to use a perceptron-based branch predictor in Zen 1. However, starting with Zen 2, AMD used a perceptron BPU only for its initial direction prediction, with a TAGE predictor overriding it because it delivered a 30% reduction in mispredictions. With Zen 5, AMD has likely fully committed to TAGE predictors, if it had not already done so with Zen 3 or Zen 4.

Moving to the SoC side, with how beefy the Olympus core is, NVIDIA has given Vera an equally beefy memory subsystem. Vera pairs eight SOCAMM2 LPDDR5X modules with up to 1.5 TB of capacity and 1.2 TB/s of bandwidth. NVIDIA claims the populated memory subsystem only consumes approximately 50 watts. A conventional EPYC or Xeon platform can offer higher-capacity DIMMs which are easier to replace, but it pays for that flexibility in board area and power.

Most importantly, we have more than NVIDIA’s results to look at. In May, Michael Larabel at Phoronix ran an early Vera system against current Arm and x86 servers. Across the NVIDIA-permitted test set, Vera’s geomean was 10% above a 5 GHz EPYC 9575F, 1.55x a Xeon 6980P, and 1.63x Grace which makes Vera the most performant Arm server CPU we have seen in public testing. There are major caveats with the testing, such as NVIDIA choosing the permitted workload scope and not allowing frequency or power monitoring. The system that Phoronix tested was pre-production and the test window was one day which puts a fairly hard limit on what they could test irrespective of the limits NVIDIA placed. This means that broader coverage will have to wait until Vera can be found in the wild rather than just in NVIDIA’s labs.

Still, the result is strong enough that we can reject the explanation that the charts in NVIDIA’s whitepaper are all fantasy. Olympus appears to be a fast CPU core, so now we can ask whether the whitepaper proves what NVIDIA says it proves.

Spatial Multithreading Is Still SMT

Here is the first major technical error in the document.

Figure 5 contrasts “Traditional SMT (x86)” with NVIDIA’s Spatial Multithreading. The x86 side depicts the branch predictor, decode, execution, load/store, and memory stages alternating between two threads. The caption says Vera avoids “opportunistic time-sharing” by partitioning resources across its two hardware threads.

NVIDIA’s diagram gives a misleading impression of how SMT is usually implemented, both on x86-64 and other ISAs. SMT implementations share various stages in the execution pipeline by either selecting a thread to service every cycle, or by behaving in a thread-agnostic manner. Fetch, decode, and allocate typically service threads on a per-cycle basis, while the execute and memory access stages are thread agnostic and can service micro-ops from both threads in the same cycle. Stages that threads arbitrate for do not leave resources unused when both threads can be fed, as NVIDIA’s diagram suggests. Static partitioning and per-cycle selection would provide the same average throughput to both threads in the absence of per-thread stalls. If there are stalls, per-cycle selection can give otherwise unused throughput to the un-stalled thread.

The same idea applies to thread agnostic stages like execute and cache access. Each thread is permitted to utilize as many execution units or cache ports as it can feed. In contrast, statically partitioning resources as NVIDIA suggests could lead to one thread being compute bound and unable to use half of the core’s execution resources because they’re reserved for the other thread.

Text in NVIDIA’s paper emphasizes “determinism, isolation, and quality of service” as advantages for NVIDIA’s Spatial Multithreading approach. Performance is conspicuously not called out. QoS may be a more important consideration than throughput for NVIDIA’s target market, and Spatial Multithreading may not be a bad design point. But NVIDIA’s figure makes it look like vertical space represents time, and gives a misleading impression that Spatial Multithreading is meant to give larger performance gains than traditional SMT.

By reducing resource interference between threads, Spatial Multithreading improves determinism, isolation, and quality of service compared to traditional SMT approaches. The result is a CPU architecture that can run large numbers of concurrent agent tasks while maintaining more consistent latency and throughput. - NVIDIA’s Vera whitepaper

Vera’s actual SMT performance is unknown of course, and a lot of variables go into SMT gains besides partitioning strategies at fetch, decode, execute, and memory access. Out-of-order resources like the reorder buffer, register files, and memory ordering queues can be duplicated, statically partitioned, watermarked, or competitively shared. Partitioned structures were split between the two logical processors in multi-threaded mode and recombined for one thread in single-thread mode, which was documented in 2002. Various SMT implementations use different strategies for each structure, and those choices can have significant implications for SMT gains.

Also something to note is that it apparently takes 10,000 cycles for an Olympus core to transition back to the single-thread mode once the sibling thread on that core is done. This means that software will have to be very aware about launching a second thread on an Olympus core due to the penalties incurred not only from the partitioning scheme but also from the delay of swapping back to a single thread.

It’ll be interesting to see what strategy Vera uses to partition its out-of-order resources, and how its SMT performance compares to that of other modern cores. NVIDIA’s whitepaper gives no information on that. What it does do is present a misleading diagram that suggests traditional SMT is prone to leaving resources unused, when it may actually be better at keeping the core fed than NVIDIA’s Spatial Multithreading.

The 32-NUMA-Node Straw Man

NVIDIA next tells us that a large two-socket x86 system can expose “as many as 32 NUMA domains,” while Vera presents one per socket. The number is not invented. On a many-chiplet EPYC system, an administrator can expose cache-local regions as separate NUMA nodes. If you turn every locality knob toward maximum granularity, the node count gets large.

What NVIDIA leaves out is that this is configurable with AMD’s tuning guide listing NPS4, NPS2, NPS1, and even NPS0 modes. The optional “LLC as NUMA” setting can expose each last-level-cache domain separately. So “32 NUMA nodes” is not the inevitable user experience of a chiplet CPU, it is one end of a locality-control spectrum. NVIDIA presents an optional high-granularity configuration as though it were an unavoidable reality of x86 systems.

Vera’s one domain per socket simplifies scheduling and memory placement, while multiple domains let tuned software exploit physical locality. Vera chooses the simpler presentation, and NVIDIA is free to argue that this better matches its intended software stack. But an OS-visible NUMA node is an abstraction, not a wormhole. Vera still has 88 cores, distributed cache and home nodes, memory controllers around a large die, and a packet-switched coherency fabric. A flat software topology can make those distances around the large monolithic compute die more consistent, but it cannot make them nonexistent.

The paper’s core-to-core heatmaps would be a good place to quantify that advantage. Instead, NVIDIA provides colored squares with no core identities, no minimum/median/maximum table, no distribution, and no measurement procedure. “Up to 50% lower” captures NVIDIA’s best result, not Vera’s typical behavior.

One NUMA node per socket is genuinely simpler, but the whitepaper compares it against an optional 32-domain x86 configuration and presents that edge case as the baseline. The counterpoint here is that Intel has a Mesh NoC just like Vera has. The difference really between these two setups is that the clustered setup of EPYC has high latency between clusters but within a cluster the latency is low, whereas Vera and Xeon Mesh setup has uniformly average latency; the different configurations are just engineering tradeoffs.

Turning SPEC into “Agentic AI”

The benchmark section is where the whitepaper, ostensibly about a CPU, starts wearing an AI conference badge it found on the floor.

NVIDIA selects four SPEC CPU 2026 integer workloads, CPython, GCC, LLVM, and Cppcheck, and calls them “agentic benchmarks.” SPEC itself describes them as a Python interpreter, two optimizing compilers, and a C/C++ static analyzer. Those are legitimate CPU programs. They stress large instruction footprints, branch-heavy code, allocation, and dependency chains. Agents can absolutely invoke programs like them.

But they are not agents: no model is serving tokens, no agent runtime is choosing tools. No sandbox is starting, blocking on I/O, retrieving context, evaluating an answer, or feeding observations back into a policy. These workloads may be useful proxies for the code-heavy portions of an agentic pipeline. Calling them “agentic benchmarks,” however, turns that partial overlap into a claim that they represent the complete end-to-end workload.

The paper does correctly label the SPEC results as estimates, because the Vera reference hardware was not generally available at the time of the run. Figure 15 shows a 1.7x to 1.8x advantage for the four selected components, normalized per physical core under a fully loaded two-socket system. Flip to the configuration pages and the full estimated SPECrate 2026 Integer Base totals are 925 for two Vera sockets and 898 for two EPYC 9755 sockets which is a 3.0% system-throughput advantage.

Both numbers can be true. Vera uses 176 physical cores across two sockets, while the EPYC system uses 256. Divide each score by physical-core count and Vera is about 50% faster per core across the full integer-rate suite with the selected tests reaching 70 to 80%.

There is another terminology collision. NVIDIA calls Figure 19 “single thread IPC” while describing a fully loaded system.

The published configuration runs 352 copies on 176 Vera cores and 512 copies on 256 EPYC cores, two copies per physical core. Maybe NVIDIA sampled one logical thread while its sibling was active, maybe it aggregated counters and divided, the paper does not say. The SPEC results are useful, and Vera’s per-core performance is genuinely strong. However, framing those tests as agentic workloads and emphasizing normalized figures makes the advantage appear broader than the disclosed results justify.

IPC Without Instructions

NVIDIA attributes Olympus’s reported IPC lead to four counter groups. Depending on the selected workload, Vera supposedly achieves up to 2.3x more branch predictions per cycle, 3.5x more taken branches per cycle, 2.4x more instruction-fetch operations per cycle, and 4.3x more backend operations per cycle.

That sounds technically specific, but it is impossible to audit without the PMU event names and definitions, raw counts, sampling intervals, clock frequencies, etc. Not to mention that an Arm instruction is not the same unit of work as an x86 instruction. An internal backend operation is even less portable: one microarchitecture may split an instruction into several micro-operations while another keeps it fused.

Cross-ISA IPC can still be informative when paired with retired-work counts, clock frequencies, and code analysis. It cannot stand alone as a performance metric. Two binaries can complete the same task in the same amount of time while reporting very different IPC. For one may simply retire more instructions that are doing less work per instruction, then you have to factor in clock frequency which could be wildly different. IPC describes the behavior of the core running a piece of code, not a universal measure of useful work.

Looking at the branch predictor results, more branch predictions per cycle could indicate a capable predictor or it could also mean the Arm binary contains more branches, the benchmark moves through code faster, or NVIDIA’s event counts speculative predictions that the EPYC event does not. Higher backend operations per cycle may correlate with performance while telling us little about which feature caused it. To isolate value prediction, graph prefetching, or the neural predictor, we need on/off experiments or at least event definitions and miss rate deltas. While the IPC advantage of Vera over Turin may be real, the charts in NVIDIA’s whitepaper don’t provide enough granularity of the results to show it.

Vera’s Memory Advantage Is Real and Misattributed

The memory section contains NVIDIA’s strongest result and one of its weakest conclusions.

Against the dual-socket EPYC 9755 system in the paper, Vera reaches roughly 1.1 TB/s in NVIDIA’s loaded-latency plot while Turin levels off near 400 GB/s. Vera also shows 12.7 GB/s per core versus 3.1 GB/s per core for Turin. Those results do not line up with our testing of Turin CPUs.

In our testing of Turin, we were able to get approximately 570 GB/s out of Turin with the 12 channel DDR5-6400 memory subsystem. This is in direct contradiction to NVIDIA’s results which top out at ~400 GB/s of memory bandwidth. This does also throw the per-core memory bandwidth numbers into dispute with the per-core bandwidth increasing to ~4.5GB/s for the EPYC 9755.

Vera still comes out ahead, but our Turin result substantially changes the size of that advantage. Comparing Vera’s roughly 1.1 TB/s against the 570 GB/s we measured gives NVIDIA a 1.9× bandwidth lead rather than the nearly 3× lead shown in the whitepaper. Revising the EPYC 9755’s per-core result from 3.1 GB/s to approximately 4.5 GB/s similarly reduces Vera’s advantage from 4.1× to roughly 2.8×. And if we look at the SKU that AMD actually puts forward as the SKU for AI head nodes, the EPYC 9575F, then the per-core result becomes 12.7 GB/s vs the 9575F’s ~9 GB/s which is about 40% improvement for Vera. Those are still good numbers for Vera, but they tell a considerably less dramatic story.

Looking at the theoretical figures, AMD lists Turin’s limit at 614 GB/s from its 12 DDR5-6400 channels. Our 570 GB/s result reaches approximately 93% of that theoretical limit. Vera’s eight LPDDR5X-9600 interfaces provide 1.2 TB/s, while NVIDIA’s measured 1.1 TB/s reaches roughly 92% of that figure. In other words, both processors convert a remarkably similar percentage of their theoretical memory bandwidth into sustained bandwidth. Vera wins because it has approximately twice the peak bandwidth of one Turin socket and fewer cores competing for it, not because Turin is unusually poor at using its available memory bandwidth.

Despite that, the whitepaper repeatedly credits Vera’s monolithic compute die while contrasting it with “traditional chiplet-based CPUs.” A monolithic die may reduce fabric traversal and improve loaded latency, but our Turin result directly weakens that explanation for the bandwidth difference. A chiplet-based EPYC 9755 reaching approximately 93% of its theoretical limit is clearly not being held back by its chiplet topology in this case. The bulk of Vera’s bandwidth advantage comes from the memory interfaces attached to the processor.

The comparison also aged almost immediately. NVIDIA published its technical blog and whitepaper on July 21, while AMD launched 6th Gen EPYC two days later. The 96-core EPYC 9686F, which is much closer to Vera’s 88-core count, provides 16 memory channels supporting DDR5-8000 or MRDIMM-12800 for 1,024 or 1,638 GB/s per socket with the top MRDIMM speed giving Venice more theoretical bandwidth than Vera both in total memory bandwidth and per-core memory bandwidth depending on what SKU you look at.

What the new EPYC specifications and our Turin testing demonstrate is narrower than NVIDIA’s claim of “3× more memory bandwidth than the latest x86 CPU” depends on a Turin result that does not represent the bandwidth we could extract from the same processor generation. Against our result, Vera delivers approximately 1.9× the total bandwidth and 2.8× the bandwidth per core, that remains an impressive platform result but it is not evidence of a bandwidth advantage for monolithic Arm processors over chiplet-based x86 CPUs.

The Graph and RL Charts Need Data

NVIDIA reports a 2.6x PageRank advantage over EPYC 9755 and shows Vera scaling almost linearly to 32 cores while EPYC flattens to just a 10X performance increase at 32 cores. NVIDIA attributes this improvement over Turin as down to the monolithic compute die with their high-bandwidth Scalable Coherent Fabric, the 1.2 TB/s of memory bandwidth that Vera has, and the graph prefetcher inside the Olympus core.

However, while the paper links GAP Benchmark Suite, it omits what variables NVIDIA used which are important factors on how this test runs. The scaling plot stops at 32 cores even though the machines have 88 and 128 cores per socket. While the 2.6x result is interesting, without the variables that NVIDIA used, the result is likely irreproducible.

For the ClickHouse testing, NVIDIA links directly to Phoronix’s result where Vera led the tested processors across three passes over a 100-million-row dataset. The whitepaper’s 1.2x chart is still selective due to not using the 9575F results, but an outside tester produced the underlying result with a recognizable workload.

Then we reach Figure 24, “Vera drives 1.8x for RL training,” with the figure being a row of little completed-task squares. There is no model, environment, CPU/GPU allocation, framework, batch size, power measurement, repetition count, or error bar. We do not even know whether the squares represent samples, steps, or some random layout of tiles at NVIDIA HQ.

This is not a bad benchmark, it simply is not a benchmark at all.

The surrounding text explains why a faster CPU could improve reinforcement-learning rollouts with faster environment steps and reward computation can feed accelerators more quickly however Figure 24 does not even pretend to measure it.

A Good CPU Does Not Need a Bad Argument

After 45 pages, my position on Vera is more positive than my position on the Vera whitepaper.

Olympus looks like a serious core with a 10-wide fixed-length decoder, large private caches, value prediction, aggressive branch handling, graph-aware prefetching, and a monolithic 88-core die all being choices pointing to a very high-performance CPU core. The memory setup is no slouch either with the LPDDR5X subsystem delivering up to 1.2 TB/s of memory bandwidth that memory bandwidth-hungry server workloads will love, and early independent benchmarks say the silicon can cash at least some of the checks that the whitepaper writes.

However, the paper’s competitive argument is much shakier, with it mischaracterizing x86 SMT, turning an optional NUMA configuration into a default burden, relabelling standard CPU tests as agentic workloads, hiding a 3% two-socket rate lead behind 1.8x per-core bars, comparing undefined cross-ISA counters, attributing a memory-interface win to monolithic virtue, and presenting an illustration as performance data.

None of that makes Vera slow, it simply makes NVIDIA’s proof smaller than NVIDIA Marketing’s prose.

The next round of Vera testing should be straightforward. Give independent reviewers unrestricted production hardware so that we can publish frequency, package power, and wall power figures along with testing Spatial Multithreading on/off results and of course running whatever benchmark/workload we wish on Vera. If Vera is as good as its architecture suggests, those tests will be much more persuasive than drawing x86 SMT as a tiny two-lane traffic light. The way things stand, NVIDIA’s marketing risks tarnishing Vera. NVIDIA has built enough CPU here, it can stop borrowing performance from the marketing pipeline.

Here, I’ll be looking at the A55 primarily in the Mediatek Genio 1200, using the Radxa NIO 12L single board computer. The Mediatek Genio 1200 has four A55 cores and four A78 cores in what Arm calls the DynamIQ Shared Unit (DSU). The DSU is a cluster with 2 MB of L3 cache, and can host different core types without creating cluster boundaries between them. The Genio 1200 runs its A55 cores at 2 GHz, and gives each A55 core 128 KB of L2 cache. For memory, the SBC comes with 8 GB of LPDDR4X running at 4266 MT/s.

I’ve somehow wound up with several A55 implementations. My previous phone, a Pixel 3A, has a Snapdragon 670 with six A55 cores. Zarif98 on Reddit kindly donated a OnePlus 7 Pro, and that comes with four A55 cores in the Snapdragon 855. Microbenchmarking on Linux is easiest because Android tends to be a noisy environment, but I’ll occasionally include data from those devices to show whether an observation comes down to implementation differences.

Overview

Arm’s Cortex A55 retains A53’s 2-wide, in-order execution formula and makes targeted improvements. Like its predecessor, A55 uses an eight stage integer pipeline and a nine stage floating point one. All instructions pass through every pipeline stage for their category, which ensures ordering and avoids writeback conflicts from different latency instructions finishing at the same time. That makes the pipeline more fixed length than in a typical out-of-order core.

A55’s improvements are concentrated in the frontend and memory subsystem, though parts of the core proper received tweaks as well.

Branch Prediction

Arm’s slides talk about a new “neural network based” predictor. This likely means A55 uses a perceptron predictor, which can learn longer patterns while requiring less growth in predictor storage than the A53’s GShare predictor.

Testing with branches that are randomly taken/not-taken in random patterns of increasing lengths shows A55 doing better than its predecessor. However, it’s still far off contemporary high performance cores.

A55 and A53’s performance counters make a like-for-like comparison difficult because A53’s performance monitoring unit (PMU) can’t restrict counting to retired branches. A rough comparison with retired branches on A55 and executed branches on A53 suggests some level of improvement across Geekbench 6’s workloads, though it’s hard to tell how significant that difference is.

Curiously, A55 doesn’t universally improve and suffers more mispredictions in Clang. Photo Library is another case where A55’s newer predictor doesn’t show a clear benefit. Tuning predictors is difficult especially with bare bones power and area budgets, so some applications are bound to fall through the cracks. Perceptron predictors also struggle to predict linearly inseparable patterns. Schemes that index into a history table using the global history don’t have that problem.

Branch prediction speed can affect frontend throughput, and A55 improves that with a new 48 entry micro-BTB. This micro-BTB can handle taken branches without introducing pipeline bubbles, and should speed up performance in tight loops with taken branches. A53 had a 16 byte branch target instruction cache (BTIC) that could accelerate instruction delivery, but only for a single branch target. A 48 entry micro-BTB is still small, but should be relevant for more cases. Branch spacing doesn’t affect micro-BTB capacity.

A55 continues to lack a large BTB with hundreds or thousands of entries like on high performance cores. The core’s decoders calculate target addresses for taken branches not captured by the micro-BTB, giving taken branches 3 cycle latency like on A53. It’s not bad for code that fits in the instruction cache, especially considering that in-order cores are likely to lose lots of throughput from data-side memory latency. However, it does mean the branch predictor can’t run ahead of instruction fetch and drive long distance prefetch.

If each branch jumps to a different 64B cache line, taken branch latency heads for the skies once the test exceeds instruction cache capacity. A55 curiously regresses compared to A53, even though A55’s private L2 cache offers better data-side latency than A53’s larger shared one. With 1536 branches (96 KB), A55 averages 28-29 cycles per taken branch compared to 15-16 cycles on A53.

Instruction Fetch

Cortex A55 comes with 16 KB, 32 KB, and 64 KB options. Compared to A53, Arm dropped the 8 KB option and increased associativity from 2-way to 4-way. Increased associativity reduces the likelihood of conflict misses, where the cache may have enough capacity to contain frequently accessed lines, but ends up getting thrashed because of restrictions on where those lines can be stored.

A55’s instruction cache stores instructions in a pre-decoded format to reduce decode costs, rather than storing raw instruction bytes. 4 byte aarch32 or aarch64 instructions are expanded to a 40-bit intermediate format, while 16-bit T32 (Thumb) instructions take 20 bits. Instruction cache tags indicate the instruction type (aarch32, aarch64, or T32). The predecode approach is common to many of Arm’s cores, and takes a midpoint between a conventional instruction cache and a micro-op cache that completely skips decoding.

Instruction cache tags and data are parity protected, with one parity bit per 32-bit tag entry and one bit per 20 raw data bits. That means A55 can detect a single bit error within each predecoded T32 instruction, or up to two flipped bits in a 4B instruction provided the two bits don’t occur in the same 20-bit half.

Mediatek and Qualcomm chose the 32 KB option for their chips. Testing with a loop of NOPs gives no surprises. All A55 implementations can sustain two instructions per cycle from the instruction cache. Spilling out of L1I drops throughput to well below 1 IPC. Arm’s Technical Reference Manual states that the instruction cache has a “128-bit read interface to the L2 memory system”. However, A55 can’t seem to fully utilize that interface in a sustained fashion in this simple test. A55 also displays lower code throughput than A53 out of L2, mirroring the situation with branches. Past L2, A55 regains the lead because it benefits from a L3 cache.

Geekbench 6 shows A55 and A53’s frontends facing similar challenges. A55/A53 are both organized into an Instruction Fetch Unit (IFU) and Data Processing Unit (DPU). The DPU decodes and executes instructions, and contains an instruction queue. Arm treats cycles with an empty DPU instruction queue as frontend bound. I’m applying the same definition to A53, even though Arm didn’t use top-down metric definitions in A53’s time. Workloads with larger instruction footprints, like Clang, cause the DPU IQ to stay empty for a significant fraction of core cycles. Further breaking down the cause shows most of this happens when there’s an instruction cache miss pending. A55’s higher L1I associativity may help a little bit, but not having a large decoupled BTB continues to hurt.

Backend (DPU)

A55 inherits most of A53’s backend, meaning it features robust dual-issue capability. Most instructions can issue from either position, and the core has two copies of execution units for the vast majority of common instructions. Interlocks take care of instruction dependencies, meaning that adjacent dependent instructions don’t cause issue-stage hiccups. Forwarding paths help deliver low execution latency even though the pipeline is fixed length.

Earlier, I applied the same methodology I used to measure structure sizes on out-of-order CPUs to A53. Those tests showed A53, and A55, could do limited reordering around a cache miss. I assumed that was because the DPU had some kind of buffer for tracking in-flight operations in the backend. After reading the optimization guides for both cores, I think a better explanation is that independent instructions issued after a load can sit in pipeline stages while waiting for the load to retire. If so, observed reordering distance is simply the number of pipeline stages that younger instructions can occupy (or the first pipeline hazard that forces an issue-stage stall). There’s no pseudo-ROB.

A55’s execution pipe layout is basically unchanged. However, floating point fused multiply-add (FMA) instructions see latency reduced to 4 cycles, down from eight in A53. FP adds and multiplies have 4 cycle latency, so A53’s 8 cycle FMA latency meant there was no latency advantage to using FMA. A55 fixes that situation. Elsewhere, A55 receives changes to support the Armv8.2 standard, as well as LDAPR from Armv8.3 and dot product instructions from Armv8.4. Arm’s reluctance to pursue larger execution-side changes is understandable. Performance often comes down to feeding execution units rather than having more of them, and in-order cores face severe challenges in keeping their execution units fed.

Many of Geekbench 6’s workloads are heavy on FP/SIMD instructions. A55 cuts down floating-point/vector related interlocks in Asset Compression, HDR, Photo Filter, and Ray Tracer.

Structure in Motion doesn’t benefit, though that could be a case of A55’s improved memory subsystem shifting bottlenecks away from load latency and towards the execution units. Integer workloads like File Compression and Navigation see little change in interlock delays between the two cores.

At a higher level, A55 and A53 both have weak vector execution units. Even though there are two floating-point/vector pipelines, each pipeline only has 64-bit vector execution width. FP/vector operations also have longer latency than scalar integer ones. Out-of-order cores can easily absorb 4 cycle latency, but that’s not always the case for in-order cores.

Load/Store

Arm delivered more significant improvements to the memory subsystem, which starts with the load/store unit on the data side. A55 can now sustain both a load and a store every cycle, compared to A53’s single memory operation per cycle. Load and store bandwidth individually remain unchanged at 8B/cycle for loads and 16B/cycle for stores. However, the improved dual issue capability can be particularly beneficial for memory copy routines that don’t put loads and stores too far apart.

A55 improves L1D hit latency too thanks to an adjusted load pipeline and a forwarding path from the beginning of the load writeback stage to the AGU’s base operand input. This forwarding path lets A55 achieve two cycle latency for dependent loads. The load writeback stage can also forward to the ALUs, giving 2 cycle load-to-use latency for ALU operations as well.

Curiously, it seems like A55 can only use one of those fast forwarding paths at once. Placing an ALU instruction between two dependent loads increases pointer chasing latency to three cycles, but only if that ALU instruction consumes the prior load’s result. An independent ALU instruction doesn’t increase latency.

Indexed addressing increases load latency to three cycles, which is still one cycle faster than doing the same on A53. A53’s optimization guide interestingly mentions two cycle pointer latency as well, though I have not been able to achieve that even with unrolled back-to-back dependent loads. I’m not sure how A53’s pipeline layout would allow for two cycle load latency in any case. A load would need three clock transitions to get from the AGU to the writeback stage, and A53’s optimization guide says forwarding happens from the beginning of the writeback stage.

Store forwarding latency increases on A55 compared to A53, though cycle count latencies remain low compared to many out-of-order cores. Perhaps supporting dual issue made A55’s load/store unit more complex, and introduced extra penalties. To summarize the notable cases:

A55 continues to prefer 8B alignment for loads and 16B alignment for stores. Both cores avoid harsh penalties when a load only partially overlaps a prior store, likely thanks to their simple in-order designs and short pipeline depth. A55’s latency regression versus A53 probably has little effect because an in-order core can’t have hundreds of instructions in flight, making store forwarding cases less likely.

Cache misses are handled similarly on A53 and A55. Neither core supports hit-under miss, so any L1D miss will delay subsequent memory accesses even if they hit L1D. A55/A53 can achieve some degree of memory level parallelism by letting loads after a cache miss access L1D and generate fill requests. A55/A53 can generate up to three outstanding cache refill requests this way. Arm’s optimization guide says that only works if the three loads are within four pipeline stages of each other.

Address Translation

In typical scenarios, every memory access uses a virtual address that has to be translated on-the-fly to a physical address that corresponds to locations in DRAM or IO. A55 uses two translation lookaside buffer (TLB) levels like many other cores. TLBs cache frequently used translations, helping reduce address translation penalties. A55’s first level TLB has 16 entries, up from 10 in A53. Both figures are small compared to even older out-of-order cores. AMD’s Athlon 64 from long ago had 32 entry L1 TLBs, for reference. L2 TLB size is more reasonable at 1024 entries, matching Intel’s Haswell, and has twice the capacity of A53’s 512 entry L2 TLB.

Getting a translation out of the L2 TLB adds three cycles of latency, compared to just two on the A53. It’s a good tradeoff because the larger first level TLB should reduce reliance on the L2 TLB, and an in-order core will struggle to deal with page walk latency. 3 cycles of L2 TLB latency is good compared to many out-of-order cores, including low clocked ones. A78 for example takes 5 extra cycles of latency on a L2 TLB hit.

To reduce page walk cost, Arm uses both an Intermediate Physical Address (IPA) cache and a walk cache. The IPA cache holds translations between a VM’s physical address and host physical address, helping reduce virtualization overhead. The walk cache holds the second-to-last translation table entry, letting page walks initialize with one step to go. Arm didn’t change IPA cache or walk cache size on A55, likely because 64 entries already go a long way. One translation table holds 512 entries, so 64 walk cache entries would already accelerate page walks for up to 128 MB of address space using 4 KB pages.

L2 TLB structures are parity protected, with parity errors causing an invalidation and a page walk.

Cache Setup

A55’s L1D cache has 16 KB, 32 KB, and 64 KB options, and is 4-way set associative like the L1I cache. L1D data and tags are ECC protected, with single-bit errors handled by evicting the line, correcting the error as part of the eviction process, and reloading the line from L2. Compared to A53, increased L1D associativity should provide a slight hitrate improvement. Arm also changed L1D addressing from physically indexed, physically tagged (PIPT) in A53 to a virtually indexed, physically tagged (VIPT) scheme in A55. Note that the 32 KB and 64 KB options are still VIPT. A common misconception is that VIPT requires the virtual address bits that give the offset into a page to exactly match the cache index bits.

Cache misses are the most common long latency instruction in most workloads, and present a severe problem for in-order cores. A55 and A53 can keep no more than about eight instructions in flight after a cache miss, which is four cycles of core throughput. Any L1D miss will incur a longer penalty than four cycles, so reducing cache miss latency is crucial to making the situation less bad. A55 also needs to support Arm’s new DynamIQ Shared Unit (DSU), which has different cache setup expectations.

As a result, A55 gets an optional core-private L2 cache, which aligns with the DSU’s expectations and allows for lower L2 hit latency. A55’s L2 mostly acts as a victim cache, though software prefetch instructions can bring data into L2 without first getting that data evicted from L1. Arm provides 64 KB, 128 KB, and 256 KB L2 capacity options, all of which use 64B cache lines and are 4-way set associative. The L2 has a 128-bit write path and 64-bit read path to the core, matching load/store bandwidth. An extremely area constrained design can have no L2 cache, though doing so would likely be catastrophic for performance. 128 KB seems to be a popular choice, and is what Mediatek picked in the Genio 1200.

L2 latency comes in at a brisk 9 cycles, or just over half the 17 cycle latency seen with A53’s 256 KB shared L2 A small 32 KB L1D will see a lot of misses, and a 128 KB L2 could service a significant portion of those misses without the latency of a shared cache. The downside is that four 128 KB caches require twice the storage capacity of a single shared 256 KB cache.

With the new core-private L2, the DSU’s L3 takes over the role of a large, last level shared cache. Arm’s first DSU, the one introduced with A55 and A75, supports up to 4 MB of L3. L3 capacity options larger than 1 MB use two slices, meaning the Genio 1200’s 2 MB L3 is built from two slices. The power of two size also means it’s 16-way set associative. Arm allows 1.5 MB and 3 MB L3 options by disabling ways to create a 12-way cache.

L3 latency varies depending on implementation options, with Arm giving a 21 cycle latency example. On the Genio 1200, L3 latency comes in at around 35 cycles. It’s much higher latency than the 256 KB L2 on the Amlogic S922X’s A53 cluster, but that’s understandable considering the Genio 1200 has eight cores in the cluster instead of two. On the capacity front, A53 did support L2 capacities up to 2 MB, but I doubt that was a common configuration.

Bandwidth tends to be de-prioritized for density optimized designs, and Arm’s 5-series cores sit far on the density optimized side of the spectrum. Still, A55 manages significant bandwidth improvements at larger test sizes. A53’s bandwidth crashes into the ground once test sizes spill out of cache. A55’s results vary across implementations, but all display milder bandwidth dips even after test sizes exceed L3 capacity. Mediatek’s Genio 1200 barely shows any single thread bandwidth decrease even at very large test sizes.

Part of this could come from prefetching. A55/A53 rely heavily on prefetch to achieve memory level parallelism, because both cores can only sustain three L1D demand misses. Arm’s documentation indicates that A55’s L1D prefetcher can make up to seven outstanding prefetch requests, with a default of five. Curiously, A53 can be configured to make up to eight outstanding prefetches. New in A55 though is a L3 prefetcher that can be configured to prefetch 32 lines ahead, with eight lines as the default. Prefetch into L3 can be more aggressive, because the L3 cache’s larger size reduces the chance of useless prefetches contending with frequently used lines. This more aggressive prefetch likely explains A55’s higher memory bandwidth.

Both A55 and A53 use a streaming mode to help store bandwidth, with a switch at each cache level triggered by reaching a threshold of consecutive “streaming” cache lines. Mediatek’s A55 cores maintain very high store bandwidth even across large test sizes, mirroring behavior with loads. Other A55 implementations show similar behavior to A53.

Performance counters show A55’s memory subsystem delivering impressive improvements. Memory-related stalls were severe on A53, especially on Geekbench 6’s Object Remover, Background Blur, and Navigation workloads. A55 nearly cuts those figures in half.

Performance

The A55 cores on Mediatek’s Genio 1200 deliver performance improvements well above Arm’s reference figures across many of Geekbench 6’s workloads. Background Blur enjoys an astounding 210% score uplift, thanks to being less memory bound and suffering fewer FP/SIMD interlocks. Gains are also massive in Object Detection and Photo Filter.

I suspect uplifts will be closer to Arm’s estimates in a more like-for-like comparison. Mediatek’s Genio 1200 does give its A55 cores a much stronger memory subsystem than the Amlogic S922X does for its A53 cores. But better caching would be expected from most A55 implementations, thanks to the new DSU setup’s expectations.

A55’s improvements are less impressive in SPEC CPU2026’s integer suite, though they still go beyond typical generation-on-generation gains from high performance cores. A 35.35% geomean uplift over A53 is nothing to sneeze at, though of course A53 represents a very low performance baseline.

SPEC CPU2026’s floating point suite shows A55 pulling way ahead again, with a 50.45% geometric mean uplift over A53. I didn’t collect performance monitoring data for SPEC CPU2026 because each run takes such a long time on these low clocked, in-order cores. However, one thought is that SPEC CPU2026’s larger instruction footprints in its integer suite punish A55. Instruction cache misses tend to be less common in SPEC’s floating point workloads, which shifts emphasis back to A55’s superior memory subsystem.

Final Words

A55’s core architecture isn’t exciting, because it leaves most of A53’s framework unchanged. Arm made a sensible choice, because in-order cores do poorly at maintaining throughput around anything that takes more than a few cycles. Better branch prediction and caching down common causes of delay-causing events, and contribute directly to performance improvements.

Even with better branch prediction and caching, A55’s IPC figures show the difficulty of feeding a 2-wide pipeline with in-order execution. Certainly there’s more room to push on the same fronts that Arm did with A55. Larger and faster caches could further mitigate backend memory stalls. More sophisticated branch prediction would help mitigate instruction side stalls.

However, there’s likely a point where out-of-order execution starts to make sense even for sub-1W power targets. Larger caches take area and power. Branch prediction involves caching branch targets and outcomes, and runs into diminishing returns for increasing storage investment. Arm’s own A73 is a small 2-wide out-of-order design, and has no trouble pulling well ahead of A55 even without a clear memory subsystem advantage. For reference, A73 uses a pre-DSU system level setup, and has a 1 MB L2 cache in the Amlogic S922X.

Arm ultimately decided in-order execution still has legs. They followed up A55 with the A510, which implements 3-wide in-order execution and uses shared components in dual core clusters to save area. I can’t imagine feeding a 3-wide core with in-order execution is any easier. But I guess for now, out-of-order execution is still too complex to pull off at sub-1W targets, even with modern process nodes.

Today we are talking about one of the stranger x86 stories I have seen in a while. According to both SemiAccurate and Reuters, Intel is giving a new startup called RosaicLabs access to the register transfer language code for its Atom CPU technology. That is already a massive departure for a company that has historically guarded both its x86 license and CPU-core RTL very closely.

Two days before Reuters published its story, however, Phoronix spotted a very weird post on the Linux kernel mailing list. The post describes working x86 implementations with 16 and 32 Advanced Matrix Extension (AMX) tile registers, compared with eight tiles on Intel’s shipping AMX implementations. It was written by the same developer who said in October 2025 that a “corporate entity other than Intel/AMD” were already using its own x86 opcodes, CPUID range, and model-specific register range.

There are no public sources connecting Rosaic to that mystery x86 implementation. I want to be very clear about that from the start.

Still, an Atom-derived x86 core attached to a much larger matrix engine would make an awful lot of architectural sense for a startup staffed by Rivos veterans. Intel’s involvement would also answer the rather important question of how a new American company could legally build a custom x86 processor in the first place. So let’s dig into what we know, what the Linux posts actually say, and where the speculation begins.

Intel Lets Atom Out of the House

Both Reuters and SemiAccurate report that Intel has started providing Atom technology to RosaicLabs Inc., a startup incorporated in Delaware in May 2026. Rosaic is led by Amarjit Gill, a longtime semiconductor executive and investor who worked with current Intel CEO Lip-Bu Tan to assemble the founding team at Rivos. Reuters also says Rosaic’s team includes other Rivos veterans, including former Rivos finance chief Amit Parikh.

The reported arrangement goes well beyond an architectural license, with Intel reportedly supplying register transfer language, or RTL, code. RTL describes the logic and data movement inside a digital design in a form that can be simulated, verified, modified, and eventually synthesized into physical gates for a selected manufacturing process. In less technical terms, Intel is not merely giving Rosaic permission to execute x86 instructions. It is reportedly handing over the source-level description of an Atom CPU design.

That does not necessarily mean Rosaic gets every piece of an Intel processor. Reuters does not identify the Atom generation, the included cache and interconnect logic, the manufacturing process, or the contractual limits on modifying and shipping the design. Access to CPU-core RTL is also not the same as receiving a complete, tapeout-ready system-on-chip.

Even with those caveats, this is highly unusual. Intel has licensed x86 technology in limited circumstances before, but it has not operated like Arm, SiFive, or another merchant CPU-IP supplier. If this arrangement becomes a repeatable business model instead of a one-off deal, it would represent a fundamental change in how Intel monetizes its CPU designs. Intel has previously signaled that it wants to make more IP available through Intel Foundry, so this deal may be part of that broader strategy.

Rosaic itself remains almost completely in stealth. Reuters found no company website or LinkedIn page with any useful information, while an amended July filing reportedly gives the company room to raise a $10 million seed round. That is enough money to assemble a small early-stage team, but it is nowhere near enough to take a large, leading-edge processor from a blank sheet of paper to mass production.

Receiving an existing CPU core changes that equation by letting Rosaic spend its engineering budget on the parts that differentiate the product instead of rebuilding an x86 frontend, execution engine, memory-ordering machinery, privilege model, and software ecosystem from scratch.

The Other Mystery x86 Processor

The first useful clue appeared in October 2025. Longtime x86 researcher Christian Ludloff posted a message to the Linux kernel and GNU Binutils mailing lists saying he had been asked to relay information about opcode allocations already “in active use by a corporate entity other than Intel/AMD.”

The original Binutils post lists several reclaimed or otherwise unused opcode regions, an E000_xxxx CPUID range, and a matching E000_xxxx MSR range. Ludloff said these ranges were already in active use, implying that this was more than an academic proposal.

As Phoronix noted at the time, that raises two questions: Who was building the processor, and what legal right did it have to implement x86? Existing x86 companies such as VIA, Zhaoxin, or Hygon would be obvious candidates, but they already have public toolchain and kernel histories. Routing the notice through an independent expert while refusing to name the company would be an odd way for one of them to reserve instruction space.

Then on July 27, 2026 Ludloff posted again, this time the subject was an x86 implementation with more than the normal eight AMX tiles. The full mailing-list post documents implementations with 16 and 32 tile registers and explains how they can be fitted into mechanisms that already exist in the AMX architecture.

From Eight Tiles to 32

Intel’s current AMX programming model provides eight tile registers. Each register can hold up to 16 rows of 64 bytes, giving each tile 1 KB of capacity and the complete tile file 8 KB of architectural state. A separate Tile Matrix Multiply unit operates on those registers. Intel’s own AMX documentation describes this as palette 1.

Interestingly enough, the 64-byte TILECFG structure used to configure AMX already contains row and column fields for 16 tiles. Shipping implementations simply use the first eight. Ludloff says a 16-tile implementation can therefore use existing VEX instruction encodings, which can address TMM0 through TMM15, along with fields already present in the configuration structure.

Moving AMX to 32 tiles requires more work. The proposed AMX implementation uses EVEX encodings to address TMM0 through TMM31 and doubles TILECFG from 64 to 128 bytes so it can describe the extra registers. The post says existing CPUID mechanisms can already report more than eight tiles, the total tile-state size, and the width of TILECFG, so the implementation does not require a new feature flag.

In other words, the original AMX framework was more extensible than Intel’s shipping hardware. However, compatibility could become a problem because existing software often assumes that TILECFG is always 64 bytes and that the tile-data state is always 8 KB. Software with those assumptions would have to run in a degraded eight-tile mode. The post even proposes MSR controls to disable the additional registers when necessary.

For a 16-tile implementation, the tile file grows to as much as 16 KB per hardware thread. At 32 tiles, it reaches 32 KB, which is a large chunk of state for an operating system or hypervisor to save and restore, especially on a many-core processor. This points to the device containing the 16- or 32-tile AMX unit being much more focused on matrix multiplication than on general-purpose server tasks.

The mailing-list post refers to both AMX and its cross-vendor successor, AI Compute Extensions (ACE). AMD and Intel developed ACE through the x86 Ecosystem Advisory Group as a common matrix-acceleration target for future x86 CPUs.

The ACE whitepaper defines eight 512-bit-by-16-row tile registers plus a block-scale register. It also places ACE inside the existing AMX palette framework, allowing operating systems to reuse much of AMX’s state-management machinery. The mysterious design’s 16- and 32-tile modes therefore go beyond the baseline state defined by both AMX and ACE. They look like a vendor-specific scaling of the same programming model rather than merely a straightforward implementation of the new common standard.

Again, this points to a matrix-accelerator chip rather than a conventional general-purpose CPU.

Now the Real Question… Is This Rosaic?

Now we get to the fun part, because the circumstantial evidence is fairly compelling with both Reuters and SemiAccurate saying a previously unknown startup staffed by Rivos veterans is receiving Atom RTL from Intel. Days earlier, an unnamed corporate entity asked Ludloff to disclose details of a heavily extended x86 matrix implementation. Ludloff had previously relayed private x86 instruction, CPUID, and MSR allocations on behalf of an unknown corporation. An appropriately broad Intel agreement could explain how such a company gained both a functional x86 core and the legal right to ship it, although neither report discloses the actual licensing terms.

Atom also makes sense as a starting point. Intel’s Atom-derived cores are designed for better area and power efficiency than its large performance cores. Rosaic could use a relatively compact x86 core as the host and control processor, then spend most of its transistor and power budget on matrix execution, cache, memory bandwidth, and interconnect.

That would look less like a conventional Xeon competitor and more like an AI accelerator that happens to execute x86 natively. Instead of pairing a RISC-V control core with a giant data-parallel engine, as many accelerator startups do, Rosaic could pair an Atom-derived core with a giant AMX- or ACE-like engine. Ordinary x86 software would handle orchestration, while long-running kernels would use the expanded tile-register file.

However, there is a fly in the ointment. Rosaic was only incorporated in May 2026. The October 2025 opcode notice predates the company’s first filing by about seven months. That does not mean Rosaic could not have been working on a chip before those filings. However, in the July mailing-list post, Ludloff says that an implementation supporting 16 and 32 tiles “has been running at large scale for some time.” It is very unlikely that a startup receiving Atom RTL could have quietly designed, taped out, and deployed a new x86 core with an AMX implementation that is not directly compatible with current AMX code at “large scale” in that time.

There are ways to make the timeline fit, but each requires another assumption on top of the assumption that Rosaic is the company Ludloff is working with. The design could have started as an internal Intel project that was later handed off to Rosaic. “Running at large scale” could refer to simulation, emulation, or FPGA instances rather than production silicon. The unnamed corporation in 2025 could also have been a predecessor entity whose work is now being commercialized through Rosaic.

It is also entirely possible that the timing is a coincidence. The mailing-list posts never name Rosaic, and Reuters never mentions AMX, ACE, custom opcodes, or a matrix accelerator. Access to Atom RTL does not imply that Intel included AMX, because Intel’s shipping Atom-derived cores do not provide the per-core AMX engines found in its large Xeon cores. Rosaic would still need to design or obtain the accelerator, integrate it with the core, and make the enlarged architectural state work across the operating system and hypervisor.

Conclusion: Who Is Hiding Behind the Tiles?

The hard evidence leaves us with two separate stories. Intel is taking the extraordinary step of providing Atom RTL to Rosaic, while an unnamed company has an x86 AMX implementation with 16 and 32 tile registers that Ludloff says has been running at large scale. Connecting those stories produces a very compelling architecture, but it does not yet produce a confirmed product.

If “running at large scale” means deployed production silicon, Rosaic is difficult to reconcile with the timeline unless it inherited a mature design. If the phrase instead covers simulation, emulation, FPGA prototypes, or work performed by a predecessor organization, Rosaic becomes much more plausible. That single ambiguity is doing an awful lot of work.

Rosaic is not the only possible name. Zhaoxin is the obvious conventional candidate because it has a legal path to x86 through VIA and is actively building modern x86 processors, while Hygon has an AMD-derived x86 lineage, a growing data-center CPU business, and public ambitions around AI acceleration. A large tile file would fit either company’s server ambitions, but both already submit compiler and kernel support under their own names, making the anonymous relay difficult to explain; neither company’s public software support exposes anything resembling 16- or 32-tile AMX.

A hyperscaler’s internal chip program may fit the “large scale” wording better: Google, Meta, Microsoft, and Amazon have the money, deployment volume, and custom-silicon teams needed to build such a device. Google is tempting because Ludloff previously worked there, while Meta has the Rivos connection, but that is extremely thin circumstantial evidence and none has publicly disclosed the x86 license or Intel-core agreement such a design would appear to require. The fun wildcard is the old Centaur/VIA lineage, because Centaur already built an x86 server SoC with a large integrated NCORE AI coprocessor. Most of that team moved to Intel years ago, and there is no evidence NCORE evolved into this mysterious AMX implementation, but it shows that pairing a third-party x86 core with a giant matrix engine is not an entirely new idea.

My best guess is that the mystery design belongs to either an Intel-backed spinout or a confidential custom-silicon customer, with Rosaic remaining the most interesting named candidate. Zhaoxin and Hygon have clearer routes to implementing AMX, while a hyperscaler better fits the claim of large-scale operation. Every option solves one part of the puzzle and creates another.

For now, the most important detail is not whether the mystery company is Rosaic. It is that Intel appears willing to let outside companies build around its CPU RTL at the same time that someone outside Intel and AMD is publicly reserving substantial new x86 architectural space for matrix compute. Even if the two stories are unrelated, both point toward a much stranger and more open x86 ecosystem than the one we are used to.

Today we are over AMD's Advancing AI 2026 event to chit chat on their newly revealed CDNA5 architecture with Alan Smith! AMD's Corporate Fellow and Chief Architect for Datacenter GPUs.

Hope y'all enjoy!

The transcript below has been edited for conciseness, readability, and clarifications.

George: So, what do you do here at AMD?

Alan: Yeah. So, I’m a Corporate Fellow of Graphics Architecture at AMD and responsible for Instinct GPU architecture, which we launched today in MI455 and Helios.

George: Starting off, all the way down into the grunt of the engine of MI455. Prior CDNA architectures were based on the GCN architecture, which dates all the way back to Tahiti from I believe 2012, if memory serves, but with CDNA 5, you have now rebased to RDNA. What were some of the considerations for that rebasing of the architecture?

Alan: Yeah, that's a great question. I think there were many, many considerations for making that transition, right? First of all, you know, we wanted to move onto a modern architecture. There were some of the things that we were carrying from the previous GCN architecture, as you mentioned, that we wanted to enhance, including the cache system and also just the execution engine itself, right? So in terms of work scheduling, sequencing of the waves, and even instruction issue and things like this. So there were many things that we wanted to improve.

And so, we knew we wanted to make a big change in the architecture. We wanted to get significant gains in efficiency, and we felt like the opportunity was there for us to bring together what we had done for RDNA, what we wanted to do with CDNA, in order to bring the AMD GPU roadmap more closely together, which gives us then the opportunity for additional optimization in the future that can benefit both.

And one of the reasons we think that’s really important is because AI is everywhere, as you saw in the keynote today. And so, even from gaming and neural rendering and upscaling and all of these things that are now leaning into AI for GPU, even for rendering on the RDNA side, and the things that we’re doing with AI for data center, there’s an opportunity to really bring that together and leverage them on both product lines. So that’s why we wanted to do that.

George: Speaking of two different product lines, I believe last week we published an article on what is effectively MI430. What are the differences in terms of design for HPC versus AI, what design considerations do you have to make at sort of the WGP level for those two different product lines?

Alan: Yeah, great question. I mean, I think not just the WGP level, but the entire SOC level, how do we think about HPC workloads versus AI workloads? They have a lot of similarities, right? They both thrive on high-bandwidth memory, they both need cluster interconnects, they both really enjoy having a high-performance CPU connected to them with a CPU memory system, cache coherency, and all these things. So all of that is already common, right?

And the only thing that's different is the type of numerics that you need for the codes that you're running. And so, the way that we looked at this is we could have included double-precision floating point in the GPU like we've done in the past, or, you know, we could do even better for both if we took advantage of our chiplet architecture and optimized two versions of the compute chiplet: one for traditional simulation with high-precision formats like double-precision floating point, where we really need all of the full IEEE compliance with 64-bit; and then the same thing for the AI workloads, leveraging as highest throughput that we can achieve for both vector operations and the tensor operations for AI.

So having said that, even within the workgroup processor, still many things are in common, right? We need similar bandwidth out of the register file, out of the VGPR. So we're able to leverage all of that, and also all the scheduling, etc. So all of the support hardware within the workgroup processor from the LDS, the caching, the front-end sequencers, all those things, and the SIMDs themselves for the vector ALUs are all shared. And all we really need to do is modulate the number of double-precision units that we implement within the SIMD.

George: Speaking of the SIMDs, previously, a feature or one of the fundamental building blocks of how GCN worked and prior CDNA architectures was that there were four SIMD16 units that were being issued a Wave64 instruction with one instruction per four cycles. With CDNA 5, you have now moved to four SIMD32 units that are being issued one Wave32 per cycle and you have said that you have deprecated Wave64 support. Why did you choose to deprecate Wave64 support considering that RDNA with a similar SIMD setup has Wave64 capabilities?

Alan: Yeah, great question. I think, you know, the way we were looking at it is there’s some additional overhead that we need to carry for Wave64. There are some opportunities to leverage that in some special cases where you can get a little bit of additional efficiency out of issuing a Wave64 on SIMD32. But we felt like the tradeoffs of branch divergence and register pressure and so forth for continuing to keep that capability in most cases, or the common cases from our traces and workload profiling that we’ve done, we think that running in Wave32 almost all the time is a better solution.

And so we were able to, you know, maybe reduce the overhead that you need to carry for Wave64 and deploy that in order to get better common-case performance, and make up for the little bit of optimization that you can get from Wave64 on SIMD32.

George: Okay. So in RDNA 3 and 4, they have four SIMD32 units, but there’s also an extra bank of ALUs per SIMD. Does CDNA 5 have that extra bank of 32 ALUs basically, or is it just 32 ALUs per SIMD?

Alan: Yeah, I think in our SIMDs we have, it’s not always exactly 32 units per lane, right, because in some cases we have more throughput than can be achieved. So we have things like VOP2 instructions...

George: Where you need dual issue?

Alan: Yeah, where you’re effectively doubling the rate, if you will. So you need effectively multiple units per lane in order to get that higher throughput.

George: Speaking of higher throughputs and specifically the register file, with CDNA 5, you have massively increased the amount of VGPRs that a single wave can access: from 256 VGPRs to 1024 VGPRs. Why was that change made?

Alan: The reason we increased the addressing was just to give the waves less register pressure effectively because more of the waves are better able to share more VGPRs.

George: And in terms of the capacity of the register file, it is also 1024 VGPRs in terms of the physical number of registers, which is less than RDNA has. Why fewer in this bigger compute architecture?

Alan: I'll need to go check that.

In a follow-up email, Alan clarified that the VGPR register file size on CDNA5 is sized for the programs that CDNA5 will be used for which has different register file behaviors to what RDNA3/4 will usually run into.

George: On the diagrams it shows a WGP cache of 384 kilobytes and it splits it as 320 and 64 for the LDS and data cache/vector cache whereas some of your LLVM commits show that that number is fungible. So can you change the capacities of the LDS and the vector on the fly, or is that set?

Alan: Yeah. So the architecture was designed to support it to be programmable, but on MI455, we're going to have it as a fixed ratio. We find that for the MI455 deployments, having the larger LDS is almost always the balance that you need for the high tensor throughputs. And so we'll have it always fixed at 320 kilobytes and 64 kilobytes, as you mentioned, for the vector data cache.

George: So then, would MI430 potentially have a different split, or would it be, again, that sort of being able to change on the fly for different workloads?

Alan: It may have a different split.

George: Speaking of caches, that’s a big change specifically at the XCD level. Whereas for CDNA 3 and 4, we had sort of the L1 that was private per CU, then you had the L2 that was per XCD, and then you had the MALL AKA Infinity Cache that acted as sort of the glue to everything, that has changed. Why was that change made?

Alan: Yeah. So there were a couple of reasons that we were addressing with these changes. One of the things that we really wanted to do for MI450 series beyond what we had in 300 and 350 series was die-scope atomics or global atomics. So we wanted to move that back into the L2 cache, where we could get a higher throughput implementation, so that was one of our design goals.

We also wanted to eliminate the kernel boundary flush that we had on the L2 cache that was implemented in the XCD for global visibility on MI350 series. And then we also just wanted to increase the bandwidth for global L2 beyond what would be capable to go across the chip boundaries between the two cache fabric dies.

And so, the way that we did that is we effectively implemented a client-side cache instead of a memory-side cache. So we have these two client-side caches. We used the area that would have been implemented, or would have implemented an Infinity Cache on MI355 on the base die, built a large global L2 cache on each of them, and now all of the shader engines that are in the accelerator complex dies situated on one of those base dies, they all share that L2 cache.

So the last thing that you get the benefit of is from L2 cache data reuse: we get more sharing across more shader engines. So we have eight shader engines now that share a common L2 cache versus what we had on MI300 and 350 series.

And then what we do with those two caches: now each one of them is able to cache all of the memory. So all of the global addresses within the whole GPU, each of them can cache locally. And so those cache accesses don’t have to traverse across the chiplet boundary on the base die, versus on MI300 and 350, where in order to access the capacity of the MALL, the transactions would have to go across longer latency and higher power to go across all of the base die chiplets on the I/O dies.

George: So then, can a WGP on FCD0 access the cache in FCD1?

Alan: They do not.

George: Okay. So what is the bandwidth between the two FCDs?

Alan: We sized the interface between the two FCDs so that we can achieve full memory bandwidth in what we call NPS1. So if you look at the bisectional bandwidth of the two dies, it’s effectively sized relative to the total HBM bandwidth of the device, such that 50% of the bandwidth when you’re sharing from one side to the other... each side needs to access 50% of their accesses on their side and 50% on the other side.

And so that interface is sized so that both sides can do that simultaneously and achieve the full bandwidth of the memory system across the two, plus, as you alluded to, we have some coherency traffic because the Infinity Fabric is maintaining cache coherency between those two GL2 caches. So it’s sized appropriately to handle that traffic in addition to the memory traffic, and then also the I/O traffic that we have coming in from the scale-up and scale-out networks.

In a follow-up email, Alan clarified that the die to die bandwidth is 14 TB/s bi-directional.

George: A way to think of this is like sort of the way that EPYC does its cache coherency, I guess?

Alan: It is very similar. From their L3s. So if you think about the L3 cache implementation on EPYC and the way that that’s implemented, it’s very similar here.

George: Cool. And so we come to the ultimate question. While I usually ask “What is your favorite cheese?”, we already know your favorite cheese! So have you tried any new cheeses as of late?

Alan: (Laughs) Um, yeah, I’m still a fan of cheddar, right?

George: I believe your favorite’s Cabot?

Alan: Cabot is, yeah, what we talked about. But you know what? Since I know that you’re a Tillamook fan... I just want to say that I have been eating a lot of Tillamook ice cream lately!

George: Oh yes! Anyway, thank you so much for sitting down with me.

Alan: Absolutely, George.

Hello you fine Internet folks, here at AMD’s Advancing AI event we are looking at AMD’s brand new Instinct MI455X, replacing the older Instinct MI355X at the top of their AI stack. It is AMD’s first GPU designed for rack-scale AI deployments and is based on the new CDNA5 architecture with major changes to the compute unit and SoC, including enhancements to compute performance, improved memory bandwidth, larger memory capacity, and packaged using TSMC’s CoWoS-L.

It comes along with the new Helios rackscale solution, enabling scaling to 72 GPUs in a single pod, up from 8 GPUs for previous MI355X systems. It also features a new fault-tolerant scale-up networking architecture based on UALink over Ethernet (UALoE) with single-hop all-to-all communication across the entire rack.

A single MI455X contains 256 Work Group Processors (WGPs) across 8 Accelerator Complex Dies (XCDs), with a max “engine” clock of 2.4GHz. This enables peak compute figures ranging from 315 TFLOP for matrix/vector FP32 and vector FP16, and up to 40.26 PFLOP for OCP MXFP4. This is paired with 12 stacks of HBM4 each on a 2048 bit bus for a total of 192 channels, giving each GPU a total of 432 GB of memory at 23.3 TB/sec.

CU/WGP changes

Starting with the changes in the “Compute Unit”, AMD now counts the WGPs instead of the CUs for CDNA5. Unlike what they have done on their consumer counterparts where each WGP counts as two CUs.

But just like RDNA4, each WGP comprises four dual issue Wave32 SIMD32 units alongside 4 scalar units which is a massive change from CDNA4’s four single issue Wave64 SIMD16 units.

This means that each WGP can do up to 256 packed FP32 operations per cycle (512 FLOPS if using FMA).

To support this new SIMD design, the VGPR register file has been reorganized with any wave now able to address up to 1,024 VGPRs which is four times the number of VGPRs that a wave could access in prior CDNA and RDNA architectures. Each SIMD still has 128kB of vector registers just like CDNA4 which means it has twice as many (1024) registers available in practice due to Wave32 vs Wave64, but it is still less than the 192kB available on RDNA4. It also means that a single wavefront is in some cases expected to occupy the entire SIMD.

The matrix units also have been beefed up with each matrix unit being able to do up to 8,192 FP4 operations per cycle and with 4 matrix units per WGP you can do up to 65,536 matrix operations per cycle per WGP.

This means that only FP4 and FP8 are practically faster in the new architecture, and the rest of the performance comes from increasing the SIMD width from 16 to 32.

The inter-WGP caches have also changed to support this new design. Both the L1 Data Cache and the LDS have doubled in size to 64 KB of L1 Data Cache and 320 KB. The bandwidth of these caches has also doubled and can now do 2 256 bytes per clock.

Each XCD has 32 WGPs active, 34 physical WGPs of which 2 are fused off, which are broken up into 2 Shader Engines (SE) per XCD each with 16 WGPs. A shader engine is paired with a “Broadcast Arbitrator” replacing the L1 buffer (GL1) from RDNA4, which is now at the SE-level instead of the lower shader array-level. It works both as a write-combine buffer along with providing “up to 4x” bandwidth amplification by presumably broadcasting data.

The memory subsystem now allows multicast loads, accelerating matrix multiplication significantly by reducing redundant memory traffic. Imagine a GEMM that calculates C=A×B, then normally we would have each wavefront individually load a tile of, for example the A operand into the LDS of each WGP. With multicast loads we can instead load this tile of A into all relevant WGPs with a single multicast load instruction.

Each wavefront still loads a different B tile, but MI455X can fetch the common A data from L2 once and use the Broadcast Arbitrator to replicate it into all relevant WGPs private LDS allocations.

AMD calls this up to 4x bandwidth amplification, though the amplification happens after L2 and it does not quadruple L2 or HBM bandwidth. Instead, one unit of L2 traffic becomes four units of locally delivered data, reducing redundant cache reads and chiplet-link traffic while leaving each WGP with a nearby copy of the data.

SoC/Cache Changes

Moving down to the base dies, MI455X has 192 MB of Global L2 cache split across two Fabric Cache Dies (FCDs). Each FCD contains 96 1 MB SRAM blocks, giving it 96 MB of L2 capacity for the 128 WGPs attached to that cache domain. Each FCD can deliver up to 27 TB/s of L2 bandwidth, for an aggregate of 54 TB/s across the package.

This arrangement also changes how the cache domains behave. On CDNA 3 and CDNA 4, CUs attached to one base die could access the cache on another base die within the same package. CDNA 5 removes that capability: a WGP attached to one FCD cannot access the L2 cache on the other FCD. In that respect, MI455X behaves more like an AMD EPYC CPU, where a core on one CCD cannot directly access the cache attached to another CCD.

AMD made this change to improve the behavior of atomic operations across XCDs, eliminate the kernel flush boundary that MI300-series XCDs required for global visibility, and increase L2 data reuse. Although the two L2 domains are isolated from each other, each 96 MB cache can hold data associated with any global address in the GPU. That allows a WGP’s local L2 domain to cache data that might otherwise require traffic across the die-to-die interface.

Speaking of that interface, the two FCDs are connected by a die-to-die interconnect providing approximately 14 TB/s of bidirectional bandwidth. The FCDs are also connected to 12 stacks of HBM4, with each stack providing 36 GB of capacity over a 2,048-bit interface. At approximately 7.6 GT/s per pin, those stacks provide a combined 432 GB of HBM4 and 23.3 TB/s of memory bandwidth.

For any I/O that is external to a MI455X package, each FCD is attached to a IO die which does the very important job of connecting the GPU package to the host CPU, the scale-up network, and to the scale-out NICs. The CPU to GPU link that was previously serviced by PCIe has been replaced by a dedicated 16 lane AMD Infinity Fabric providing 256GB/s of bi-directional bandwidth which allows a coherent link between the GPU package and the host CPU.

Networking and AMD Helios Rackscale

Speaking of scaling up, along with the MI455X accelerator AMD is launching a validated rackscale platform for large scale AI infrastructure deployment. This is enabled by an increase in scale-up interfaces with MI455X having 36 x 400Gbit/s UALoE interfaces that implement 2 x 200G ethernet lanes each, comprising 3.6 TB/s of peak bidirectional bandwidth per GPU.

MI455X also introduces a split DMA architecture, which automatically associates traffic with the optimal link that reduces the topology awareness required for communication.

Helios uses OCP’s new Open Rack Wide form factor, compromising a cabinet 1.2m wide and 1.3m deep that provides. GPU’s are arranged in two groups of nine Compute Trays, each 1 OU in height. Each tray has four MI455X’s and one 96 core EPYC 9006 SP7. Each CPU is paired with 16 x 64GB DIMMS (for a total of 1TB of memory) and five E1.S slots for SSD’s. Six Helios Switch Trays provide a total of 12 switches connected directly via 3 UALoE links per GPU. Each switch provides 432 links at 200Gb/s, for an aggregate of 21.6TB/s per switch. This amounts to a scale-up bandwidth of 260TB/s bidirectionally.

Scale-out is provided via two boards that can contain either 4 or 6 Pensando NICs, depending on how much scale-out the end customer would like, providing up to 43TB/s of backend bandwidth. With 72 GPUs in one rack, a Helios deployment is able to provide up to 2.9 EF at MXFP4 or 22.6 PF at FP32, alongside 31TB of shared HBM4 at a combined 1.7PB/s of memory bandwidth.

Conclusion: The King is Dead, Long Live the King

The basic GCN microarchitecture underpinned every single one of AMD’s compute accelerators for nearly 15 years starting with Tahiti, followed by Fiji, Vega, and the first 4 generations of CDNA. With CDNA5 AMD has moved over to a microarchitecture that is based on the RDNA series putting a bookend to the long-lived line that was the GCN microarchitecture.

And with that bookend comes the start of a new story for AMD’s Datacenter Accelerators, one that now isn’t just about a single GPU but scaling to 72 GPUs in a rack along with scaling the number of racks. For that AMD is relying on nearly every part of their business from EPYC Server CPUs, to Pensando Networking, to the base GFX12 from the Radeon Division, to combine all of them into the AMD Helios Rack.

So if you want to know what GFX1250 is changing compared to the prior generation of CDNA along with RDNA4, I would recommend reading that article first then coming back to this article.

The Shared Foundation

Starting with what is shared between both GFX1250 and GFX1251, they both inherit LLVM’s common GFX12.5 target which includes Wave32-only execution, 64-bit literals, a 32-bank LDS, packed FP32 operations, FP8 conversion support, and 1,024 addressable VGPR. Both targets also have 320 KiB of addressable LDS along with the same GFX12.5 SWMMAC instruction sets and the same WMMA and transcendental co-execution hazards.

The scheduling model is similarly close as well, with the notable exceptions being 64-bit math and WMMA AKA matrix math. Outside of those two cases, LLVM shows both GFX1250 and GFX1251 as having the same latency for 32-bit, LDS, and scalar memory operations along with the same modeled branch and VMEM costs among other similarities.

Getting Serious About FP64

The biggest difference between GFX1250 and GFX1251 is the 64-bit paths on GFX1251 getting a significant improvement. LLVM gives GFX1251 the same ‘FullRate64Ops’ feature as prior CDNA-based accelerators which means that the compiler expects the hardware to have full-rate operations for most FP64 instructions.

Packing FP64 in like Sardines

Moving to packed arithmetic, GFX1251 adds v_pk_add_f64, v_pk_mul_f64, v_pk_fma_f64, v_pk_min_num_f64, and v_pk_max_num_f64 where each instruction operates on a v2f64 or two 64-bit values per active SIMD lane. Because a VGPR contains 32 bits per lane, every packed FP64 operand occupies four VGPRs meaning that the three sources consume 12 VGPRs and the result consumes four.

v_pk_fma_f64 v[4:7], v[8:11], v[12:15], v[16:19]

Across a Wave32, one instruction represents 64 FP64 FMAs operations which equates to 128 FP64 FLOPS.

The packed integer side follows the same 128-bit register layout. GFX1251 adds v_pk_add_nc_u64, v_pk_sub_nc_u64, and v_pk_lshl_add_u64, each operating on two unsigned 64-bit values per lane. Those instructions are useful for paired address calculations and counters, though LLVM does not expose a packed U64 instruction set as broad as the FP64 one.

Importantly, LLVM’s optimizer does know these instructions exist because the GFX1251 cost model advertises a 128-bit fixed vector width when packed FP64 or U64 operations are available, permits a vectorization factor of two for 64-bit elements, and cuts the estimated cost of work in half. In other words, this is not just an ISA feature for hand-written assembly, LLVM is being taught to combine pairs of 64-bit operations so that regular compiled code can use it.

FP64 Matrix Math

The v_wmma_f64_16x16x4_f64 instruction for FP64 Matrix Math is a GFX1251-only instruction appropriately enough in the LLVM feature set labeled GFX1251GEMMInsts. Notably, despite the feature set implying multiple instructions in this set, as of the time of writing the only instruction in this feature set is v_wmma_f64_16x16x4_f64.

v_wmma_f64_16x16x4_f64 v[8:23], v[0:3], v[4:7], v[8:23]

Per wave, the two matrix inputs consume four VGPRs each, while the accumulator and result occupy 16 VGPRs. Across a wave, the instruction updates a 16x16 FP64 output tile over a K dimension of four. That works out to 1,024 FP64 FMA operations or 2,048 FLOPS per wave with LLVM scheduling it as a four-pass WMMA instruction with a 16-cycle latency.

Trading Off Low-Precision WMMA Math

The tradeoff for all of this FP64 compute is a reduction of low precision matrix math. LLVM enables GFX125xLowestRateWMMA for GFX1251 and places many instructions into latency classes that are two to four times as long as GFX1250’s.

The slower instruction rate also makes GFX1251’s dependency hazards longer. LLVM groups GFX12.5 WMMA instructions into rate categories and counts the number of unrelated VALU instructions or `v_nop` slots required between dependent operations. The slowest GFX1251 category can require up to 17 intervening slots before a dependent WMMA operation and 16 before a dependent VALU operation, whereas the faster categories require only 9 slots before a dependent WMMA operation and 8 slots before a dependent VALU operation.

Conclusion

GFX1250 and GFX1251 share enough features that calling them members of the same GFX12.5 family makes sense. They use the same wave size, expose the same architectural VGPR range and LDS capacity, and look almost identical to LLVM for ordinary 32-bit and memory operations which points to GFX1250 and GFX1251 using the same base architecture.

However, it is very clear that AMD has decided to specialize the two GFX12.5 architectures far more than even the MI300 family. The MI300 series had the MI300A and MI300X which while specializing the SoCs, with MI300A having 3 Zen 4 CCDs plus 6 CDNA3 XCDs onboard compared to MI300X having 8 CDNA3 XCDs, reused the same XCDs between them. The MI430X and MI455X are very clearly not sharing the same XCDs but have specialized the XCDs for either HPC or ML workloads.

GFX1250 is the better target for low-precision matrix math. Depending on the instruction, GFX1251 takes two to four times as many modeled cycles and can require much longer spacing before dependent operations. GFX1251 instead gets full-rate FP64, packed FP64 and U64 arithmetic, 64-bit DPP, and support for FP64 matrix math.

That makes GFX1251 much more interesting in the broader HPC space. A large portion of traditional HPC still depends on FP64 for numerical stability and reproducibility, including computational fluid dynamics, weather and climate modeling, molecular dynamics, finite-element analysis, seismic processing, quantum chemistry, and other scientific workloads. These applications do not only need a high peak FP64 number, they also need FP64 arithmetic, conversions, reductions, address calculations, and dense linear algebra to work well together. GFX1251’s full-rate FP64 path, packed 64-bit instructions, 64-bit DPP support, and FP64 WMMA instruction all point to a HPC focus.

FP64 matrix operations are especially relevant because DGEMM sits underneath a large amount of scientific computing, directly through BLAS and LAPACK or indirectly through larger simulation and solver stacks. A 16x16x4 FP64 tile does not by itself establish application performance, but it gives GFX1251 a dedicated way to accelerate the double-precision matrix math used by workloads such as HPL and dense numerical solvers. Meanwhile, packed FP64 and 64-bit DPP should help with the less tidy vector math and reductions which make up a good portion of HPC compute. That makes the overall FP64 story both broader and stronger than adding one impressive matrix opcode.

In the end what we are seeing with the GFX12.5 family is a story of different architectures for different markets. While GFX1250 is very much focused on the insanely lucrative ML market, GFX1251 is focused on the still very important HPC market where high precision compute still reigns supreme.

LLVM currently has preliminary support for two new accelerators from AMD, under the codenames GFX1250 and GFX1251.

GFX1250 is an accelerator chip oriented at the machine learning market called MI455X and will be what powers the Helios rack. While the GFX1251 is oriented at the HPC market, named the MI430X and is projected by AMD to deliver over 200 TFLOPs of native double-precision compute.

We’ll focus on the GFX1250 for now though and save the GFX1251 for another day.

We’ll compare the architecture to the consumer variant, the previous generation of CDNA accelerators, and to some extent Nvidia’s Blackwell.

A curious note here is that while we know that these accelerators are not APUs, they are listed as such in LLVM.

Similarities to RDNA4

The biggest similarity with the consumer architectures is that we now have a WGP (Workgroup Processor) that are built up from two pairs of two SIMD32 arrays, but there is also four SIMDs per CU just like there was in GCN and CDNA. This seeming discrepancy is because there is no longer any need for the software to separate the two CUs in the WGP and AMD seems to use the terms interchangeably.

This “change” is probably not as structural as it might at first seem and might not even exist in hardware, the changes in the caching hierarchy simply makes the distinction pointless for the compiler as there is now a shared vector L0 cache for the whole WGP.

GFX1250 operates only in Wave32 mode, unlike RDNA that can run in Wave64 in addition to Wave32 mode. Prior CDNA GPUs can only run in Wave64 mode and we can expect to see some oddness there from a performance engineering point of view when porting things to the new accelerators and a lot of kernels will have to be re-evaluated for the new architecture.

It also seems like GFX1250 can run 20 waves per SIMD, which is four more than RDNA4 can do.

We do not know if any of these characteristics are a sign for where AMD goes on their GPUs, but I personally guess that it is a sign of times to come and a future generation of AMD GPUs will be locked to Wave32 and the “WGP” as it currently exist goes the way of the dodo but it seems like this is not happening anytime soon.

Similarly to RDNA4 we now get “WMMA” support, unlike the older “MFMA” featured in CDNA and this is where most of the machine learning capabilities of the GFX1250 will come from and we’ll look into it more later.

The dynamic VGPR allocation feature that was heavily touted in RDNA4 is missing, which is a bit strange to me since I would imagine it is even more important for machine learning workloads that are under much higher register pressure due to working with whole tensors instead of just individual vectors or matrices. The mode-switch is still there but the instruction to allocate new registers is documented as a no-op.

A Change in (C)DNA

One of the largest changes found in GFX125X is that each wave in a SIMD can now address up to 1024 Vector General Purpose Registers (VGPRs) from the Register File. This is a major improvement over prior CDNA generations where you had a split allocation of registers with up to 256 VGPRs and 256 Accumulation VGPRs, which can only be used by the matrix units, for a total of up to 512 registers allocated to a single wave at a time.

This is an even larger improvement over the register allocation behavior of the RDNA series of GPUs, RDNA can only allocate up to 256 registers to a single Wave with no allocation split. This is a major update that might at some point be part of a future generation GPU IP but I wouldn’t bet on it coming soon. We will just have to wait to see if the feature is going to stay locked to the Instinct line or if this ability to allocate up to 1024 registers to a single wave will make an appearance in the Radeon line up as well.

The maximum addressable local memory (LDS) is now 320kB per wavefront which is twice as much as CDNA4 and dwarfs the 64kB addressable in RDNA. But the actual LDS can actually be even larger, a major change in GFX125X is the merging of the LDS and vector L0 caches into what AMD is calling the WGP Cache (WGP$). While both Nvidia and Intel have had a single structure for both cache and shared memory for a long time, AMD has been the odd vendor out with a split cache and LDS. GFX1250 now has a single 448KB structure that can be broken up into different allocations depending on what the compiler or programmer requests.

GFX1250 still supports packed fp32 operations allowing a single lane in the SIMD to process two fp32 operations at once just like CDNA2 and later, this practically means that when using packed instructions it has twice the vector width of RDNA.

It also inherits the more numerous and powerful SDMA units from CDNA.

Graphics Support

It is notable that Nvidia and Intel do have some vestigial graphics hardware left on their accelerators and GFX125x differs a little from their offerings on this front.

• No export instructions and therefore no rasterizer (like CDNA)

• No image (texture) instructions (like CDNA)

• No BVH (raytracing) instructions (like CDNA)

• No vector parameter interpolation instructions that are used to access and interpolate vertex data in pixel shaders (like CDNA)

• No MTBUF (typed buffer) instructions are supported (unlike CDNA and RDNA3)

• No MUBUF (untyped buffer) instructions are supported (unlike CDNA and RDNA)

• No LDS parameter and parameter direct loads (like CDNA)

We’re seeing that this GFX125x is even more of a pure compute accelerator than even prior CDNA architectures with nearly all graphics features having been removed. This probably saves a little bit of die space and removes features no one will miss, so it makes a lot of sense to me.

Tensor Operations

The headline feature of the MI455X is its AI capabilities, and the tensor units are what powers that. This seems to be a best of both worlds situation where AMD is combining the simpler programming model from RDNA4 with the higher performance and capabilities of CDNA4.

On RDNA4 WMMA operations are M = N = K = 16 (matrices are 16x16) except for the 4-bit ones that have K = 32 (arguments are 16x32 or 32x16 and accumulated into 16x16).

GFX1250 keeps M = N = 16, but K instead is the same as it was on CDNA4. K = 4 for 64-bit and 32-bit floating-point elements, K = 32 for floating-point 16-bit elements, K = 64 for fixed-point 8-bit elements and K = 128 for floating-point 8-bit elements. Interestingly the i4 operations stay at RDNA4 level at K=32 and are no longer a very popular data format in large AI inference scenarios due to the adoption of MX floating point operations.

Unlike on RDNA4 the sparse versions of WMMA seem to do less work per instruction than the dense ones by keeping K the same.

RDNA4 supports the following matrix of formats for WMMA

While CDNA4 supports the following for MFMA

And GFX1250 seems to support all the datatypes that are found on both CDNA4 and RDNA4 except for fp64 which seems to be restricted to GFX1251, including the support of OCP MX-style scaling that was previously available solely on CDNA4.

In addition there’s new instructions that allow you to transpose submatrices in LDS to complement the ones already in RNDA4. The fp4/fp6 versions of these have their own feature flag, that might be a sign that these will show up on consumer GPUs or it is just a coincidence.

It’s HIP to be … CUDA compatible?

Clusters

This seems to be AMD’s version of “Thread Block Clusters” that Nvidia introduced in their Hopper architecture, and it seems like they allow you to schedule a cluster to a specific shader engine and seems to be a feature that will be available on future graphics hardware too.

In addition to allowing you to access the cluster information, AMD has also introduced cluster-level loads which allow you to write to the LDS of other workgroups within a cluster, similar or equivalent to multicast loads on Nvidia hardware.

Author’s Note: A claim that wasn’t substantiated about DSMEM was removed here.

Barriers

GFX1250 adds support for cluster-level barriers, which is a feature that is essentially required to make use of clusters and is equivalent to the Nvidia feature of the same name.

In addition we’re getting intrinsics for barrier arrival, the equivalent of mbarrier.arrive in Nvidia PTX. AMD does seem to take this support one step further though by not just implementing barriers in hardware but also full monitors, allowing the wave to be notified if a specific cache line is modified or evicted from the L2 cache.

It’s unclear what these exactly are intended to be useful for since Nvidia doesn’t have a direct equivalent already, but I guess GPU programmers will find a lot of good uses for these intrinsics.

In addition to the split barriers and barrier objects already added in RDNA4, it also supports 16 named barrier objects. These are likely added as a compatibility thing, since they have been around since Nvidia’s Tesla architecture from 2006.

Transcendentals

AMD is adding hardware support for tanh, which is rare in traditional geometry but very common in neural networks as an activation function and in the solution of many differential equations.

But the whole unit seems to have been improved, they added bf16 support and the latency for fp32 operations have gone down from 9 cycles in RDNA4 to 8 cycles in GFX1250.

Cooperative Atomics

With GFX1250 we now have intrinsics to do 32x4B, 16x8B, and 8x16B atomic loads and store to global memory, the funky naming is because these are actually 128B atomic loads and stores, the first number implies how many lanes of each wavefront are involved in the atomic but the result is essentially 1, 2, or 4 complete 128B atomic memory operations.

At the hardware-level they aren’t special instructions but just a sequence of regular loads or stores that the hardware knows how to handle so that they end up on the bus correctly.

The first use-case for this that will pop into certain peoples minds is to use this to accelerate RPCS3’s Cell emulation, and while someone is obviously going to try and use it for that, there’s a bunch of use-cases for this. But a goal is likely to accelerate AMD’s equivalent to Nvidia’s Collective Communications Library (NCCL), creatively named ROCm Collective Communications Library, as explored in the paper Demystifying NCCL.

LL128 support was to some extent already enabled on CDNA but the format has now been changed to be equivalent to the Nvidia one instead of splitting into two 64B halves.

For technical reasons these operations are not available over PCIe but I assume these will be usable over CXL and UALink. Nvidia has a very similar limitation that it is only available in global memory or over NVLink.

Tensor Data Mover

Nvidia’s marketing name for this feature is the “Tensor Memory Accelerator”, and while this feature was already available to some extent in earlier CDNA iterations it has now gained more features especially related to tensors.

The basic version just allows you to load data from global memory into LDS without using the VGPRS.

While it isn’t completely clear to me exactly what capabilities are available to the tensor version since the types involved are opaque, the code isn’t totally obvious and the documentation isn’t completely clear. But Triton has some good documentation where the capabilities are enumerated. It is somewhat similar to Nvidia’s tcgen05.cp in capability except they transfer between global memory and LDS instead of between shared memory and tensor memory.

Explicit Prefetching of Vector Memory

GFX1250 adds explicit prefetching of vector memory, and AMD has finally added this feature that is available on many CPUs that are old enough to vote. Nvidia has supported this since Fermi so it really isn’t that exciting.

Some people might think that this has been a curious omission for GPUs, but unlike CPUs we’re rarely starved for registers so we can load data early and we can use the many waves running on the same core to hide the latency.

But I can imagine that inference kernels can gain some minor performance from this so it makes sense to add now I guess.

Data Dependencies

On AMD GPUs and accelerators the hardware can most of the time figure out data dependencies automatically, but in certain cases you need to do it manually.

On GCN and by extension CDNA you had three counters that allowed you to wait on operations to be completed so that you could ensure that previous operations had been completed before continuing.

• VM_CNT for vector memory operations.

  • Incremented when a vector-memory load is issued and decremented when the data is written to the VGPRs.

  • Incremented when a vector memory store is issued and decremented when the data is written to L2 cache.

  • Incremented when a sample or image instruction is issued and decremented when the data is returned to VGPRs.

• LGKM_CNT for LDS, GDS, Konstant (scalar memory read), and Message instructions.

  • Incremented when a LDS or GDS load is issued and decremented when the data is written to the VGPRs.

  • Incremented when a LDS or GDS store is issued and decremented when the data is written to the LDS or GDS.

  • Incremented for every DWORD worth of scalar load issued and decremented again for every DWORD read from the scalar data cache (SMEM)

  • Incremented for every message send issued and decremented when the message is actually sent.

• EXP_CNT for VGPR exports, this is a graphics feature used for passing things like vertex positions and color values between shaders. This exists in CDNA but isn’t used so we’re going to conveniently ignore this counter

The big issue here with the GCN/CDNA setup of these counters is that instructions of different types are returned out-of-order. And in the case of scalar loads they are also returned out-of-order even within the same instruction type.

This regretfully means that you are likely to need to insert more waits more than would otherwise be required, especially for scalar reads where waiting for the counter to reach zero is the only useful value.

So AMD redesigned this part of the instruction set for RDNA4 and made it much more fine-grained. We now have a lot more of these counters.

• LOADcnt counts the number of vector loads issued but not yet completed, similar to the first part of VM_CNT.

• STOREcnt counts the number of vector stores issued but not yet completed, similar to the second part of VM_CNT.

• SAMPLEcnt counts vector sample/gather/msaa/lod instructions issued but not yet completed, similar to the third part of VM_CNT.

• DScnt counts the number of LDS instructions issued but not yet completed, equivalent to the first half of LGKM_CNT.

• KMcnt counts the number of scalar loads and message sends issued but not yet completed, similar to the second half of LGKM_CNT.

• EXPcnt counts the number of export or parameter loads outstanding, similar to EXP_CNT.

• BVHcnt counts the number of vector memory BVH (raytracing) instructions issued but haven’t returned data yet. There’s obviously no GCN or CDNA equivalent since there’s raytracing hardware.

This means that RDNA4 has a lot more situations where it can avoid having to explicitly stall the core even though there is useful work to perform compared to GCN or CDNA prior to GFX1250.

And while this isn’t as big of a deal in compute tasks, in modern graphics tasks it does make a difference since for example you can issue texture reads far ahead and not have that block a regular memory load.

On GFX125x we don’t have the following graphics wait counts,

• SAMPLEcnt doesn’t exist because there are no texture samplers.

• EXPcnt doesn’t exist because there is no rasterizer.

• BVHcnt doesn’t exist because there are no raytracing instructions.

But to compensate we get new ones that do not exist in RDNA4 at all,

• ASYNCcnt, counts the number of outstanding asynchronous load and store instructions to LDS.

• TENSORcnt, counts the number of outstanding tensor load and store instructions.

• Xcnt, counts the number of memory operations that have not completed address translation yet and a wait here is required to be inserted before instructions that must not be retried in case of a page fault or similar.

The final one is either the most or the least interesting one to you, since it allows a simpler way to guarantee atomicity during XNACK replay.

Summary

It seems like a good improvement and AMD is catching up on features with both Hopper and Blackwell, hopefully that means that they can not only beat Nvidia in theoretical performance but also challenge the market leader in practical performance with MI455X and Helios.

AMX TMUL offered a highly configurable setup where code would specify matrix tile parameters for each tile register. For instance, the tile register tmm0 could be set up as a 16x64 matrix of INT8 values by specifying 16 rows and 64 bytes per row (“colsb”). TMUL matrix multiply instructions like TDPBSSD would take the tile configuration into account and perform an entire matrix multiplication operation between specified tiles. On the data type side, AMX TMUL could operate on INT8, FP16, and BF16 values. The latest iteration of TMUL, implemented on Granite Rapids-D CPUs, also supports complex numbers with FP16 real and FP16 imaginary components.

ACE does away tile register configuration options, and always considers them 64 bytes by 16 rows. Complex numbers are gone, but FP8 is in. On the compute-side, ACE provides outer product instructions instead of the inner product instructions given by AMX.

Arm’s Scalable Matrix Extension (SME) and its SME2 extension is an obvious comparison point. Both ISA extensions seek to accelerate matrix multiplication within the framework of a CPU ISA, providing a low latency alternative to using less integrated accelerators like GPUs. The two ISA extensions however differ in a number of areas. ACE is built on AMX, and continues to use AMX’s 8 KB of tile registers for holding matrix values.

Arm’s SME in contrast has a variable “streaming” vector length (SVL) just like SVE’s vector length (VL). The SVE and SME vector lengths do not need to be the same, and often are not. Like SVE, SME allows the vector lengths from 128 to 2048 bits, growing in powers of two.

The streaming vector length defines the size of the “ZA” storage array, which is SME’s equivalent of AMX tile registers. ZA storage is a 2D array where each side matches the SME streaming vector length. ZA storage capacity therefore ranges from 256 bytes with a 128-bit streaming vector length, to 64 KB with the maximum 2048-bit vector length.

Outer Products

While AVX512-VNNI and AMX accelerate inner products, ACE and SME accelerate outer products.

An inner product (or dot product in this special case) of two vectors a

and b is,

Or if you are a physicist you might have learned it via the geometrical interpretation of,

Where θ is the angle between the vectors. The outer product (or tensor product in the general case) of the vectors a and b on the other hand results in a rank-1 matrix C like the following.

The columns of C are all proportional to a and this is how we know that it is a rank-1 matrix, but also that matrix multiplication is just a sum of outer products like,

In fact many operations in linear algebra can be seen as a linear combination of outer products, making outer products a natural primitive to implement in a processor, the most obvious example the singular value decomposition of a matrix,

isn't trivial to understand when written as an inner product, but it is just the sum of the outer product of the columns of U and V multiplied by each corresponding singular value or,

Even algorithms that aren’t as obviously converted to outer product forms like the FFT have been reformulated in terms of it to utilise SME as an accelerator. Arm gave the following example on how to do it, but if you search for papers you can find various more effective ways to do it.

We have historically mostly used inner products because that reduces the amount of state we need to keep in registers, and since that is a precious resource that has been important to optimise for.

However, as we’ve seen above almost all linear algebra can be seen through both lenses, and code can easily convert between inner and outer products. SME takes advantage of this to do matrix multiplication via outer products and ACE seeks to do the same.

Data Type Conversion/De-Quantization

Model weights are often quantized to very small bit-widths to reduce memory bandwidth and capacity pressure. ACE and SME, unlike for example NVIDIA’s TensorCores, pre-process the input vectors in software without having to bounce them via the register file.

Quantized weights are converted to natively supported data types to make use of the accelerators. ACE leans on AVX-512/AVX10’s fixed 512-bit vector width to accelerate this conversion process. A 512-bit vector is large enough to hold a lookup table mapping a data type up to 6 bits to an 8-bit output using VPERMB, and for a 7-bit input VPERMI2B can use two 512-bit vector registers together as the lookup table.

A new VUNPACKB instruction added in ACE/AVX10.3 can extract 2 to 7-bit elements into byte-aligned positions and then the previously mentioned vector permute instructions can perform the data type conversion. ACE thus provides a degree of future proofing by being able to handle any data type between 2 and 7 bits with just these three instructions, and you can with software implement essentially any method of your choosing.

The x86-64 EAG also hopes this flexibility will let ACE hardware apply to applications beyond de-quantizing model weights, such as codebooks for compression of data.

Arm can’t rely on having vector registers wide enough to work as lookup tables for data conversions, because SVE lets implementations define streaming vector lengths from 128 to 2048 bits. SME2 therefore adds a 512-bit ZT0 register specifically designed to work as a 16 x 4B lookup table. LUTI2 and LUTI4 instructions perform data conversions by decompressing 2-bit or 4-bit index values from a vector register, looking up their values in ZT0, and placing output values into a destination vector register.

Adding the fixed width ZT0 register lets Arm accelerate data conversions within the variable vector length SVE/SME framework, but isn’t as flexible as ACE’s VUNPACKB + permute combination. Model weights quantized down to anything other than 2 or 4 bits wouldn’t benefit from SME2’s lookup table acceleration and more complicated codebook approaches that require multiple different lookup tables cannot be supported.

On the other hand, SME2’s mechanism lets Arm express a data type conversion with a single instruction instead of two. It also reduces pressure on vector registers, because the lookup table is stored in a separate register and LUT instructions don’t need a separate vector register to hold intermediate unpacked values. On the flexibility front, Arm could continue extending the ISA and add LUT instruction variants to support more data type widths.

Block Scaling Support

Low precision data types like FP8 suffer from low dynamic range. The Open Compute Project’s Microscaling Formats specification tackles this with scaling factors. One scaling value applies across a block of values, increasing the dynamic range of multiplication results without increasing the bit width of each data element. ACE supports this with a new 1024-bit Block Scale Register named BSR0. BSR0 is divided into two 512-bit halves, one for each of the two inputs of an outer product operation. Each half is further divided into four groups of 16 8-bit scale values, selected by an immediate operand in outer product instructions.

Arm’s SME supports scaling via 8-bit LSCALE and LSCALE2 fields in the Floating Point Mode Register (FPMR). BF1CVT uses LSCALE, and BF2CVT uses LSCALE2. Although Arm documents two instructions, the scheme is actually similar to how ACE selects BSR groups using an immediate. There’s a 1-bit opcode difference between BF1CVT and BF2CVT, which a creative decoder could treat as an immediate. Using scale factors is a two-step process with SME, where BF1CVT/BF2CVT apply a scale and a separate outer product instruction carries out the math. Update instructions for both ACE’s BSR0 and SME’s FPMR will overwrite the entire register, signaling that ISA designers on both sides don’t expect scale values to change often.

There’s a distinct possibility that neither register gets renamed in hardware implementations, so any scale value update would create a serialization barrier for FP8 instructions. If scale values do change often enough to warrant renaming, then both ISAs are in a difficult position. All of ACE’s FP8 outer product instructions reference BSR0, and therefore depend on the last BSR0 write. SME has separate instructions for scaling and outer product math, but FPMR also includes fields that control FP8 format and overflow behavior. It therefore ends up in the same situation of where all FP8 math depends on the last FPMR write.

Tile Size: Leveraging Vector Registers

Quantization is one way to reduce memory bandwidth demands. Tiling is another. Simple matrix multiplication creates memory access patterns with poor locality. Splitting a matrix into submatrices (tiles) helps improve data reuse by fitting into high bandwidth caches, rather than frequently streaming in data from DRAM. But bandwidth challenges remain even with tiling. Even cache bandwidth can become a constraint with enough compute available. If there’s enough compute to get through tiles faster than they can be streamed in from lower levels of the memory subsystem, L2, L3, and even DRAM bandwidth can come into play.

Larger tiles help mitigate this problem. Calculating each tile of the result matrix involves multiplying each tile in the corresponding row of the first input matrix with every tile in the corresponding column of the second input matrix. Larger tiles mean a matrix can be covered with fewer total tiles. Because each input tile appears less often in an output tile’s matching row or column, it’s loaded fewer times throughout the matrix multiplication process.

Register capacity influences tile size, because we want to keep the accumulator in registers. Each accumulation is a read-modify-write operation, which is two memory accesses. Moreover, most memory subsystems have less store than load bandwidth, reflecting typical application behavior where memory reads far outnumber writes. ACE keeps the same 8 KB of tile registers that AMX introduced, but changes the math instructions to take two inputs from AVX-512 vector registers. TMUL math instructions take all of their operands from tile registers, forcing code to use some tile register capacity to hold temporary input data.

For example, take a C = AB matrix multiplication operation with 32K by 32K matrices of 8-bit integer values (INT8). A and B will be input matrices, and C will be the result. The matrix multiplication requires 32768^3 or 35.2 trillion multiply-add (MAC) operations. C will be kept in registers in all cases, while A and B will only be given registers as necessary.

AVX512-VNNI

AVX-512 provides 32 512-bit vector registers. We can use about 24 of those registers to store accumulators for C, with the other eight used to store input values. AVX-512 FMA instructions can only take one input from memory, meaning the other input has to be loaded into registers before being operated on.

We need to load 16 x 32k elements from A and 24 x 32k elements from B to calculate our output tile, so we need to load 40 x 32k ≈ 1.3M elements from memory. We need to run this kernel 32k x 32k / (16 x 24) ≈ 2.7M times, for a total of about 3.4 TB of cache traffic. That works out to approximately 0.10 bytes loaded per MAC operation. A data cache capable of sustaining 128B/cycle of load bandwidth would be able to feed 1228 MAC operations per cycle, which is plenty considering each 512-bit vector contains 64 8-bit elements, and no CPU is likely to have over 19 512-bit vector pipes.

SME

If we use the same trivial tiling with a SME implementation with a 512-bit streaming vector length, and a 4 byte accumulator, we get 4 SME ZA tiles of 16 x 16 elements each for a total of 32 x 32 elements. For each iteration we need to load 32 x 32k elements from A and B for a total of 2M elements in total. But we only need to run the kernel 32k x 32k / (32 x 32) = 1M times, for a total of 2.0TB of cache traffic. SME therefore cuts cache bandwidth demands by almost 50% compared to AVX-512-VNNI. At about 0.06 bytes loaded per MAC operation, a data cache capable of delivering 128B/cycle could feed 2088 MAC operations per cycle.

In addition, due to the outer product design we do not need any extra L1 traffic from spilling registers since we have plenty of vector registers available to us to do any pre-processing we want to do for practical applications.

Apple’s M4 is an example of a hardware implementation with a 512-bit streaming vector length. A hypothetical implementation with a larger vector length would benefit from further bandwidth savings. For example, a 1024-bit SME implementation would have a 64 x 64 element output, loading 2 x 64 x 32k elements and 0.25 million kernel invocations for a total of 1.0TB of cache traffic.

AMX

For AMX, we’ll use six tiles to store the output accumulator, and use one tile each for A and B. This should be doable according to my calculations and backed up by this paper you at most need to spill two tiles per seven you load which should fit trivially in L1 and therefore we’ll conveniently ignore the extra cache traffic.

Our six tile registers allocated mean that we have 32 x 48 element output, and we again need to load 32 x 32k elements from A and 48 x 32k elements from B per kernel invocation for a total of 2560 tile loads. But to compensate we just need to run the kernel 0.7 million times for a total of 1.7 billion tiles loaded or 1.7TB of cache traffic. That works out to 0.052 bytes loaded per MAC operation, meaning a 128B/cycle L1D could feed 2457 MACs/cycle.

If we instead used Intel’s recommended strategy we would need to run the kernel 1.0 million times and load 2048 tiles per invocation for 2.0TB of data loaded which is so significantly worse that I am going to conveniently ignore this number for the rest of the article since I can’t think of an obvious reason you would ever do this tiling strategy.

ACE

ACE matrix instructions take inputs from AVX-512/AVX10 vector registers and accumulate into tile registers, so A and B don’t have to be kept in tile registers like with AMX. That allows for a larger 32 x 64 element output. We need to load 96 x 32k elements for a total of 48k vector loads but with a mere 0.5 million kernel invocations for a total of 1.5 TB of cache traffic. At 0.046 bytes loaded per MAC operation, a 128B/cycle L1D would be able to feed 2785 MACs/cycle.

If M = N = K and T_a and T_b are the dimensions of the tile, then we need to load (T_a + T_b) x N elements per kernel invocation, and require N x N / (T_a x T_b) kernel invocations for a total of (T_a + T_b) x N x N x N / (T_a x T_b) elements loaded.

Now that I have done all the calculations I can reveal to you who glossed over the earlier numbers that the general lower bound for the amount of data reuse is 2 x M x N x K / sqrt(T) where T is the size of the accumulator tile in elements and if you want to consider this problem deeper I can recommend reading this paper on the MOMMS family of matrix multiplication algorithms.

This matches our experimental numbers, when we quadrupled the SME tile size we cut bandwidth requirements in half and this theoretical lower bound is hit when the accumulator is square, but the ACE register file was not expanded to the full 16 registers available in the VEX encoding used in AMX.

Since the AMX documentation mentions this as a thing they are considering for the future we can expect it to happen at some point if ACE stays around and then it will perform like a 1024-bit streaming vector length implementation of SME in this experiment.

ACE’s increased tile size can let Intel implement a larger MAC array with a given level of L1D bandwidth, but the benefits over AMX will likely be minor in this regard. Using AVX-512 registers to hold A and B only increases tile size slightly, because the bulk of register storage capacity is concentrated in the tile registers. Still, any improvement from an ISA extension is welcome. Data movement consumes significant power in modern cores.

Zen 5 for example reduces clocks if it has to fully feed its vector units and sustain maximum L1D bandwidth. ACE could therefore improve performance in compute bound scenarios if fewer L1D accesses allow for higher boost clocks. If an ACE implementation has enough compute to become bound by how fast it can stream tiles in from lower levels of the memory hierarchy, bandwidth savings from covering the matrix with fewer tiles can translate directly to improved performance.

Other Uses of ACE Tile Registers

ACE doesn’t change AMX’s tile registers, but could still open up interesting use cases for tile registers. ACE’s TILEMOVROW and TILEMOVCOL copy rows or columns between vector registers and AMX tile registers. Data no longer has to go through L1D as it did on AMX, making it more feasible to use the AMX tile registers as a general purpose scratchpad. In theory, software can stash values in the AMX tile registers instead of writing them to memory to reduce pollution in the L1 data cache. If copying in values from the AMX tile registers takes an independent hardware path from L1D loads, using the tile registers could alleviate L1D bandwidth pressure too.

In practice, using tile registers as a scratchpad would be difficult. Data movement between tile and vector registers don’t benefit from optimized forwarding paths the way L1D accesses do. An ACE hardware implementation could even have relatively long latency for tile to vector register data movement with minimal performance impact. Accumulator data can stay in place over many tile computations before being written out, making data movement out of the tile registers a comparatively rare operation, and therefore one that’s less important to optimize. Long latency scratchpad loads on the other hand could frequently hold up dependent instructions.

Tile register capacity is another limitation. OpenCL 1.0 for reference requires 16 KB of scratchpad memory (termed local memory) at minimum, matching per-core scratchpad memory from early AMD and Nvidia GPUs. 8 KB is a lot for a register file, but is on the low side for scratchpad memory. Finally, initializing AMX or ACE would require the operating system to preserve that extra 8 KB of registers across context switches, potentially increasing context switch latency.

Therefore, I don’t expect many applications (if any) to use tile registers as a software managed scratchpad. Spilling registers to memory is simpler, is already supported by every existing CPU, and may even result in better performance.

Final Words

ACE has many parallels to Arm’s SME2, which isn’t a surprise because Arm also took part in drafting the OCP Microscaling Formats specification. Both ISA extensions focus on outer products, and feature built-in support for scaling factors. The biggest difference is ACE’s focus on leveraging AVX-512’s large vector registers, and SME2’s focus on scalable vector length. Each ISA extension has advantages in some areas, and could face difficulties in others. AMD, Arm, and Intel could also extend their ISAs further as they get more experience and requirements evolve.

Hardware design is ultimately the big wildcard in evaluating ACE. How an ISA extension gets implemented in hardware influences performance and efficiency far more than the ISA design does. No ACE hardware implementations exist to date, making that impossible to evaluate. Intel and AMD may be looking to support ACE in hardware as early as their next generation of CPUs, but at the time of writing that hardware is not available. The quality of those hardware implementations will ultimately determine how well ACE does, and we look forward to seeing how that goes.

Here at ISC 2026 in Hamburg, Germany, we got the 67th TOP500 list where there was a surprise awaiting us. That surprise being a new Number 1 Supercomputer on the TOP500.

LineShine Shining on Through

The new number one Supercomputer on the TOP500 list is the LineShine Supercomputer in Shenzhen, China. This is the first Chinese submission to the TOP500 in 9 years and they came in swinging with a massive CPU-only system.

Starting with the specifications of the CPU that powers the LineShine system, the LX2.

The LX2 is an Armv9-compliant CPU with support for SVE2 and SME. Each core has 32 KB of L1 instruction cache and 32 KB of L1 data cache. Physically, the chip is built from two compute dies, with each die containing four 40-core clusters. Two cores are disabled per cluster, leaving 38 active cores per cluster, or 152 active cores per die. Each cluster is backed by 28.5 MB of L2 cache, giving each die 114 MB of L2 and the full LX2 package 304 active cores with 228 MB of total L2 cache.

Those 304 cores run at 1.55 GHz and deliver a quoted 60.3 TFLOP/s of FP64 compute at 690 watts per LX2 CPU. The package also includes eight stacks of “high-bandwidth memory,” with 4 GB per stack for 32 GB of on-package high-bandwidth memory and 4 TB/s of bandwidth. However, this memory likely is not conventional HBM, despite being described in similar terms and could be an indigenous Chinese development. Because 32 GB is relatively small for a CPU of this scale, each LX2 is also backed by 256 GB of DDR5 memory that acts as a larger spillover tier.

Transitioning from the CPU level to the node level, each node has two LX2 CPUs with 800 Gbps of networking for a total of 1.6Tbps of networking per node. 8 of these nodes are then combined into a compute blade, with 16 compute blades in a compute frame, and 2 frames per compute cabinet. In total the LineShine supercomputer has 90 compute cabinets which means that there are over 22,000 nodes and 13 million CPU cores in the full system.

This puts the LineShine system at a total of 2.198 Exaflops of sustained FP64 (Rmax) out of an on-paper 2.735 Exaflops of FP64 (Rpeak). For this result, the LineShine system pulled 42.22 Megawatts of power for a FP64 efficiency of 52.07 Gigaflops per Watt, which, while it is well behind the leader of 73.282 FP64 Gigaflops per Watt, is very impressive for a CPU only system. And unlike prior Chinese Supercomputers, this system is not just a “LINPACK-special” but is also the number 1 in the HPCG benchmark with a result of 22.004 Petaflops per second beating out El Capitan’s 17.406 Petaflops per second result.

The Rest of the Owl

There is a new number 6 system on list, the HPC7 system from Eni in Italy. This is functionally El Capitan, just shrunk down to 30% the size. It is using the same HPE Cray EX4000 platform and AMD Instinct MI300A APUs, which is getting an Rmax of 571.5 Petaflops of FP64 performance out of an Rpeak of 861 Petaflops at 8.735 Megawatts of power consumed. With HPC7, Italy now has more compute listed on the TOP500 than any other country in Europe, despite Germany having Europe’s only Exascale system on the TOP500.

The former number 1 supercomputer, Fugaku, may have been pushed down to number 9 in the TOP500 ranking but it is still a lean, mean, science crunching machine. Despite being 6 years old, Fugaku is still number 3 on the HPCG list, which is a testament to the HPC focused design that Fugaku has, as it is still in the top 3 supercomputers as far as the HPCG list is concerned.

Moving to the Green500, there have been no changes to the Green500’s Top 10 list of most efficient supercomputers. This is the first time in the Green500’s history where there have been no changes to the Top 10 list. However that is not to say that HPC hasn’t gotten more efficient.

Due to retirements of older systems, the overall field of HPC has gotten more efficient in the past 6 months.

Conclusion

Personally, I wasn’t expecting to see a new number 1 on the TOP500, until either Discovery or Fukagu-Next was brought online. We have known that China has Exascale systems for a while now, but this is the first time that we have seen a China Exascale system on the TOP500.

This does beg the question of if China will also submit TOP500 results for their other exascale systems, Sunway Oceanlight and CNIS, or if they will keep those off of the TOP500 list.

I do wonder if this will cause the US Government to spend more money for large DOE systems and I am not the only one thinking that it could drive more money towards the DOE.

If the submission of LineShine ends up with the DOE getting more funding for more, and larger, supercomputers, then bully for us in the HPC community.

Something that is quite interesting is that Eni, an oil and gas company, has been submitting their systems to the TOP500 and they currently have 2 of the Top 10 systems on the TOP500. Which then begs the question, why aren’t the truly large AI systems, such as xAI’s Colossus 2 with over 550,000 Blackwell GPUs, on the TOP500? Why aren’t these AI companies submitting to the TOP500 to show off their computing prowess? To be honest, I don’t know why. A full HPL run on a supercomputer seems like a good way to stress all of the compute, memory, and networking of the system before declaring it to be operational.

There was one last key piece of news that came out of the TOP500 here at ISC 2026 and that is that the ISC Group will be handing over the TOP500 list to ACM SIGHPC moving forward. This functionally means that now all TOP500 lists will have dedicated DOI numbers which will make referencing specific TOP500 lists easier moving forward.

Transcript below has been edited for conciseness:

George Cozma: Hello, you fine internet folks. We’re here at Computex 2026 at the Intel booth, or booth-room-floor area, whatever you want to...

Kira Boyko: Call it Intel space.

George Cozma: Yeah, well, whatever you want to call it. And I’m here with...

Kira Boyko: I’m Kira Boyko, and I am the product director for Intel Xeon 6+, which we have just launched at the event this week.

George Cozma: Now, starting off with a kind of simple question: what is a product director?

Kira Boyko: That is a good question. Actually, I get asked that somewhat frequently. A product director is someone who defines what the product is going to look like. So we look at the market requirements, what segments you want to target, how you want the product to perform, the different KPIs, and then you work all the way through to execution and delivery to make sure we are staying on track to our customers’ needs.

George Cozma: So are you actively saying, okay, this product needs to have at least X number of cores at X clock speed for X performance?

Kira Boyko: With X frequency, X application for this segment is going to need this KPI of performance. It needs to be able to support this and that, and this other part doesn’t matter. And then, working with our customers, of course, to answer those and understand their various applications, we build out models to ensure that we are supporting those appropriately.

George Cozma: Cool. And speaking of models, I assume you also are part of the SKU-ing, the people who decide what goes into, let’s say, the 6990...90+...E-Plus?...

Kira Boyko: 6990E+.

George Cozma: Okay, thank you.

Kira Boyko: Very confusing, yes.

George Cozma: And then, like, the SKUs down there, all the way down to the, I believe, the 6960E+.

Kira Boyko: Yes.

George Cozma: And so you determine what the gaps are and all that, or...?

Kira Boyko: Absolutely.

George Cozma: Okay. How are you doing all that? Are you just asking your customers what they want, or are you sort of just deciding through what can be cut down from different yields and whatnot?

Kira Boyko: It’s a combination of all the above. So it’s looking at all of our segments, understanding what those segments are going to need from an application perspective, and then determining how we best create SKUs to fit that. And then, of course, to your point, there is a lot of looking at the overall utilization to ensure that we are hitting our marks in terms of demand on a per-SKU basis.

But yeah, we try to keep it as simple as possible. Xeon 6+ has a simpler roadmap, which I’m particularly proud of, because Xeon honestly has a huge number. Yes. And we’re going to see a lot of crossover and application between some of our segments there, which is also going to help with the overall build and supply so that we have more material available for folks down the road to choose from.

George Cozma: Okay, excellent. And sort of, I guess, since you’re working with your customers, where in the design segment do you come in? Do you come in sort of as the product is being defined? So I guess before launch, how far back are you sort of involved?

Kira Boyko: Oh, years.

George Cozma: Okay, okay. So you’re sort of at the beginning.

Kira Boyko: Yes. Okay, yeah. The product manager actually starts out with the full product concept, or product kickoff, as well, right? So you start by saying, hey, this is what we need to deliver in this time frame. This is what we’re hearing from our customers. And then you start working with an architect and various other engineers to actually figure out how you’re going to build this and achieve it.

George Cozma: Okay. Now, sort of moving into Xeon 6+, the thing that really interested me the most was AET. Could you talk a little bit more about what that is, what the acronym is, and what that gives you over standard performance counters?

Kira Boyko: And we love a really good acronym. It is Intel Application Energy Telemetry, which you hopefully have, like, a sub-caption below this saying that, because...

George Cozma: I was like, when I first wrote it, I thought it was Advanced Energy Telemetry. I was like, wait...

Kira Boyko: I like that. I mean, sure, we can go with that too. I like that.

George Cozma: Should be XET. It didn’t just sort of fall into the everybody-has-an-X stuff.

Kira Boyko: That, and maybe a plus at the end, you know, just to make it super fun.

George Cozma: Or throw Ultra in the box as well.

Kira Boyko: Yeah, exactly. So it’s a new feature that we are introducing with 6+ that will be rolling out on all of our Xeons moving forward. So that’s important to know because, while 6+ is very focused on specific scalar workloads, we will have other CPUs, of course, that are focused on others, where this is going to be equally applicable. It is a feature that was highly requested by some of our customers.

So we developed that alongside feedback throughout the process. And it allows them to track, at a hardware core level, the actual energy usage of their workload as it’s moving from core to core. With that information, they can orchestrate differently so they’re using less energy. They can provide end-customer applications like visible chargeback to the actual energy utilized, or incentives with rebates to get them to utilize differently in order to reduce their energy consumption as well.

Previously, I believe there have been similar offerings that are on a software level, which adds additional tax and leaves space for misapplications, because it’s always different workloads that they’re tested on versus actually utilized by an end customer. So we’re really excited about this one. And it is available on all of our SKUs, which is great. And it works out of the box, so everyone can take advantage of it.

George Cozma: Cool. I assume it’s plumbed straight into perf?

Kira Boyko: Perf, from a metrics standpoint?

George Cozma: Yeah. Well, the performance counter suite, perf, in Linux.

Kira Boyko: Yes, yes, okay, yes. And it’s also compatible with all of our tools as well. Okay. So no additional work needed to be done to utilize that.

George Cozma: Are these hardware-level sensors, or is this like software modeling? So I know, for example, on laptops, instead of sometimes what you do is you actually just sort of model what the power should be instead of having an actual counter. Is this like hardware sensors embedded into the Xeon processor for this?

Kira Boyko: It is. It is a hardware-level hook into the core that allows tracking.

George Cozma: Okay. And then how, I guess, so Xeon 6 is you have effectively 72 clusters of four cores. How is it at the core level, or is it at the cluster level? Okay. Can you also get cluster-level monitoring as well?

Kira Boyko: I would have to check. I imagine that can certainly be viewed through it, but we’ll check and we’ll get back to you, okay.

George Cozma: Because, I’m sorry, zooming out from there, it’s not only do you have that cluster, you then have the whole SoC, you have the uncore and stuff. So I was asking if you could monitor the power of the entire system, sort of.

Kira Boyko: Oh, yes.

George Cozma: Okay. So all the way from the package level straight down into the single-core level. Okay, cool. It’s always great. You get more data points.

Kira Boyko: Yeah, yeah.

George Cozma: And more hardware sensors as well, also really fun to measure. What power differences can you get from an integer workload versus an FP workload and all that fun stuff? And you said moving forward, next-generation Xeons and continuing on will have this, are you thinking about potentially, I know this may not be your area, bringing that down into the consumer realm and sort of plumbing that into consumer devices as well?

Kira Boyko: Yeah, not my realm on the consumer side. But we do usually share information about our features that we’re building on the datacenter side to see if there’s application and use for clients.

George Cozma: Okay, awesome. Because I guess another question is, sort of continuing on that, how much do server and client talk to each other in terms of not just technology, but the sharing of different written knowledge?

Kira Boyko: Oh, quite a bit. Yeah, we’re very transparent. If they’re doing something super cool over there, we want to know if it can be applied to our space, and vice versa. We also have a lot of open dialogue around new features that we have coming out to support specific cores that are going to cross over. For example, 6+ is the first 18A Xeon. There’s also an 18A client that already launched.

George Cozma: Which just so happens to be in my bag.

Kira Boyko: Awesome. So we do a lot of cross-pollination to see where we can best leverage our resources and make sure that we are supporting the feature sets that are right for both of our audiences, as well as separately too.

George Cozma: Okay. I guess that leaves me with one final question, the most important question of this interview: what’s your favorite type of cheese?

Kira Boyko: I love blue cheese, specifically because I am a blue-cheese-stuffed-olive-in-a-martini person, so I’ll say that.

George Cozma: Blue cheese is not my personal favorite, but I have to respect people who enjoy it.

Kira Boyko: Are you going to share your [favorite cheese]?

George Cozma: It’s cheddar and my favorite as of late has been, because I just got it, somebody from Oregon just sent it to me, Tillamook 12-year cheddar. Oh, so good. But I have been getting into Manchego, so sheep’s cheese, so sheep’s milk cheese, which is a lot more sort of fatty, and...

Kira Boyko: Have you tried goat milk cheddar?

George Cozma: Cheddar? Yes, I have. Goat milk cheese, for me, is a bit hit or miss. It depends on how it’s been done. But I really like Humboldt Fog from Cypress Grove.

Kira Boyko: I know that one. Yep, delicious.

George Cozma: Yep, yep, I’m with you. Well, thank you so much for sitting down to interview sort of last minute. But thank you so much for watching. If you like interviews like this, hit like, hit subscribe. It does help, as much as it pains me to say it, to show the like and subscribe buttons. But if you would like a transcript of this, that will be on the Chips and Cheese Substack, along with links to our Patreon and PayPal. Thank you so much, folks. Have a good one.

Because of SPEC’s importance in the CPU performance world, I’ll be looking over the new suite’s workloads to examine the challenges that they serve up to CPUs. I’m interested in hardware rather than compiler comparisons, so I’ll be using GCC 14.2.0 with -O3 and native architecture/optimization targets. I did try GCC 15.2.0, but ran into various issues and decided to stick with GCC 14.2.0 to conserve time. I’m testing all of the systems below using Linux.

Scores

SPEC CPU scores represent speedup ratios relative to a reference system. Each SPEC CPU update tends to update the reference system, though not always to one that’s a relevant comparison for recent hardware.

An Ampere eMAG 8180 system provides the reference score of 1.0 for SPEC CPU2026. Ampere’s eMAG is faster than the Sun Fire V490 used for SPEC CPU2017, in much the same way that a Cessna 172 might be faster than a Sopwith Camel. Neither is a good comparison against modern airliners. Similarly, Ampere eMAG was not a widely deployed platform, and even systems that pre-date it by many years outperform it by a large margin. I’ve heard part of the motivation behind using a slower system was to let most systems achieve high scores, but that could have been accomplished by making the reference score a higher number, like 1000, instead of 1.0. Geekbench 6 takes a more reasonable approach, with a Core i7-12700 calibrated to a reference score of 2500. Unlike Ampere’s eMAG, Intel’s Core i7-12700 and similar CPUs were widely deployed across consumer systems, and Intel today uses a similar core architecture in Xeon 6.

Examples of Intel and AMD’s latest desktop CPUs show similar performance in SPEC CPU2026’s integer suite, while Zen 5 tends to pull ahead in floating point tests. I had to run SPEC CPU2026 on a Lion Cove core that only reached 5.5 GHz, because the two 5.7 GHz capable cores had trouble completing the test suite without crashes. There seems to be something wrong with my sample, but I suspect a Lion Cove core that does complete the tests at 5.7 GHz would narrow the gap.

Individual scores in the integer suite show Zen 5 and Lion Cove closely matched, as the aggregate scores would suggest. Absolute scores show just how far outmatched the Ampere eMAG system is. Current desktop cores obliterate ones in the Ampere eMAG, especially in 706.stockfish. Even the FX-8350, which is over a decade old, might be a better reference point. It also sails way past the eMAG system in nearly every test.

Floating point workloads are even worse for the old eMAG system. Zen 5 has a field day in several of the floating point workloads. Some of that is because GCC was able to generate AVX-512 instructions. I used Intel’s Software Development Emulator to get instruction counts for the last invocation of each workload, to give an idea of what instruction types are in use. Some SPEC CPU2026 tests run multiple commands to test the same binary against different input data, but I only profiled the last invocation to save time.

706.stockfish, 749.fotonik3d, and 765.roms all include AVX-512 code when compiled with GCC 14.2.0. Several other tests take advantage of 128-bit or 256-bit vectors too.

Workload Difficulty: IPC

Average IPC, or instructions per cycle, gives a rough idea of how well CPU cores can bring their execution resources to bear. Low IPC often indicates cache misses, branch mispredicts, or less commonly, specific performance hazards in a core’s architecture. High IPC suggests performance is limited more by execution latency, execution resources, or core width. IPC of course shouldn’t be confused with actual performance, which depends on clock speed as well as work done per instruction.

SPEC CPU2026’s integer suite displays a higher and tighter IPC distribution than SPEC CPU2017’s integer suite on both AMD’s Zen 5 and Intel’s Lion Cove. The IPC spread feels close to that of Geekbench 6, which tends to emphasize core throughput with few branch mispredicts or last level cache misses.

Floating point workloads get a wider spread on Zen 5 with the updated suite, while SPEC CPU2017’s floating point tests tended to bunch up around 2-3 IPC. Lion Cove had a wider IPC spread with the older floating point suite, and continues to see a wide spread in the new one.

505.mcf and 520.omnetpp were low IPC workloads in SPEC CPU2017. Both suffered plenty of cache misses, and 505.mcf suffered a lot of branch mispredicts too. Neither test has an equivalent in SPEC CPU2026’s integer suite. MCF is gone, and 710.omnetpp is a completely different animal despite sharing the same name as the old test. Two code compilation tests, 721.gcc and 725.llvm, take over as the lowest IPC integer tests. They still average above 1.5 IPC though, which isn’t too low. For perspective, most PC games average around 1 IPC.

On the other hand, SPEC CPU2026’s integer suite includes plenty of high IPC workloads. More than half average close to 3 IPC if not beyond. 750.sealcrypto reaches the highest IPC across all integer workloads on both Zen 5 and Lion Cove. It’s also a high IPC workload on the Skymont E-Core, if just a little less so.

SPEC’s floating point workloads tend to focus on core throughput, and that remains the case with SPEC CPU2026. Many workloads sit above 2 IPC, which I consider a medium to medium-high IPC range. 749.fotonik3d and 765.roms fill the same role as their SPEC CPU2017 counterparts, and continue to stress DRAM performance by often missing in the last level cache. At the high IPC end, 538.imagick is gone with no direct replacement. Several SPEC CPU2026 floating point workloads do average well above 3 IPC, but none go as astronomically high as 538.imagick did.

Top-Down Analysis

Top-down analysis attributes under-utilized core width to various reasons, and provides a high level overview of where potential core throughput is being lost. It works at the rename/allocate stage, which is typically the narrowest part of a CPU’s pipeline that all instructions must pass through. Each pipeline slot at that stage is classified as follows:

• Retiring: Slot was used by a micro-op that ultimately retired, meaning its results were made architecturally visible after passing all checks. Retired micro-ops represent useful work.

• Backend Bound: A micro-op was available from the frontend, but couldn’t be sent to the backend because a required out-of-order tracking resource wasn’t available. That could mean the reorder buffer, register files, or memory ordering queues didn’t have a free entry when one was needed.

• Frontend Bound: The frontend didn’t supply enough micro-ops to feed all renamer slots

• Bad Speculation: A micro-op went through the slot, but wasn’t retired. This represents wasted work, for example from branch mispredicts

Retiring slots are up across the board compared to SPEC CPU2017’s integer suite, which is expected given the higher IPC figures seen above. 723.llvm and 721.gcc are mostly held back by frontend latency, which is classified as cycles where the frontend didn’t supply any micro-ops to the renamer. Both 723.llvm and 721.gcc are branchy workloads and occasionally suffer mispredicts, which interrupts the frontend’s ability to follow the instruction stream and smoothly deliver micro-ops. At the other end, 750.sealcrypto barely has any branches. Both Lion Cove and Zen 5 blast through it unimpeded, since there’s not much slowing their backends either.

SPEC CPU2017’s floating point workloads tend to focus more on core throughput, and that continues to be the case in SPEC CPU2026. Branches tend to be rare and predictable, letting the frontend on Lion Cove and Zen 5 operate with high efficiency.

Fotonik3d and roms are heavily backend memory bound on both Lion Cove and Zen 5. Most other tests are core-bound, though there are exceptions. Both Lion Cove and Zen 5 are challenged in similar ways, except for 709.cactus. In that test, Zen 5 is bound by frontend bandwidth, meaning the frontend delivered some micro-ops but not enough to fill all eight renamer slots.

Lion Cove’s frontend has a field day in 709.cactus, but the core runs into backend limitations and overall Intel isn’t able to pull ahead of AMD.

Branch Prediction

Branch prediction has been a key challenge in designing high performance CPUs for decades. SPEC CPU2017’s integer tests were often challenging from a branch prediction perspective, even for modern cores with sophisticated branch predictors. 505.mcf, 541.leela, and 557.xz all suffer plenty of mispredicts on AMD’s Zen 5. Those tests would reward CPUs with better branch prediction, faster branch resolution, and the ability to hide mispredict penalties behind other work.

SPEC CPU2026’s integer suite places less emphasis on branch prediction. 721.llvm remains a moderate challenge, but suffers fewer mispredicts per instruction than 557.xz, to say nothing of 505.mcf and 541.leela. Lower difficulty on the branch prediction side contributes to higher IPC averages.

Floating point workloads in SPEC CPU2017 tend to be less challenging for branch predictors, though not always. 526.blender did lose throughput to bad speculation, but SPEC CPU2017’s floating point tests didn’t get close to 557.xz, 505.mcf, or 541.leela in terms of branch prediction difficulty.

SPEC CPU2026 moves the needle on that, with 731.astcenc stepping up as the single biggest branch prediction challenge across both the integer and floating point suites. It’s still not as big of a challenge as 505.mcf from SPEC CPU2017’s integer suite, but it should still suffice to represent a workload with unpredictable control flow.

Code Footprint and Instruction Caching

Good caching is important for instructions as well as data. AMD’s recent cores lean towards optimizing for small instruction footprints, and seek to capture much of the instruction stream in a highly optimized micro-op cache. SPEC CPU2026’s integer suite challenges that approach more than SPEC CPU2017’s did. Several workloads have less than 80% op cache coverage, so it appears that SPEC’s move to workloads with more source code lines sometimes correlates with worse code locality. Still, Zen 5 is able to mostly run code out of its op cache, and the decoders don’t see much use.

Further down the cache hierarchy, many of the tests with less than 90% op cache coverage see significant L1 instruction cache misses too. AMD has stuck with a 32 KB L1 instruction cache ever since Zen 2 which doesn’t offer too much extra coverage over the op cache. In nearly all tests, the 1 MB L2 cache is enough to capture the vast majority of L1 instruction cache misses. Only the two code compilation workloads see a noticeable degree of code fetches that miss L2, which could contribute to their lower IPC.

Intel takes a different frontend approach, with a smaller 5.2K entry op cache backed by a larger 64 KB instruction cache. Intel also uses a 8-wide decoder, which can provide higher instruction bandwidth to a single thread for large code footprints than AMD’s 4-wide, per-thread decoders. Lion Cove’s larger instruction cache suffers fewer misses across the board, and nearly eliminates code fetches from L2 in 714.cpython and 706.stockfish. Where the L1 instruction cache does see a significant number of misses, Lion Cove’s huge 3 MB L2 is able to step in and largely prevents code fetches from going through to L3.

For very small code footprints, Intel has a Loop Stream Detector that locks down the micro-op queue and uses it as a loop buffer. The LSD plays a minor role in most integer tests, except for 750.sealcrypto.

SPEC CPU2017’s floating point suite tended to be core throughput focused, with smaller instruction footprints than their integer counterparts, with exceptions of course. SPEC CPU2026’s updated floating point suite has more of those exceptions, and tougher ones too. 709.cactus has Zen 5 mostly feeding itself from its decoders, which explains why top-down metrics showed it as being frontend bandwidth bound. As with the integer suite, tests with lower op cache coverage tend to also miss in L1i. 749.fotonik3d doesn’t though. It spills out of the op cache more than its identically named predecessor in SPEC CPU2017, but all of that is caught by Zen 5’s L1 instruction cache.

LSD data from Lion Cove shows that many floating point tests spend a lot of runtime in tiny loops. 722.palm is a funny case, with Lion Cove’s 192 entry micro-op queue achieving over 40% coverage. Then once it gets out of that loop, it has a good chance of missing the op cache and the 64 KB L1 instruction cache.

None of the floating point workloads presents enough of a challenge on the code footprint side to spill out of L2.

Data-Side Cache Footprint

Data accesses that miss cache present another performance limiter for modern CPUs, especially for PC games. SPEC CPU2026’s integer suite has plenty of workloads that often miss in a 48 KB first level cache, and many that challenge a 1 MB L2 as well. However, the integer suite is light on workloads that encounter last level cache demand misses.

714.cpython and 750.sealcrypto rarely miss even in Zen 5’s L1D cache, which explains their high IPC alongside other factors.

Lion Cove performance counter data does a good job of justifying its 192 KB L1.5 data cache. As it turns out, quite a few workloads with significant L1D misses have nearly all of those misses contained within 192 KB. Last level cache misses are rare, suggesting a 36 MB cache is sufficient for many of SPEC CPU2026’s integer workloads.

On the floating point side, 709.cactus is prickly for a 48 KB L1 data cache, but Zen 5’s larger caches do a good job of cleaning up the misses. AMD’s L2 and L3 caches do well in general across the floating point suite. Only 765.roms and 759.fotonik3d have significant last level cache miss activity.

L3 MPKI figures are lower on Intel, but those don’t tell the whole story. Intel’s performance counters limit counting to retired loads, which disregards accesses from any loads that are flushed. Performance counters on both AMD and Intel also only count the first miss to a cache line that initiates a refill request.

Intel’s performance counters can also account for Fill Buffer (FB) hits, which occur when a load asks for data from a cache line that already has an outstanding miss request. That can happen if data accesses have good spatial locality, and multiple instructions request data from the same cache line. FB hits can also happen if the prefetcher initiates a cache refill request before an instruction asks for it, but data hasn’t arrived yet when an instruction does make the request. I suspect 749.fotonik3d and 765.roms run into a combination of both. They have a lot of FB hits, but few instructions cause a fresh L3 miss. I suspect many instructions matched an existing miss request started by the prefetchers.

Final Words

SPEC CPU2026 is a very different animal compared to its predecessor. The new suite has more variety on the code footprint side, with more tests that spill out of op caches and L1 instruction caches. On the other hand, branch prediction and data-side footprint have less variety. Few tests spill out of last level caches on AMD and Intel’s latest consumer chips, except for a couple of floating point workloads. I’m disappointed to see 520.omnetpp leave the room, because its behavior was the closest match across SPEC CPU2017 for gaming workloads.

Overall, those changes mean SPEC CPU2026 focuses more on core throughput than its predecessor. Larger instruction-side footprints do little to change that, because lowered branch prediction difficulties mean modern cores can smoothly stream code from their L2 caches. High IPC workloads certainly do exist, but it would have been nice if the newer suite improved coverage for lower IPC workloads too. Retaining an application like 520.omnetpp would have been nice. Instead, I feel like SPEC CPU2026 augments SPEC CPU2017’s coverage rather than being a perfect replacement.

Those differences make Geekbench 6 an interesting suite to evaluate alongside SPEC CPU2017. I looked at SPEC CPU2017 on Chips and Cheese several years ago. Now, I have a license for Geekbench 6 courtesy of Primate Labs’s founder, John Poole. It’s time to dig into the challenges Geekbench 6 workloads present for modern CPUs.

Instruction Composition

Because it’s distributed in binary form, Geekbench 6 can easily target ISA-specific features. Here, I’m running various workloads through Intel’s Software Development Emulator, which provides exact instruction counts and can emulate different ISA extensions without regard to what hardware it’s running on. With Intel’s Granite Rapids as an ISA target, AVX-512 plays a prominent role in Background Blur, Object Detection, and Structure from Motion. Granite Rapids represents Intel’s latest server platform, and comes with support for AMX for matrix multiplication acceleration. AMX shows up in Object Detection and Photo Library, where it accounts for 0.2% and 0.02% of executed instructions, respectively.

Even though AMX accounts for a small minority of executed instructions, AMX has an outsized impact. Ice Lake X represents an AVX-512 capable ISA target without AMX. Comparing results from Ice Lake X and Granite Rapids targets show that AMX can dramatically drop the number of AVX2 and AVX-512 instructions required to do the same work. Other tests show little to now difference between those two ISA targets, as expected.

AVX(2) has a huge presence across Geekbench 6 workloads, to the point that it’s easier to name workloads that aren’t heavily vectorized than ones that are. Text Processing, File Compression, Clang, and PDF Renderer don’t have much vectorization. Everything else uses plenty of 128-bit or 256-bit vectors.

Moving to Haswell gives an AVX2 capable baseline with no AVX-512 support. The three heavy AVX-512 workloads see those AVX-512 instructions replaced by more AVX2 ones. Object Detection is a particularly prominent example, with 256-bit AVX2 dominating the instruction stream.

An Ivy Bridge ISA target gives an idea of what happens with AVX support, but not AVX2. AVX provides 256-bit vector registers and 256-bit load/store operations, but 256-bit vector math operations are limited to floating point. With SDE emulating an Ivy Bridge ISA target, the executed instruction stream skews heavily towards 128-bit packed SSE operations. 256-bit AVX still shows up in many workloads, but accounts for less than 1% of executed instructions across most of them. Only Background Blur makes heavy use of 256-bit AVX.

Lastly, Intel’s Pentium 4 “Prescott” represents an x86-64 baseline from long ago. It’s mostly here for curiosity, as there are likely very few systems in service limited to this level of ISA extension support. The executed instruction distribution is surprisingly close to that of Ivy Bridge. 128-bit packed SSE operations dominate across most workloads.

SPEC CPU2017 doesn’t explicitly target ISA features, including vector extensions. To maximize portability, SPEC CPU relies on compilers to find opportunities to use ISA features. With GCC 14.2.0 compiling for an AVX-512 capable target (Zen 5), vector ISA extensions do show up in a few floating point tests. 549.fotonik3d and 554.roms execute a lot of AVX-512 instructions. Elsewhere, auto-vectorization plays a comparatively minor role. It’s almost absent in SPEC CPU2017’s integer suite, except in 525.x264 and 548.exchange2. Those two workloads use some 128-bit vectors.

Assessing Workload Difficulty: IPC

Instructions per cycle gives a quick overview of how “difficult” a workload is, for lack of a better word. Low IPC often indicates a workload is bound by branch mispredicts or cache misses, or less often, by hitting a particular deficiency in the core. Games for example tend to be low IPC workloads, bound primarily by the memory subsystem. High IPC in contrast points to a well fed core, with focus shifting to how fast the core can crunch through instructions.

Geekbench 6’s IPC distribution is skewed towards the medium to medium-high IPC range. Many workloads average well beyond 2 IPC on high performance cores like Intel’s Lion Cove and AMD’s Zen 5. Skymont isn’t explicitly a high performance core, but can often get close in IPC terms. However, it’s more prone to “glass jaw” cases, as can be seen in Clang, Asset Compression, Photo Library, and Structure From Motion.

Poking around with a couple of aarch64 cores tells much the same story. Arm’s Neoverse N1 and Neoverse N2 achieve reasonably good throughput considering their likely design targets. I consider anything above 2 IPC to be high for a 4-wide core, and Neoverse N1 hits plenty of those cases. Neoverse N2 is a 5-wide core ,and sails past 3 IPC on several tests. Neoverse N1 does end up with somewhat low IPC on Object Remover, but I suspect that’s down to a weaker vector execution setup. Only Navigation sticks out as a persistently difficult, low IPC workload across all the CPUs I tested.

SPEC CPU2017 shows a wider IPC distribution, with more difficult low IPC workloads. That’s especially the case for SPEC’s integer suite, where 505.mcf and 520.omnetpp pose huge challenges even for sophisticated, high performance cores.

Geekbench 6’s IPC distribution lands closer to that of SPEC’s floating point suite, though SPEC still shows more IPC variation. 549.fotonik3d is a very low IPC outlier, and represents a corner case bound by how much bandwidth a single core can pull from DRAM. In general, Geekbench 6 has a much tighter IPC distribution across its workloads than SPEC does.

Branch Prediction

Branch prediction is both difficult and vital to high performance. Modern cores devote enormous resources to branch prediction. Geekbench 6 has several workloads that challenge branch predictors even on modern cores.

Navigation is by far the worst, with a high enough MPKI figure that mispredicts are likely the single biggest reason behind its low IPC. More advanced predictors help, but don’t solve the problem. That creates a curious situation where newer cores see little IPC improvement compared to older, smaller ones from over a decade ago.

Aside from Navigation, File Compression and Clang present moderate challenges for branch predictors. Newer predictors tend to help in Clang, while older ones tend to get shredded.

Asset Compression is an interesting case, because branch prediction normally isn’t a problem in that workload. MPKI is well under control on Lion Cove, Zen 5, and both Neoverse cores. Even much older predictors with less storage budget and less advanced prediction techniques do reasonably well. Piledriver gets away with 2.76 MPKI, while Jaguar is just a tad worse at 2.96 MPKI. However, Skymont’s normally competent predictor trips over itself in this workload suffering 4.33 branch MPKI.

Op Cache Coverage

Cache misses can be another major challenge for modern CPUs. On the instruction side, only Clang has a large enough code footprint to spill out of Lion Cove’s 5.2k entry op cache. On the other end of the spectrum, several Geekbench 6 workloads appear to spend much of their time executing tiny loops. In PDF Renderer, HDR, and Photo Filter, Lion Cove’s frontend is often able to feed the core out of its 192 entry loop buffer. That doesn’t improve core throughput because Lion Cove is still limited by its 8-wide rename and allocate stage downstream, but it does suggest excellent code locality in those workloads. High loop buffer usage can also help save power, because much of the frontend could be turned off when running code out of the loop buffer.

Zen 5 has an even larger op cache with 6K entries, and a well optimized one at that. Again, Clang is the only workload that offers any significant challenge to the op cache. Even then, Zen 5 can mostly feed itself from the op cache rather than the decoders. That’s very much where AMD wants to be, since Zen 5’s decoder setup is weaker than Lion Cove’s when running a single thread.

Data Access

Data-side memory accesses tend to have more challenging patterns. Many of Geekbench 6’s workloads create heavy L1D miss traffic on Lion Cove. Most of these L1D misses are caught at L2. Lion Cove’s large 3 MB L2 pulls a lot of weight in making sure the core can sustain high IPC across much of Geekbench 6’s suite, despite high L3 latency. The new 192 KB L1.5D cache does well to, and occasionally captures the vast majority of L1D misses. Object Remover stands out as the only test with very high L2 miss traffic on Lion Cove.

L3 misses tend to be low, hinting at good data locality or predictable access patterns. Performance counters here specifically count the first retired load that created a miss to a 64B line. If a load requests data from a cache line that already has a miss request in progress, initiated either by a previous load or a prefetch, that counts as a FB (fill buffer) hit. I didn’t log that here. However, I did track traffic right in front of the CPU tile’s die-to-die interface, at what Intel calls the arbitration queue. I’m using that as a proxy for CPU-side L3 miss and DRAM bandwidth.

Many of Geekbench 6’s workloads request multiple gigabytes per second across the die-to-die interface, though none push the memory subsystem from a single thread as hard as SPEC CPU2017’s fotonik3d. Photo Filter is the most bandwidth heavy test in Geekbench 6, despite having a tame-looking 0.23 L3 MPKI figure. That suggests Lion Cove’s prefetcher is able to initiate many memory requests before an instruction asks for the data. Prefetching helps mitigate DRAM latency and keep IPC up. That low MPKI figure contrasts with behavior in games, which tend to miss L3 and can be sensitive to DRAM latency.

AMD’s performance events count demand data cache refills, rather than tagging loads with data sources and counting at retirement. Demand means a refill initiated by an instruction, as opposed to the prefetcher. They’ll generally give higher counts compared to events at retirement, especially if a lot of instructions execute but are later flushed (for example from branch mispredicts).

With 96 MB of L3, the Ryzen 7 9800X3D practically eliminates L3 misses for the Navigation workload and improves in Object Remover too. Elsewhere, the difference in where events are counted and the low L3 miss activity overall makes it hard to tell if the larger cache helps compared to the 36 MB L3 in Arrow Lake. Further up in the memory subsystem, Zen 5’s smaller 1 MB L2 can still be very effective, but often suffers more misses than Lion Cove’s larger L2. AMD has to rely a little more on their L3, which does have better performance than Intel’s.

I also tracked traffic at the Ryzen 7 9800X3D’s memory controllers. I tried to gather data at the IO die side of the die-to-die interface (CCMs), but couldn’t get Data Fabric performance events figured out for write traffic. Unified Memory Controller (UMC) data should be adequate anyway because I have the iGPU disabled and should have very little IO traffic while running Geekbench 6 workloads. The Ryzen 7 9800X3D overall has lower DRAM traffic than on the 285K, likely thanks to the former’s larger L3. Generally though, the pattern is similar. Photo Filter stands out as a bandwidth heavy but prefetch friendly workload. HTML5 Browser and Horizon Detection slot into the same category.

Score Calculation

Both Geekbench 6 and SPEC CPU2017 scores are given in speedup relative to a reference system. Geekbench 6’s reference system is a Dell Precision 3460 with a Core i7-12700, which is set to a baseline score of 2500. SPEC CPU2017’s reference system is a Sun Fire V490 with 2.1 GHz UltraSPARC-IV+ processors, which is set to a score of 1.

I find SPEC CPU2017’s score easier to interpret because it’s simply a speedup ratio, while Geekbench 6’s score takes a bit more math to get there. On the flip side, Geekbench 6’s reference system is more modern and relevant. The Sun Fire V490 dates back to 2009, and already performed poorly compared to systems from just a few years later. A Sandy Bridge or Bulldozer system would provide a far better reference point.

Geekbench 6’s reference system also has strong vector execution capabilities, assuming the workloads mostly ran on the 12700’s P-Cores. Geekbench 6’s workloads tend to be vector heavy, so the baseline sets rather demanding expectations. Skymont has weak vector execution despite improving on prior E-Cores, and often falls below the 2500 score baseline. The same applies to Arm’s Neoverse N1 and N2, though those cores also fall behind because they’re optimized for lower performance targets in general.

Final Words

Geekbench 6 is a vector-heavy suite that emphasizes core throughput. Many workloads have small instruction footprints, enjoy good branch prediction accuracy, and are prefetcher friendly. There are exceptions of course. Navigation stands out as one of the only low IPC workloads, bound by branch mispredicts. Clang does run into some op cache misses, though the larger op cache on Zen 5 still does very well, and other factors seem to hold back its IPC before Zen 5’s per-thread, 4-wide decoders present a limitation.

SPEC CPU2017 shows some of the same characteristics, with a large number of workloads that don’t pressure the memory subsystem as much as games do. While SPEC captures a broader range of IPC-related challenges, its emphasis on portability means it doesn’t stress vector execution as much as Geekbench 6 can. Both suites ultimately serve different purposes, and neither can be used in place of the other. Going forward, I expect both SPEC and Primate Labs to continue evaluating application behavior, and update their workloads as typical application behavior evolves.

Bus locks are problematic because they’re slow, and taking a bus lock “potentially disrupts performance on other cores and brings the whole system to its knees”. AMD and Intel’s newer cores can trap split locks, letting the kernel easily detect processes that use split locks and potentially mitigate that noisy neighbor effect. Linux defaults to using this feature and inserting an artificial delay to mitigate the performance impact.

Testing Split Locks

I have a core to core latency test that bounces an incrementing counter between cores using _InterlockedCompareExchange64. That compiles to lock cmpxchg on x86-64, which is an atomic test and set operation. I normally target a value at the start of a 64B aligned block of memory, but here I’m modifying it to push the targeted value’s start address to just before the end of the cache line. Doing so places some bytes of the targeted 8B (64-bit) value on the first cache line, and the rest on the next one. As expected, “core to core latency” with split locks range from bad to horrifying.

To assess the potential disruption from split locks, I ran memory latency and bandwidth microbenchmarks on cores excluded from the core to core latency test. Besides microbenchmarks, I ran Geekbench 6’s photo filter and asset compression workloads. The photo filter workload generates a lot of cache miss traffic, while asset compression tends to be the opposite. Many recent CPUs only achieve their highest clock speeds with two or fewer cores active. One core will be loaded by the workload being tested for contention effects, and another pair will be used for the core to core latency test. I therefore turned off boost or lowered clock speeds on some of the tested hardware to reduce noisy neighbor effects from clock speed variation, helping isolate the effects of split locks.

Intel Core Ultra 9 285K

Intel’s Arrow Lake gets to be the first victim. Normal core to core latency results look like this:

Split locks send latency to 7 microseconds, which remains mostly constant across different core types.

On Arrow Lake, split locks only affect L2 misses. It’s close to a “bus lock” in the traditional sense because it affects the first level in the memory hierarchy shared by all CPU cores. In theory a program can be completely unaffected by split locks as long as it keeps hitting in L2 or faster caches.

Past L2, split lock contention roughly halves memory access performance. Curiously, the 4 MB L2 caches shared across quad-core E-Core clusters aren’t affected, even if split locks are being looped from a pair of cores within the same cluster.

Split locks heavily impact GB6’s photo filter workload. Asset compression also takes a hit, but gets away relatively unscathed.

AMD Ryzen 9 9900X

Zen 5 has better split lock latency than Arrow Lake, though 500 ns is still bad in an absolute sense. As on Arrow Lake, core cluster boundaries don’t affect split lock performance.

Split locks trash everything beyond L1D. L2 and L3 performance regresses by a factor of ten. Zen 5 suffers severe penalties from split locks compared to Arrow Lake, though this situation is certainly a very contrived and unrealistic one.

Both Geekbench 6 workloads take heavy performance regressions under split lock contention. Asset Compression doesn’t generate a lot of L3 miss traffic, but even a L1D miss is very costly with split locks being looped on Zen 5.

Intel Core i7-1265U

Alder Lake uses a hybrid core setup with Golden Cove P-Cores and Gracemont E-Cores. Split locks perform horribly, and also invert the latency picture compared to intra-cacheline locks. Bouncing a cache line between P-Cores normally happens with less latency than on the E-Cores. Prior to Arrow Lake, Intel had to take a trip through the ring bus even when bouncing a line between E-Cores in the same cluster.

With split locks, P-Cores suffer agonizingly poor latency. Split locks between P and E-Cores have just above 7 microseconds of latency, matching Arrow Lake. E-Cores have the best split lock latency.

Alder Lake’s memory subsystem tolerates split locks well. L3 performance only takes a modest hit. DRAM latency goes up with split locks come into play, but not by a large amount.

Geekbench 6’s photo filter and asset compression workloads barely show any performance loss, as foreshadowed by microbenchmark results. Alder Lake might have particularly poor split lock performance, but does an excellent job of insulating other applications from their effects.

AMD Ryzen 9 3950X

Zen 2’s split lock latency lands above Zen 5’s, though it’s still better than what Intel’s newer architectures achieve. Latencies remain constant regardless of cluster boundaries, mirroring behavior seen with Zen 5 and Arrow Lake.

As on Zen 5, split locks have a devastating effect on any L1 miss. Bandwidth and latency regress by around 10x with the core to core latency test spamming split locks.

Curiously, L3 latency takes another hit after 2 MB if split locks are being looped on the same CCX. Zen 5 didn’t have an extra inflection point under split lock contention even within the same CCX.

As on Zen 5, both Geekbench 6 workloads suffer heavily under split lock contention.

Intel Core i5-6600K

Intel’s older Skylake architecture actually has better split lock latency than Arrow Lake and Alder Lake. It’s a bit worse than Zen 2, but doesn’t reach into the microsecond range.

Skylake also takes split lock contention penalties for L2 misses, but not for L2 or L1 hits.

Both tested Geekbench 6 workloads show measurable performance impact from split locks. Photo filter loses 34.24% of its performance, while asset compression takes a lower 16.2% loss.

AMD FX-8350

AMD’s Piledriver has high core to core latency results, but remarkably has the best results for split locks. Split lock latency is double to triple the latency of intra-cacheline locks, but that’s worlds better than on newer platforms.

Split locks don’t affect cache hits. Remarkably, that extends to the shared L3 cache. DRAM performance does get hit, with latency doubled and bandwidth cut by more than half.

Microbenchmark results translate to higher level workloads. Geekbench 6’s photo filter and asset compression workloads both get away with only minor performance impact. The hit to DRAM performance certainly hurts, but a Piledriver core has 10 MB of cache completely unaffected by split locks. No other hardware tested here can field that much split-lock-unaffected cache capacity.

Intel Celeron J4125

Goldmont Plus has high split lock latency, though it’s still curiously better than Intel’s more modern Arrow Lake and Alder Lake.

Just like Arrow Lake’s E-Cores, Goldmont Plus can hit L2 and be completely unaffected by split locks. DRAM bandwidth does take a substantial drop, but latency only regresses slightly. It’s worth noting that Goldmont Plus starts with very poor baseline DRAM latency though.

Slight differences in L1D and L2 performance can be attributed to lower clock speeds, since I didn’t underclock the chip.

Both Geekbench 6 workloads show performance regressions, but it’s not much considering there’s also multi-core clock speed drops in the mix. Much of the score decrease for asset compression can probably be attributed to lower clocks. The photo filter workload definitely suffers, but not to the same extent as Arrow Lake or AMD’s Zen 2 or Zen 5.

What’s a Bus Lock, Really?

The term “bus lock” dates back to early multiprocessor systems, which placed multiple CPUs on a shared bus. Each CPU had its bus pins connected to the same motherboard traces, which connected them to the chipset. Locking the bus prevents other CPUs from making memory accesses, preventing them from interfering with atomic operations. Modern CPUs no longer use a shared bus, and instead use non-blocking, distributed interconnect setups. It’s not clear what a “bus lock” really means. And while Intel and AMD still use the terminology, “bus locks” caused by split locks clearly have a range of effects depending on the hardware in question.

I suspect Intel’s bus locks are implemented in their IDI (in-die interconnect) protocol. P-Cores and E-Core use IDI to talk to the uncore, which starts with the ring bus on most Intel client designs. Goldmont Plus’s performance monitoring documentation curiously indicates that a “bus lock” is a special request to the L2 controller, though L2 hits aren’t impacted just like on other Intel designs.

AMD’s implementation goes beyond a traditional bus lock in Zen 2 and Zen 5, with core-private L2 caches impacted as well. One possibility is that AMD falls back to the Infinity Fabric layer when handling split locks, though I don’t have strong evidence to support that. Performance monitoring events increment at the Data Fabric’s Coherent Stations during core to core latency test runs. But if they’re responsible, they only handle control path traffic because the increments aren’t proportional to L2 hit traffic observed when also running a memory bandwidth microbenchmark.

Piledriver also throws a wrench into the works when trying to define bus locks. Maybe AMD’s old architecture doesn’t use bus locks at all, and is able to use its cache coherency protocol to let unrelated accesses proceed as long as they hit cache.

Overall, there’s no good analogy to the classical shared bus on a modern CPU. Perhaps CPU designers should drop the term “bus lock” from documentation, and precisely document how a split lock impacts their systems.

Linux’s Split Lock Mitigation

Linux’s split lock mitigation traps split locks and introduces millisecond level delays, aiming to both make split locks “annoying” and provide better quality of service to other applications.

Testing on Linux without touching sysctl settings shows latency comparable to mechanical hard drive seek time, which is an eternity for a CPU. This kind of delay should also eliminate the noisy neighbor effects above, since the split lock core to core latency test simply isn’t running most of the time.

I think Linux’s default makes sense for mutli-user or server systems, which value consistent performance when running a large number of tasks. Limiting noisy neighbor effects from a rogue application, like my test code above that spams split locks, is a sensible option. I can see it being used alongside other QoS mechanisms like running at lower, locked clock speeds, partitioning caches, and throttling memory bandwidth usage to ensure consistency.

For consumer systems though, Linux’s default feels like an overreaction to a problem that didn’t exist. A core-to-core latency test uses locks at a rate no normal application would. A split lock variant of that test is even more ridiculous. Games have apparently been using split locks for quite a while, and have not created issues even on AMD’s Zen 2 and Zen 5. However, a game that runs at 10 FPS on Linux and 200 FPS on Windows is a problem for Linux.

Linux traditionally struggled on the desktop scene because it required a high level of technical skill and substantial troubleshooting time from users. Linux distros have made tremendous progress in chipping away at those requirements over the past two decades. But doing a scream test on users - that is, attempting to impose a vision of the “proper” way to do things by creating an artificial performance problem - is a step backward. Ease of use is everything in the consumer world. Every OS problem that a user has to troubleshoot is a failure on the OS’s part. Going forward, I hope Linux dodges avoidable problems like this.

Final Words

Split locks don’t stop other cores from running code. They’re not the hardware equivalent of Python’s global interpreter lock, or another similar construct that blocks concurrency. On modern CPUs, split locks don’t even block all memory accesses. They only introduce a performance penalty when those memory accesses miss a certain cache level. That performance penalty varies wildly too.

Applications tend to suffer varying degrees depending on how often they miss cache. Hardware has a huge influence too. AMD’s Piledriver and Intel’s Alder Lake do the best job of minimizing noisy neighbor effects. Piledriver is especially impressive because it manages that while delivering the best split lock latency across all hardware tested here. On the other hand, AMD’s Zen 2 and Zen 5 suffer heavily in this corner case. Another application could slow down so much that you’d be forgiven for thinking you got a version of it compiled using Claude’s C Compiler.

From these results, programmers should obviously strive to avoid split locks. They perform poorly, and have a heavier effect on other applications than intra-cacheline locks. From the hardware side, there’s clearly room to optimize split locks for better performance and reduced noisy neighbor effects. I hope to see both hardware and software developers take a measured, data-driven approach to tackling the split lock problem, and one that doesn’t involve introducing new performance problems.

Extras

Zen 5 in the Ryzen AI MAX+ 395 (Strix Halo) has higher split lock latency than its desktop Zen 5 counterparts. I suspect split locks are handled at Infinity Fabric’s Coherent Station blocks, but curiously those counters also increment for intra-cacheline locks that are handled within a CCX. Perhaps the Coherent Station gets some sort of notification about a line changing state.

Zen 4 in a high performance mobile implementation shows similar split lock latency to Zen 2.

Arrow Lake E-Cores have a lot of monitoring sophistication around bus locks. Under the split lock core to core latency test, any E-Core involved gets blocked for nearly all cycles. However, other E-Cores are only blocked around half the time. That would explain the roughly 50% L3/DRAM performance degradation, though it does not explain how L2 performance remains unscathed.

I took rough notes on die to die traffic when running Geekbench 6 workloads on Arrow Lake by observing performance counters sampled at 1 second interval and picking out spikes. My goal isn’t to give an accurate figure for average L3 miss traffic per workload, but rather to select two interesting workloads for measuring split lock impact.

Our society has gone all-in on AI. Software and related technologies represent one of the biggest sectors of the US economy, and technology companies have all but bet the farm on LLMs being able to do everything under the sun. By the law of sunk costs*, AI investments will increase without bound until that objective is achieved. AGI is therefore inevitable, and everyone must fully commit to ensuring AI’s success because things will get more painful the longer it takes. When bank money runs out, the government will bail them out. When government funds are exhausted, aliens will get involved, followed by various deities. Please don’t ask what happens after that. Anyway, it’s in everyone’s best interest to embrace the AI revolution with absolute conviction. Therefore, I’m excited to announce that Chips and Cheese will stand at the vanguard of the AI revolution by adopting Claude’s AI-generated compiler, CCC.

CCC, or Claudes’ C Compiler, is a “from-scratch optimizing compiler with no dependencies […] able to compile the Linux kernel.” Anthropic used their Claude large language model (LLM) to write the entire compiler. Human effort was relegated to prompting and maintaining AI coding agents, and standing up a very robust test suite to keep those coding agents on track. Past Chips and Cheese articles used microbenchmarks and benchmarks built using GCC. GCC was developed by humans, which is backwards and reactionary by the standards of many cadres today. Comparing GCC and CCC should be a powerful showcase of revolutionary software development practices.

Compiling a Microbenchmark

Microbenchmarking is a useful tool for understanding low level hardware characteristics like cache and memory latency. My first shot at testing load latency used a simple loop of dependent array accesses in C:

A = A[current] creates a dependency chain where one array access result feeds into the next, preventing accesses from overlapping even on an out-of-order core. Dividing access time by iteration count gives latency. CCC and GCC compile the loop into the following assembly code:

CCC generates more code from the same high level source than GCC does, and sometimes, more is better. CCC’s code is more ideologically correct from a RISC standpoint. GCC uses indexed addressing modes that express a dependent shift, add, and load sequence in a single instruction. RISC advocates for a small set of simple instructions and larger code sizes, and indexed addressing modes are quite complex. CCC follows the RISC philosophy, and splits the array access into separate shift, add, and load instructions.

CCC doesn’t stop there and exceeds quotas by decomposing current = A[current] into a nine instruction dependency chain. On x86-64, it shuffles the index through several registers and sends it on a trip through the stack. It does the same on aarch64, but at least keeps the index in registers. On both architectures, CCC brings the array base address in from the stack, and sends it on another round trip through the stack for good measure before using it.

Compiling the microbenchmark with CCC results in higher observed array access latency. Zen 5 and Lion Cove need two extra cycles, likely from the shift + add dependency chain. Exceptionally strong move elimination and zero-latency store forwarding seem to let Intel and AMD collapse the nine-instruction dependency chain. After that, their out-of-order core width absorbs the additional instructions. Arm’s recent cores have less robust move elimination and can’t do zero latency store forwarding. They suffer a 6-7 cycle penalty.

Smaller cores with narrower execution engines struggle with CCC-compiled code, likely because they don’t have enough core throughput to absorb loop overhead and the sum += current sink statement. CCC also compiles those into very long assembly sequences, placing pressure on core throughput.

In-order cores suffer the largest penalties. They’re not embracing the AI revolution with adequate enthusiasm, and should seriously reconsider their stance or risk getting denounced.

Challenging the Old Type Based Order

Old, traditional compilers force the masses to follow a tight set of rules dreamt up by the bourgeoisie. If a programmer doesn’t follow these rules, the compiler will complain and may even refuse to complete the compilation process. Warnings and errors are of no use to the proletariat, who just want results. CCC boldly crushes type systems with revolutionary fervor, as if to shout “out with the old, in with the new!” Take the following code:

GCC cowers behind error messages with weak excuses about incompatible types and fails to produce a binary. In doing so, GCC engages in counter-revolutionary wrecking by undermining the work of the masses, who have to waste time fixing errors.

Rather than make excuses, CCC raises the revolutionary banner high and produces an executable.

Performance: SPEC CPU2017

Claude’s C Compiler only compiles C code. All other languages are therefore invalid. Eight of SPEC CPU2017’s workloads are C-only. I tested them on the following cores:

• Arm Cortex X925 in Nvidia’s GB10

• AMD Zen 5 in the Ryzen 7 9800X3D, with boost disabled

• Intel Lion Cove in the Core Ultra 9 285K

I ran with boost disabled on the 9800X3D because running CCC-built workloads takes a lot of time. Extra time spent in high performance states mean more power consumption, and we should save every last bit of power possible to feed AI applications.

CCC’s version of 502.gcc consistently crashed with a segmentation fault on Cortex X925, even though it managed to complete some runs on x86-64. When it ran to completion on x86-64, CCC’s 502.gcc delivered 23.6% and 27.1% of the performance of its GCC-compiled counterpart on Lion Cove and Zen 5, respectively. CCC hit its relative performance peak on 505.mcf, where it managed less than a 35% regression versus GCC. Across the eight tested SPEC CPU2017 workloads, using CCC on average regressed performance by somewhere above 70%.

Arm’s Cortex X925 suffers hardest from the CCC switch. With GCC, Arm’s core was very much a match for Zen 5 and Lion Cove. Now, it loses in every workload except for 500.perlbench, where Zen 5 takes a dive to let X925 slip ahead. But X925 still can’t compare to Lion Cove in that workload.

In turn, Lion Cove can’t compare to Sun’s UltraSPARC IV+. SPEC CPU2017 scores are speedup ratios relative to a reference machine, which is “a historical Sun Microsystems server, the Sun Fire V490 with 2100 MHz UltraSPARC-IV+”. Lion Cove may have the best 500.perlbench score of the three cores tested here at 0.76, but that’s below 1. A vibe-coded compiler is the product of the latest software development techniques and can’t be at fault. It therefore follows that Sun’s UltraSPARC-IV+ is amazing. The UltraSPARC-IV+ is a dual core version of the UltraSPARC III, which uses a 4-wide in-order pipeline. Lion Cove is an 8-wide out-of-order design with massive reordering capacity and much higher clock speeds, but clearly none of that is necessary to win.

Performance is overrated. Analysts today agree that IPC is more important than performance, and CCC code achieves excellent IPC. On 525.x264, Arrow Lake averages 6.09 IPC when running CCC code. That’s the highest IPC average across all SPEC CPU2017 runs above. GCC’s best IPC average tops out at a wimpy 5.5. CCC’s IPC advantage isn’t universal though, and GCC does manage higher IPC on 538.imagick.

As foreshadowed with the cache and memory latency microbenchmark, CCC takes a great leap forward in instruction counts across all eight SPEC CPU2017 workloads. Some workloads execute more than 10x as many instructions.

A Top-Down View

Hardware performance monitoring reveals a different set of challenges for the CPU pipeline when comparing GCC- and CCC-compiled code. Branch prediction is less important because branch count didn’t grow proportionately to total instruction count. SPEC CPU2017’s three C-only floating point workloads become very core-bound when built with CCC. On Zen 5, that means retirement is blocked by an instruction that doesn’t read from memory. That indicates performance is limited by how fast the core can execute instructions, rather than how fast the memory subsystem can feed the core with data. The five integer workloads become less frontend bound, and face steeper backend challenges. CCC workloads across the board achieve higher core utilization than their GCC-compiled counterparts, as the IPC figures from before suggest.

500.perlbench is a catastrophically bad case for Zen 5 when compiled with CCC because of a much lower op cache hitrate. Zen 5 primarily feeds itself with a 6K entry, 16-way op cache, which can fully feed the 8-wide rename/allocate stage. 4-wide, per-thread decoders handle larger instruction footprints. Typically applications with poor code locality suffer from other IPC limiters besides frontend throughput, especially when the core isn’t running a second thread that can hide latency. 500.perlbench has higher IPC potential if not for the per-thread decode throughput limitation, putting Zen 5 behind X925 and very far behind Lion Cove. It’s the first case I’ve seen outside a microbenchmark where Zen 5’s decoder arrangement comes out to bite. Zen 5’s op cache manages high coverage in other workloads, so 500.perlbench is an exceptional case.

CCC challenges Lion Cove’s pipeline in a similar fashion, though Lion Cove is generally more backend bound than Zen 5. This could be because Lion Cove sits atop a higher latency uncore. Or, the difference could be down to accounting methodology. For example, Intel might be counting cycles where any instruction is delayed by a load as “memory bound”, which could lead to higher counts than AMD’s approach. Lion Cove’s 8-wide decoder excels in 500.perlbench, where frontend limitations are minor compared to what they were in Zen 5.

Arm’s Cortex X925 is very core-bound when running CCC code, and not just in the three FP workloads. I’m distinguishing between core and memory bound backend bound slots by looking at the proportion of backend bound cycles covered by the STALL_BACKEND_MEMBOUND event, which counts when there’s a backend stall and the “backend interface to memory is busy or stalled”.

I partially attribute the core-bound slot increase to Cortex X925 not having the robust move elimination and zero-latency store forwarding that Lion Cove and Zen 5 do. Long chains of register-to-register MOVs or frequent trips through the stack are rare in typical code. As evidence, Intel's Ice Lake only took a 1-3% performance loss with move elimination disabled. Move elimination therefore acts as a cherry on top in most applications, with most performance coming from out-of-order execution basics and an efficient cache hierarchy to feed the core. Of course, that changes with CCC.

519.lbm’s large increase in backend memory bound slots looks related to store forwarding latency. A hot section in CCC-compiled 519.lbm repeatedly spills register x0 to the stack, only to reload it several instructions later, making sure the memory system contributes to all phases of code execution. Store forwarding latency forms part of a long dependency chain, and X925 can’t do zero-latency forwarding the way Zen 5 and Lion Cove can.

Final Words

Compilers are crucial to modern software development, and performance of compiled code is key to making high level languages practical. Switching to CCC causes a 70%+ performance degradation across C-only SPEC CPU2017 workloads. Most consumers will readily accept this, because a fast core like Intel’s Lion Cove still does well when running CCC code. As evidence, AMD’s Bulldozer architecture fascinated enthusiasts about 15 years ago. Bulldozer is 36.8% faster than Lion Cove, the fastest CPU tested here for running CCC code. Simply overclocking Lion Cove to 8 GHz could close much of the performance gap.

Of course, some provocateurs may only reach 7.5 GHz, undoubtedly due to a combination of personal skill issues and insufficient enthusiasm for the AI revolution. Severe agitators may theorize that Bulldozer’s level of single threaded performance is not top-notch in some scenarios, and demand ever more performance. To prevent these people from discrediting the revolution, the stock performance problem must be addressed.

Performance problems can be addressed from either hardware or software, because hardware and software optimization go hand-in-hand. Well optimized software performs well across a wide variety of hardware. Likewise, well designed hardware delivers good performance across a wide range of software. CCC-generated code dramatically shifts what software demands from hardware, upsetting the existing social contract between software and hardware. Faulting the software side would undermine the crowning achievements of the AI revolution. Therefore, hardware must boldly tackle the performance question.

Completely addressing the performance gap would be easy by my estimate. With 20-30 GHz clock speeds and several generations worth of architectural improvements, CCC code could reach the performance achieved by GCC-compiled binaries on current generation hardware. These targets might have been dismissed in the past. But they’re achievable now because the AI revolution isn’t just any revolution. Rather, it is a cultural revolution that upends views regarding power consumption. Infinite power can make many previously unthinkable designs a reality. Some hardware designers may see red flags in this approach and persistently complain. But proper hardware engineers will rally around those flags and chant slogans. By correctly following the path of the revolution, they’ll be sure to succeed and may even exceed their quotas. May the odds be ever in their favor.

Footnotes

* The proof is trivial and similar to the proof for the gambler’s law (99% of gamblers quit just before they win big)

Testing the Chipset

I’m using Nemes’s Vulkan-based GPU benchmark suite here, but with some modifications that let me allocate memory with the VK_MEMORY_PROPERTY_HOST_COHERENT_BIT set, and the VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT unset. That gives me a buffer allocated out of host memory. Then, I can run the Vulkan latency test against host memory, giving me GPU memory access latency over PCIe. The GPU in play doesn’t matter here, because I’m concerned with latency differences between using CPU PCIe slots and the southbridge ones. I’m using Nvidia’s T1000 because it’s a single slot card and can fit into the bottom PCIe slot in my various systems. All of the systems below run Windows 10, except for the FX-8350 system, which I have running Windows Server 2016. I’m using the latest drivers available for those operating systems, which would be 553.50 on Windows 10 and 475.14 on Windows Server 2016.

I also tried testing in the other direction by having the CPU access GPU device memory. Nemes actually does that for her Vulkan uplink latency test. However, I hit unexplainable inflection points when testing across a range of test sizes. Windows performance metrics indicated a ton of page faults at larger test sizes, perhaps hinting at driver intervention. Investigating that is a rabbit hole I don’t want to go down, and the CPU to VRAM access behavior I observed with the T1000 didn’t apply in the same way across different GPUs. I’ll therefore be focusing on GPU to host memory access latency, though I’ll leave a footnote at the end for CPU-side results.

Cache Bandwidth and Probes

Setting the VK_MEMORY_PROPERTY_HOST_COHERENT_BIT means writes can be made visible between the CPU and GPU without explicit cache flush and invalidate commands. With the T1000, this has the strange side effect of causing a lot of probe traffic even when the GPU hits its own caches. GPU cache hit bandwidth can be constrained depending on platform configuration, possibly depending on probe throughput. AMD’s old Piledriver CPU has northbridge performance events that can track probe results. These events show a flood of probes when the test goes through small sizes that fit within the T1000’s caches. Probe activity dips when the test reaches larger sizes and bandwidth is constrained by the PCIe link.

There’s a lot of mysteries here. Probes work at cacheline granularity, and AMD’s 15h BKDG makes that clear. But probe rate suggests the T1000 is initiating a probe for every 512 bytes of cache hits, not every 64. Another mystery is why cache hit latency appears unaffected.

Mysteries aside, I’ll be dropping a note when GPU cache hit bandwidth appears to be affected by the platform. But I won’t comment on GPU cache miss bandwidth, which will obviously be limited by PCIe bandwidth. The T1000 has a x16 PCIe Gen 3 interface to the host, and the newest platforms I test here can support PCIe Gen 4 or Gen 5. I don’t have a single slot card with PCIe Gen 4 or Gen 5 support.

AMD’s AM5 Platform

AMD’s latest desktop platform is built around the PROM21 chip. PROM21 has a PCIe 4.0 x4 uplink, and exposes a variety of downstream IO including more PCIe 4.0 links. AMD’s X670E chipset chains two PROM21 chips together, basically giving the platform two southbridge chips for better connectivity. Gigabyte’s X670E Aorus Master has three PCIe slots. One connects to the CPU, while the two others connect to the lower PROM21 chip. The PROM21 chip connected to the CPU isn’t hooked up to any PCIe slots. Instead, it’s only used to provide M.2 slots and some USB ports. The B650 chipset, tested here on the ASRock B650 PG Lightning, only uses one PROM21 chip and connects it to PCIe slots.

Using the CPU’s PCIe lanes gives about 650 ns of baseline latency. Going through one PROM21 chip on the B650 chipset brings latency to 1221 ns, or 569.7 ns over the baseline. On X670E, hopping over two PROM21 chips brings latency to 921.3 ns over the baseline. Going over the chipset therefore comes with a hefty latency penalty. Going through an extra PROM21 chip further increases latency.

I saw no difference in GPU cache hit bandwidth with the T1000 connected to Zen 5’s CPU lanes. Switching the GPU to the chipset lanes dropped cache hit bandwidth to just above 25 GB/s, and that figure remains similar regardless of whether one or two PROM21 chips sat between the CPU and GPU.

Intel’s Arrow Lake Platform

Arrow Lake chipsets follow the same topology as Intel chipsets going back to Sandy Bridge. A Platform Controller Hub (PCH) serves the southbridge role. The PCH connects to the CPU via a Direct Media Interface (DMI), which is a PCIe-like interface. Z890 uses DMI Gen 4, which can use transfer speeds of up to 16 GT/s with eight lanes. I’m testing on the MSI PRO Z890-A WIFI, which provides three PCIe slots. The first is a x16 slot connected to the CPU, while the two others are x4 slots connected to the PCH.

Baseline latency on Arrow Lake is higher than on Zen 5, with the GPU seeing 785 ns of load-to-use latency from host memory. Going through the PCH adds approximately 550 ns of latency, which is in line with what one PROM21 chip adds on the AM5 platform.

I’m going to skip commenting about the T1000’s cache hit bandwidth on Arrow Lake. It does not behave as expected, even when attached to the CPU’s PCIe lanes. The T1000’s cache hit behavior also differs from that of other GPUs. For example, Intel’s Arc B580 maintains full cache hit bandwidth when attached to the Arrow Lake platform, though curiously it can’t cache host memory in its L2.

Intel’s Skylake Platform

Skylake’s Z170 chipset uses a familiar southbridge-only topology like Z890, but with less connectivity and lower bandwidth. I’m testing on the MSI Z170 Gaming Pro Carbon, which connects the top PCIe x16 slot to the CPU. A middle slot also connects to the CPU to allow splitting the CPU’s 16 PCIe lanes into two sets of eight. The bottom slot connects to the PCH.

Skylake’s CPU PCIe lanes provide excellent baseline latency at 535.59 ns, while going through the Z170 PCH adds 338 ns of latency. That’s high compared to regular DRAM access latency, but compares well against current generation desktop platforms.

L1 bandwidth on the T1000 isn’t affected when the GPU is attached to CPU PCIe lanes, though L2 hit bandwidth is strangely lower (though still well above VRAM bandwidth). Switching the card to the Z170’s PCH lanes reduces cache hit bandwidth to just over 51 GB/s.

AMD’s AM3+ Platform

AMD’s Bulldozer and Piledriver chipsets use an older setup where the CPU has an integrated memory controller, but the chipset is still responsible for all PCIe connectivity. I’m testing on the ASUS M5A99X EVO R2.0, which implements the 990X chipset. The RD980 (990X) northbridge connects to the CPU via a 5.3 GT/s HyperTransport link, which I assume is running in 16-bit wide “ganged” mode. Two PCIe slots on the M5A99X connect to the RD980, and can run in either single x16 or 2x8 mode. A x4 A-Link Express III interface connects the northbridge to the SB950 southbridge, which provides more PCIe lanes and slower IO. A-Link Express is based on PCIe Gen 2, and runs at 5 GT/s.

Baseline latency is very acceptable at 769.67 ns when using the 990X northbridge’s PCIe lanes, which puts it between AMD’s Zen 5 and Intel’s Arrow Lake platforms. That’s a very good latency figure considering the AM3+ platform’s fastest PCIe lanes don’t come directly from the CPU like on modern platforms.

Going through the SB950 southbridge increases latency by 602 ns. It’s a higher latency increase compared to the PCH on Arrow Lake or one PROM21 chip on B650. However, southbridge latency is still better than going through two PROM21 chips on X670E.

On the 990X platform, the T1000’s cache hit bandwidth is capped at around 132 GB/s when connected using the northbridge PCIe lanes. With the southbridge PCIe lanes, cache hit bandwidth drops to just over 20 GB/s. Both figures are well in excess of the platform’s IO bandwidth. The 990X’s HyperTransport lanes can only deliver 10.5 GB/s in each direction, and cache hit bandwidth far exceeds that even when the T100 is attached to the southbridge. If the T1000’s cache hit bandwidth is limited by probe throughput, then the 990X chipset has 10x more probe throughput than is necessary to saturate its IO bandwidth. Of course, that’s nowhere near enough to cover GPU cache hit bandwidth.

Sidebar: Latency from the CPU side

As mentioned before, having the CPU access GPU VRAM is another way to measure PCIe latency. Host-visible device memory allocated on the T1000 is uncacheable from the CPU side, likely to ensure coherency. If caches aren’t in play, then small test sizes provide the best estimate of baseline latency because they don’t run into address translation penalties or page faults. Here, I’m plotting CPU-to-VRAM results at the 4 KB test size next to the GPU-to-system-memory results above. There’s some difference between the two measurement methods, but not enough to paint a different picture.

Final Words

Chipsets today don’t take on performance critical roles, and accordingly aren’t optimized with low latency in mind. Going over chipset PCIe lanes adds hundreds of nanoseconds of latency, and imposes bandwidth restrictions. Chipsets can affect cache hit bandwidth too, possibly due to probe throughput constraints.

Older chipsets like AMD’s 990X show it’s possible to deliver reasonable PCIe latency even with an external northbridge. But I don’t see latency optimizations coming into play on future chipsets. With the decline of multi-GPU setups, chipsets today largely serve high latency IO like SSDs and network adapters. Those devices have latency in the microsecond range, if not milliseconds. A few hundred nanoseconds won’t make a difference. The same applies for probe performance. Accessing host-coherent memory is a corner case for GPUs, which expect to mostly work out of fast on-device memory. Needless to say, cache hits for a VRAM-backed buffer won’t require sending probes to the CPU.

I expect future chipset designs to optimize for reduced cost and better connectivity options, while keeping pace with higher bandwidth demands from newer SSDs. I don’t expect latency or probe path optimizations, though it will be interesting to keep an eye out to see if behavior changes.

Unlike AMD’s Strix Halo, Nvidia’s GB10 focuses on AI applications. Nvidia’s proprietary CUDA ecosystem is key, because CUDA is nearly the only name in the GPU compute game. GPU compute applications are optimized first and foremost for CUDA and Nvidia GPUs. Everything else is an afterthought, if a thought at all.Combining significant GPU compute power with iGPU advantages and CUDA support gives GB10 a lot of potential. Of course, hardware has to deliver high performance for those ecosystem advantages to show. Achieving high performance involves building a balanced memory hierarchy, as well as tailoring hardware architecture to do well in targeted applications.

Cache and Memory Access

GB10’s GPU-side memory subsystem brings few surprises. It’s basically a Blackwell GPU integrated, with a familiar two-level caching setup. The L2 serves as both a high capacity last level cache and the first stop for L1 misses. AMD in contrast uses more cache levels and gradually increasing cache capacity at each level, which mirrors their discrete GPUs too. In a cache and memory latency test, GB10 and Strix Halo trade blows depending on test size. Larger caches are hard to optimize for high performance, so AMD wins when it’s able to hit in its smaller 256 KB L1 or 2 MB L2 compared to Nvidia’s higher latency 24 MB L2. Nvidia wins when GB10 can keep accesses within L2 while AMD has to service them out of Strix Halo’s 32 MB memory side cache. Both Strix Halo and GB10 use LPDDR5X memory. Strix Halo’s iGPU enjoys better latency to LPDDR5X than Nvidia’s GB10, reversing the situation seen on the CPU side.

Nvidia’s L1 cache offers an impressive combination of low latency and high capacity. With dependent array accesses, it can return data nearly as fast as AMD’s 16 KB scalar cache while offering more capacity than AMD’s first level scalar and vector caches combined. However, dependent array accesses don’t tell the whole story. Pointer dereferences indicate that AMD takes a lot of overhead from array addressing calculations. Cutting that out shows that RDNA3.5’s scalar cache offers very low latency. Nvidia already had efficient array address generation, so switching to pointer dereferences makes less of a difference on GB10. Regardless of access method, GB10’s L1 offers higher capacity and lower latency than RDNA3.5’s L0 vector cache.

GB10’s system level cache is invisible in the latency plot above because it’s smaller than the GPU’s 24 MB L2. It also doesn’t seem to maintain a mostly exclusive relationship with the GPU L2, so its capacity isn’t additive on top of the L2. Nvidia’s slides say the SLC aims to enable "power efficient data-sharing between engines", implying that feeding compute isn't its primary purpose.

Like many modern GPUs, GB10 supports OpenCL’s Shared Virtual Memory (SVM) feature. SVM allows seamless pointer sharing by mapping a buffer into the same virtual address space on both the CPU and GPU. GB10 supports “coarse grained” SVM, meaning the buffer cannot be simultaneously mapped on both the CPU and GPU. Some GPUs with only coarse grained SVM appear to copy the entire buffer under the hood, as can be seen with Qualcomm and Arm iGPUs. Thankfully, GB10 can handle this feature without requiring a full buffer copy under the hood.

Compared to Strix Halo, GB10’s memory subsystem strategy moves high capacity caching up a level. Strix Halo’s “Infinity Cache” setup was awkward because it placed a GPU-focused cache at the other end of a system level interconnect. That places high pressure on Infinity Fabric, which AMD built to handle nearly 1 TB/s of bandwidth. GB10 in contrast uses the 24 MB graphics L2 to filter out a large portion of GPU memory traffic, so the system level interconnect primarily deals with DRAM accesses. That said, using 16 MB of cache just to optimize data sharing feels awkward too. SRAM is one of the biggest area consumers in modern chips because of how far compute has outpaced DRAM performance. A hypothetical setup that concentrates last level caching capacity in the SLC could be more area efficient, and could boost CPU performance.

Bandwidth Measurements

48 SMs mean 48 L1 cache instances, giving GB10’s iGPU impressive cache hit bandwidth. With Nemes’s Vulkan based benchmark, GB10 easily outpaces Strix Halo. Strix Halo’s iGPU can achieve around 10 TB/s using a narrowly targeted OpenCL kernel that can’t work with arbitrary test sizes, but GB10 is still ahead even with that methodology. Nvidia’s advantage continues at L2, which offers both higher bandwidth and higher capacity than AMD’s 2 MB L2. Needless to say, Nvidia’s L2 continues to compare well as test sizes increase and Strix Halo has to use its 32 MB Infinity Cache. Finally, both iGPUs achieve similar bandwidth from system memory, with GB10 taking a slight lead with faster LPDDR5X.

Both GPUs use write-through L1 caches, so writes from compute kernels go straight to L2. GB10’s L2 handles writes at half rate, and provides just under 1 TB/s of write bandwidth. Write bandwidth on Strix Halo is somewhat lower at approximately 700 GB/s. Write behavior on both iGPUs is similar to that of larger discrete GPUs.

L1 and Shared Memory Capacity

Nvidia’s GPUs use a single block of SM-private memory for both L1 cache and Shared Memory. Shared Memory is a software managed scratchpad private to a group of threads. It fills an analogous role to AMD’s Local Data Share (LDS), letting code explicitly keep data close to compute when data movement is predictable. A larger L1/Shared Memory block can let each group of threads use more scratchpad space with lower impact to occupancy or L1 cache capacity. Nvidia’s datacenter GPUs get 256 KB of L1/Shared Memory per SM. GB10 does not.

Instead, GB10 behaves much like Nvidia’s consumer Blackwell architecture and has 128 KB of L1/Shared Memory. In exchange for lower L1/Shared Memory capacity, GB10 achieves 13 ns of latency in Shared Memory using dependent array accesses. That’s slightly better than 14.84 ns for B200.

Instruction Caching

Instruction fetch behaves as expected for any Blackwell GPU. Each SM sub-partition has a 32 KB L0 instruction cache, which can fully feed a partition’s maximum throughput of one instruction per cycle. Blackwell uses fixed length 16-byte instructions, so the A 128 KB L1 instruction cache feeds all four partitions, and can also deliver one instruction per cycle. Bandwidth from the L1I is shared by all four partitions, and per-partition throughput drops to 0.5 IPC when two partitions contend for L1I access. GPU shader programs are typically small and simple compared to CPU programs, so I suspect most code fetches will hit in the 32 KB L0 instruction caches.

RDNA3.5 Workgroup Processors use a simpler instruction caching strategy, with one 32 KB instruction cache servicing four SIMDs. Bringing a second wave into play does not cause bandwidth contention at the L1I, though two waves executing different code sections do contend for L1I capacity. Running code from L2 leads to poor instruction throughput on both architectures.

Compute Performance

If Strix Halo had high compute throughput for an integrated GPU, GB10 takes things a step further. Strix Halo’s iGPU has 20 Workgroup Processors (WGPs) to GB10’s 48 SMs. These basic building blocks are the closest analogy to cores on a CPU. RDNA3.5 WGPs do have doubled execution units for basic operations and higher clocks. But even in the best case, Strix Halo’s iGPU lands a bit behind GB10’s.

Both GPUs have low FP64 throughput, another characteristic that sets them apart from datacenter ones. GB10 has a 1:64 FP64 to FP32 ratio, while Strix Halo has a 1:32 ratio.

Which Blackwell?

Nvidia’s GB10 presentation at Hot Chips 2025 started by talking about scaling their Blackwell architecture from server racks, to single GB300 GPUs, to the GB10 chip in DGX Spark. GB300, like B200, uses Nvidia’s B100 datacenter Blackwell architecture. The datacenter distinction is important. Ever since Pascal, Nvidia’s datacenter and consumer GPU architectures look quite different even when they use the same name. Cache setup and FP64 performance are often differentiating factors.

GB10’s GPU uses Blackwell’s consumer variant. It supports compute capability 12.1, putting under the same column that describes Nvidia’s RTX 5xxx consumer GPUs in Nvidia's chart. B200 in contrast has compute capability 10.0. Compared to GB10’s SMs, datacenter variants can keep more work in flight, have more L1/Shared Memory, and more FP64 units. 5th generation Tensor Core features are also limited to the datacenter variant.

These differences mean that GB10 doesn’t act like a smaller version of Nvidia’s Blackwell datacenter GPUs. Presenting GB10 as having the same architecture creates confusion, as can be seen in forum posts and GitHub issues, where kernels written for Nvidia datacenter GPUs can't run on GB10. Even if code doesn’t use any features specific to datacenter Blackwell, GB10 presents a different optimization target. Saying GB10 uses the same architecture as GB300 would be like saying Strix Halo’s RDNA3.5 iGPU uses the same architecture as MI300X. It’s great for investors, but not useful for a more technical audience where accuracy is important. I feel it would have been better to present GB10’s GPU as a member of Nvidia’s consumer RTX lineup. Consumer GPU architectures are perfectly capable of handling a wide variety of compute applications. Nvidia could still position GB10 as a compute-focused GPU.

Nvidia’s consumer Blackwell variant has advantages too. It has hardware raytracing acceleration, which can be useful in workloads like Blender with Optix. Consumer SMs clock higher than their datacenter counterparts, which can reduce per-wave latency on code that doesn’t benefit from any of datacenter Blackwell’s advantages. Finally, consumer Blackwell SMs take less area, which translates to more vector compute throughput with a given die area budget.

Some Light Benchmarking

FluidX3D simulates fluid behavior using the lattice Boltzmann method. It carries out computations using FP32, but can store data in 16-bit floating point formats to reduce pressure on memory bandwidth and capacity. FP16S mode stores values in the IEEE-754 standard 16-bit floating point format, and benefits from hardware instructions when converting between FP32 and FP16. FP16C uses a custom 16-bit floating point format to minimize accuracy loss. Conversions between the custom FP16 format and FP32 have to be done in software, increasing the compute to bandwidth ratio.

GB10 leads AMD’s Strix Halo when running FluidX3D’s delta wing simulation example with FP32, and continues to have an edge in FP16S. However, it falls behind in FP16C mode. Intel’s Arc B580 nominally sits in the same performance segment as Strix Halo, but manages a huge performance lead over both iGPUs. VRAM bandwidth is likely a factor. FluidX3D has poor memory access locality and therefore tends to be bound by VRAM bandwidth. Besides reporting simulation speed, FluidX3D outputs memory bandwidth figures that account for accesses made from its algorithm’s perspective.

These accesses can include cache hits, which are transparent to user code. Profiling FluidX3D on Intel’s Arc B580 however shows that FluidX3D’s reported bandwidth aligns well with VRAM bandwidth usage from performance monitoring metrics.

GB10 and Strix Halo use 8533 and 8000 MT/s LPDDR5X over a 256-bit bus, giving 273 and 256 GB/s of theoretical bandwidth, respectively. That’s very high bandwidth for a consumer platform, but can’t compare to the 456 GB/s that the Arc B580 gets from GDDR6 over a 192-bit bus. Lack of memory bandwidth mean both iGPUs lean harder on caching than their discrete counterparts, and could suffer in workloads with poor access locality.

VkFFT implements Fast Fourier Transforms in multiple APIs. Here, I’m running its Vulkan benchmarks. GB10 leads Strix Halo, and turns in the most consistent performance across different test configurations. Strix Halo’s performance is more varied, and never beats GB10’s. Intel’s B580 is all over the place. It ends up with a higher average score than GB10, but takes huge losses in certain test configurations.

VkFFT outputs bandwidth figures alongside test results, and again shows what a discrete GPU can do with a fast, dedicated memory pool.

VkFFT can also target double precision (FP64) and half precision (FP16). GB10 is able to take an overall lead with double precision, while Strix Halo still can’t pull ahead in any test. Intel’s B580 can use its bandwidth advantage to win in some configurations, but not enough to get an overall score lead. None of the three GPUs here have strong FP64 performance, but VkFFT is often memory bound. The primary challenge is feeding the execution units, not having more of them.

Half precision runs tell a similar story, but with narrower gaps between the three GPUs. GB10, Strix Halo, and B580 average 76793.25, 62498.79, and 70980.17 respectively across all of the half precision test configurations.

In this self-penned benchmark, I perform a brute-force gravitational potential calculation over column density values. It’s a GPU accelerated version of something I did during a high school internship. Datacenter GPUs tend to do very well because I use FP64 values. GB10 and Strix Halo fall very far behind.

FAHBench is a benchmark for Folding at Home, a distributed computing project that runs protein folding simulations. The benchmark comes with three workloads, which can be run with either single or double precision. Two workloads use a DHFR (dihydrofolate reductase) system in explicit or implicit mode. Explicit simulates all 23558 atoms, while implicit only uses 2489 atoms to approximate a solution. NAV, or voltage gated sodium channel, is the third workload and uses 173112 atoms.

GB10 easily tops FAHBench’s workloads in single precision mode. FAHBench makes good use of local memory to keep data on-chip, reducing pressure on the global memory hierarchy. Global memory accesses that do happen have reasonable locality, even on the heavier NAV workload. Data Fabric performance monitoring on Strix Halo shows about 200 GB/s of GPU memory requests, and ~50 GB/s of traffic at the Unified Memory Controllers. That implies Strix Halo’s 32 MB system level cache caught about 150 GB/s of memory traffic. GB10’s 24 MB L2 is likely almost as effective, which would mean Nvidia also avoids DRAM bandwidth bottlenecks.

As a result, FAHBench favors compute throughput and plays well into GB10’s advantages. In some cases though, Strix Halo can keep up surprisingly well. In the DHFR single precision test, Strix Halo starts with a score just above 200 ns/day. If it maintained that, it would have nearly tied GB10. However, performance quickly drops as the chip pushes past 90 C. By targeting thin and light devices, Strix Halo has to contend with tighter thermal and power constraints than GB10 does in mini-PCs. Dell’s cooling in their mini-PC is excellent, and manages to keep GB10 in the low 60 C range during FAHBench runs.

Double precision workloads on FAHBench are very compute bound on all three GPUs. Intel’s B580 has the least horrible FP64 to FP32 ratio at 1:16. Strix Halo comes next at 1:32, followed by GB10 at 1:64.

Final Words

Large iGPUs are fascinating, and it’s great to see Nvidia entering the scene. GB10’s GPU has plenty of strengths compared to the one in AMD’s Strix Halo. It uses Nvidia’s latest Blackwell architecture, while Strix Halo makes do with RDNA3.5 instead of AMD’s latest RDNA4. Nvidia also uses TSMC’s 3nm process, while Strix Halo uses 4nm. GB10’s GPU is sized to be a bit larger than Strix Halo’s. Compute benchmarks show GB10’s strength. It beats Strix Halo more often than not. All else being equal, GB10’s iGPU could be a dangerous competitor to Strix Halo and a welcome arrival for PC enthusiasts.

Unfortunately, gaming on GB10 is off to a rough start. iGPU setups are locked into whichever CPU they come with. GB10 uses ARM CPU cores, and quite excellent ones from an ISA-agnostic point of view. However, ISA does matter when software can’t be easily ported. Nearly all PC games target x86-64, are closed source, and lack 64-bit Arm ports. Emulation and binary translation may let some games run, but at a performance cost. For example, Retrotom on Reddit ran Cyberpunk 2077 on GB10 and achieved about 50 FPS at 1080P with medium settings. Similar settings on Strix Halo in Cyberpunk 2077's built-in benchmark give just under 90 FPS. Strix Halo already suffered a CPU performance penalty because of contention at the memory bus and high memory latency compared to desktop discrete GPU setups. Taking another heavy hit to CPU performance is a tough pill to swallow.

Nvidia understandably positions GB10 as a compute solution, aimed at developers who want to test their code without going to a datacenter GPU. Developers can recompile their application to target 64-bit Arm, minimizing compatibility issues. GB10’s strong compute performance, high memory capacity, and iGPU advantages play well into this purpose. In the compute role however, GB10 has a VRAM bandwidth disadvantage compared to discrete peers.

The factors above mean GB10 is an exciting product, but possibly one with a limited audience. Like Strix Halo, GB10’s unified memory and more compact form factor have to face off against iGPU compromises and an extremely high price compared to discrete GPU setups with similar compute throughput. I hope to see both Nvidia and AMD iterate on their large iGPU designs to bring price down and minimize their compromises. There’s a lot of potential for large iGPU designs, and I’m excited to see that segment develop.

Cortex X925 in Nvidia’s GB10 achieves performance parity with AMD’s Zen 5 and Intel’s Lion Cove in their fastest desktop implementations. That gives Arm a core fast enough to not just play in laptop segments, but potentially in the most performance sensitive desktop applications too. Nvidia’s GB10 uses ten X925 cores, split across two clusters. One of those X925 cores reaches 4 GHz, while the others are not far behind at 3.9 GHz. Dell uses the GB10 chip in their Pro Max series, and we’re grateful to Dell for letting us test that product.

Overview

Arm’s Cortex X925 is a massive 10-wide core with a lot of everything. It has more reordering capacity than AMD’s Zen 5, and L2 capacity comparable to that of Intel’s recent P-Cores. Unlike Arm’s 7-series cores, X925 makes few concessions to reduce power and area. It’s a core designed through and through to maximize performance.

In Arm tradition, X925 has a number of configuration options. However, X925 omits the shoestring budget options present for A725. X925’s caches are all either parity or ECC protected, dropping A725’s option to do without error detection or correction. L1 caches on X925 are fixed at 64 KB, removing the 32 KB options on A725. X925’s most significant configuration options happen at L2, where implementers can pick between 2 MB or 3 MB of capacity. They can also choose either a 128-bit or 256-bit ECC granule to make area and reliability tradeoffs.

X925 interfaces with the rest of the system via Arm’s DSU-120, which acts as a cluster-level interconnect and hosts a L3 cache with up to 32 MB of capacity. X925 and its DSU support 40-bit physical addresses, which is adequate for consumer systems. However, it’s clearly not designed for server applications, where larger 48-bit or even 52-bit physical address spaces are common.

Branch Prediction

Performance and power efficiency starts with good branch prediction. Arm knows this, and X925 doesn’t disappoint. Its branch predictor can recognize extremely long repeating patterns. In a test with branches that are taken or not-taken in random patterns of increasing lengths, X925 behaves a lot like AMD’s Zen 5. AMD’s cores have featured very strong branch predictors since Zen 2, so X925’s results are impressive.

Cortex X925’s branch target caching compares well too. Arm has a large first level BTB capable of handling two taken branches per cycle. Capacity for this first level BTB varies with branch spacing, but it seems capable of tracking up to 2048 branches. This large capacity brings X925’s branch target caching strategy closer to Zen 5’s, rather than prior Arm cores that used small micro-BTBs with 32 to 64 entries. For larger branch footprints, X925 has slower BTB levels that can track up to 16384 branches and deliver targets with 2-3 cycle latency. There may be a mid-level BTB with 4096 to 8192 entries, though it’s hard to tell.

Compared to AMD’s Zen 5, X925 has roughly comparable capacity in its fastest BTB level depending on branch spacing. Zen 5 has more maximum branch target caching capacity, especially when it can use a single BTB entry to track two branches. Still, X925 has more branch target storage than Arm cores from a few years ago. Cortex X2 for example topped out at about 10K branch targets.

A 29 entry return stack helps predict returns from function calls, or branch-with-link in Arm instruction terms. Like Intel’s Sunny Cove and later cores, the return stack doesn’t work if return sites aren’t spaced far enough apart. I spaced the test “function” by 128 bytes to get clear results.

In SPEC CPU2017, Cortex X925 achieves branch prediction accuracy roughly on par with AMD’s Zen 5 across most tests, and may even be slightly ahead. 505.mcf and 541.leela consistently challenge branch predictors, and X925 pulls ahead in both. Intel’s Lion Cove is a bit behind both Zen 5 and X925.

SPEC’s floating point workloads are gentler on the branch predictor, but X925 still shows its strength. It’s again on par or slightly better than Zen 5.

Instruction Fetch and Decode

Cortex X925 ditches the MOP cache from prior Arm generations, much like mid-core companion (A725). While Cortex X925 doesn’t have the same tight power and area restrictions as A725, many of the same justifications apply. Arm already tackles decode costs through a variety of measures like predecode and running at lower clock speeds. A MOP cache would be excessive.

On the predecode side, X925’s TRM suggests the L1I stores data at 76-bit granularity. Arm instructions are 32-bits, so 76 bits would store two instructions and 12 bits of overhead. Unlike A725, Arm doesn’t indicate that any subset of bits correspond to an aarch64 opcode. They may have neglected to document it, or X925’s L1I may store instructions in an intermediate format that doesn’t preserve the original opcodes.

X925’s frontend can sustain 10 instructions per cycle, but strangely has lower throughput when using 4 KB pages. Using 2 MB pages lets it achieve 10 instructions per cycle as long as the test fits within the 64 KB instruction cache. Cortex X925 can fuse NOP pairs into a single MOP, but that fusion doesn’t bring throughput above 10 instructions per cycle. Details aside, X925 has high per-cycle frontend throughput compared to its x86-64 peer, but slightly lower actual throughput when considering Zen 5 and Lion Cove’s much higher clock speed. With larger code footprints, Cortex X925 continues to perform well until test sizes exceed L2 capacity. Compared to X925, AMD’s Zen 5 relies on its op cache to deliver high throughput for a single thread.

Rename and Allocate

MOPs from the frontend go through register renaming and have various bookkeeping resources allocated for them, letting the backend carry out out-of-order execution while ensuring results are consistent with in-order execution. While allocating resources, the core can carry out various optimizations to expose additional parallelism. X925 can do move elimination like prior Arm cores, and has special handling for moving an immediate value of zero into a register. Like on A725, the move elimination mechanism tends to fail if there are enough register-to-register MOVs close by. Neither optimization can be carried out at full renamer width, though that’s typical as cores get very wide.

Unlike A725, X925 does not have special handling for PTRUE, which sets a SVE predicate register to enable all lanes. A725 could eliminate PTRUE like a zeroing idiom, and process it without allocating a physical register. While this is a minor detail, it does show divergence between Arm’s mid-core and big-core lines.

Out-of-Order Execution

A CPU’s out-of-order backend executes operations as their inputs become ready, letting the core keep its execution units fed while waiting for long latency instructions to complete. Different sources give conflicting information about Cortex X925’s reordering window. Android Authority claims 750 MOPs. Wikichip believes it’s 768 instructions, based off an Arm slide that states Cortex X925 doubled reordering capacity over Cortex X4. Testing shows X925 can keep 948 NOPs in flight, which doesn’t correspond with either figure unless NOP fusion only worked some of the time.

Because results with NOPs were inconclusive, I tried testing with combinations of various instructions designed to dodge other resource limits. Mixing instructions that write to the integer and floating point registers showed X925 could have a maximum of 448 renamed registers allocated across its register files. Recognized zeroing idioms like MOV r,0 do not allocate an integer register, but also run up against the 448 instruction limit. I tried mixing in predicate register writes, but those also share the 448 instruction limit. Adding in stores showed the core could have slightly more than 525 instructions in flight. Adding in not-taken branches did not increase reordering capacity further. Putting an exact number on X925’s reorder buffer capacity is therefore difficult, but it’s safe to say there’s a practical limitation of around 525 instructions in flight. That puts it in the same neighborhood as Intel’s Lion Cove (576) and ahead of AMD’s Zen 5 (448).

X925’s register files, memory ordering queues, and other resources have comparable capacity to those in Zen 5 and Lion Cove. The only weakness is 128-bit vector execution, with correspondingly wide register file entries. AMD and Intel’s big cores have wider vector registers, and more of them available for renaming.

Execution Units and Schedulers

Arm laid out Cortex X925’s integer side to deliver high throughput while controlling port count for both the integer register file and scheduling queues. Eight ALU ports and three branch units are distributed across four schedulers in a layout that maximizes symmetry for common ALU operations. All four schedulers have two ALU ports and 28 entries. Similarly, each scheduler has one multiply-capable ALU pipe. Branches and special integer operations see a split, with the first three schedulers getting a branch pipe and the fourth scheduler getting support for pointer authentication and SVE predicate operations.

The aarch64 instruction set has a madd instruction that performs integer multiply-adds. Cortex A725 and older Arm cores had dedicated integer multi-cycle pipes that could handle madd along with other complex integer instructions. Cortex X925 instead breaks madd into two micro-ops, and handles it with any of its four multiply-capable integer pipes. Likely, Arm wanted to increase throughput for that instruction without the cost of implementing three register file read ports for each multiply-capable pipe. Curiously, Arm’s optimization guide refers to the fourth scheduler’s pipes as “single/multi-cycle” pipes. “Multi-cycle” is now a misnomer though, because the core’s “single-cycle” integer pipes can handle multiplies, which have two cycle latency. On Cortex X925, “multi-cycle” pipes distinguish themselves by handling special operations and being able to access FP/vector related registers.

Thanks to symmetry across the integer schedulers, X925’s renamer likely uses a simple round-robin allocation scheme for operations that can go to multiple schedulers. If I test scheduler capacity by interleaving dependent and independent integer adds, X925 can only keep half as many dependent adds in flight. Following dependent adds by independent ones only slightly reduces measured scheduling capacity. That suggests the renamer assigns a scheduler for each pending operation, and stalls if the targeted scheduling queue is full without scanning other eligible schedulers for free entries.

Cortex X925’s FPU has six pipes, all of which can handle vector floating point adds, multiplies, and multiply-adds. All six pipes also support vector integer adds and multiplies. Less common instructions like addv are still serviced by four pipes. X925’s FP schedulers are impressively large with approximately 53 entries each. For perspective, each of X925’s three FP schedulers has nearly as much capacity as Bulldozer’s 60 entry unified FP scheduler. Bulldozer used that scheduler to service two threads, while X925 uses its three FP schedulers to service a single thread.

High scheduler capacity and high pipe count should give X925 good performance in vectorized applications despite its 128-bit vector width.

Load/Store

Memory accesses are among the most complicated and performance critical operations on a modern CPU. For each memory access, the load/store unit has to translate program-visible virtual addresses into physical addresses. It also has to determine whether loads should get data from an older store, or from the cache hierarchy. Cortex X925 has four address generation units that calculate virtual addresses. Two of those can handle stores.

Address translations are cached in a standard two-level TLB setup. The L1 DTLB has 96 entries and is fully associative. A 2048 entry 8-way L2 TLB handles larger data footprints, and adds 6 cycles of latency. Zen 5 for comparison has the same L1 DTLB capacity and associativity, but a larger 4096 entry L2 DTLB that adds 7 cycles of latency. Another difference is that Zen 5 has a separate L2 ITLB for instruction-side translations, while Cortex X925 uses a unified L2 TLB for both instructions and data. AMD’s approach could further increase TLB reach, because data and instructions often reside on different pages.

Store forwarding on the integer side works for all loads contained within a prior store. It’s an improvement over prior arm cores like the Cortex X2, which could only forward either half of a 64-bit store to a 32-bit load. Forwarding on the FP/vector side still works like older Arm cores, and only works for specific load alignments with respect to the store address. Unlike recent Intel and AMD cores, Cortex X925 can’t do zero latency forwarding when store and load addresses match exactly. To summarize store forwarding behavior:

Memory dependencies can’t be definitively determined until address translation finishes. Some cores do an early check before address translation completes, using bits of the address that represent an offset into a page. Cortex X925 might be doing that because it takes a barely measurable penalty when loads and stores appear to overlap in the low 12 bits.

Core-Private Caches

Cortex X925 has a 64 KB L1 data cache with 4 cycle latency like A725 companions in GB10, but takes advantage of its larger power and area budget to make that capacity go further. It uses a more sophisticated re-reference interval prediction (RRIP) replacement policy rather than the pseudo-LRU policy used on A725. Bandwidth is higher too. Arm’s technical reference manual says the L1D has “4x128-bit read paths and 4x128-bit write paths”. Sustaining more than two stores per cycle is impossible because the core only has two store-capable AGUs. Loads can use all four AGUs, and can achieve 64B/cycle from the L1 data cache. That’s competitive against many AVX2-capable x86-64 CPUs from a few generations ago. However, more recent Intel and AMD cores can use their wider vector width and faster clocks to achieve much higher L1D bandwidth, even if they also have four AGUs.

Arm offers 2 MB 8-way and 3 MB 12-way L2 cache options. Mediatek and Nvidia chose the 2 MB option, and testing shows it has 12 cycles of latency. THis low cycle count latency lets Arm remain competitive against Intel and AMD’s L2 caches, despite running at lower clock speeds. L2 bandwidth comes in at 32 bytes per cycle for reads, and increases to approximately 45 bytes per cycle with a read-modify-write pattern.

Like AMD, Arm makes the L2 strictly inclusive of the L1 data cache, which lets the L2 act as a snoop filter. If an incoming snoop misses in the L2, the core can be sure it won’t hit in the L1D either.

Performance: SPEC CPU2017

Cortex X925 turns in an excellent performance in SPEC CPU2017’s integer suite. Its score in the integer suite is within margin of error compared to Intel and AMD’s highest performance cores, in their highest performance desktop configurations. AMD’s Zen 5 still slips ahead in SPEC’s floating point suite, but not by a large margin. Running Zen 5 with faster DDR5-6000 instead of DDR5-5600 memory can slightly increase its integer score, but only to 11.9 - not enough to pull away from X925. Creating a high performance core involves making the right tradeoff between frequency and performance per clock, while keeping everything else in balance. It’s safe to say Arm has found a combination of frequency and performance per clock that lets them compete with AMD and Intel’s best.

Diving deeper into individual workloads shows a complex picture. X925 trades blows with the higher clocked Intel and AMD cores in core-bound workloads. 548.exchange2 and 500.perlbench both show the advantages of clock speed scaling, with Intel and AMD’s higher clocking 8-wide cores easily outpacing Arm’s 4 GHz 10-wide one. But 525.x264 turns things around. Cortex X925 is able to finish that workload with fewer instructions than its x86-64 peers, while maintaining a large IPC advantage. X925 continues to do well in workloads that challenge the branch predictor, like 541.leela and 505.mcf. Finally, memory bound tests like 520.omnetpp are heavily influenced by factors outside the core.

Performance monitoring counter data shows how Cortex X925 uses high IPC to make up for a clock speed deficit. Whether it’s enough IPC to match Lion Cove and Zen 5 varies depending on the individual test, but overall Arm’s chosen IPC and clock speed targets are just as viable as Intel’s and AMD’s.

SPEC CPU2017’s floating point workloads throw a wrench into the works. Cortex X925 falls behind Zen 5 on enough tests to leave AMD’s latest and greatest core with a clear victory. In Arm’s favor, they are able to keep pace with Intel’s Lion Cove.

PMU data indicates X925 is able to maintain higher IPC relative to the competition. Unfortunately for X925, several tests require many more instructions to complete with the aarch64 instruction set compared to x86-64. Arm needs a large enough IPC advantage to overcome both a clock speed deficit and a less efficient representation of the work. Cortex X925 does achieve high IPC, but it’s not high enough.

Plotting instruction counts shows just how severe the situation can get for Cortex X925. 507.cactuBSSN, 521.wrf, 549.fotonik3d, and 554.roms all require more instructions on X925, and by no small margin. 554.roms is the worst offender, and makes X925 execute more than twice as many instructions compared to Zen 5. Average IPC in these four tests is nowhere near core width for any of these tested cores, but crunching through extra instructions isn’t the only issue. Higher instruction counts place more pressure on core out-of-order resources, impacting its ability to hide latency.

Final Words

Arm now has a core with enough performance to take on not only laptop, but also desktop use cases. They’ve also shown it’s possible to deliver that performance at a modest 4 GHz clock speed. Arm achieved that by executing well on the fundamentals throughout the core pipeline. X925’s branch predictor is fast and state-of-the-art. Its out-of-order execution engine is truly gargantuan. Penalties are few, and tradeoffs appear well considered. There aren’t a lot of companies out there capable of building a core with this level of performance, so Arm has plenty to be proud of.

That said, getting a high performance core is only one piece of the puzzle. Gaming workloads are very important in the consumer space, and benefit more from a strong memory subsystem than high core throughput. A DSU variant with L3 capacity options greater than 32 MB could help in that area. X86-64’s strong software ecosystem is another challenge to tackle. And finally, Arm still relies on its partners to carry out its vision. I look forward to seeing Arm take on all of these challenges, while also iterating on their core line to keep pace as AMD and Intel improve their cores. Hopefully, extra competition will make better, more affordable CPUs for all of us.

Here, I’ll be looking into the Cortex A725 in Nvidia’s GB10. GB10 has ten A725 cores and ten X925 cores split across two clusters, with five of each core type in each. The A725 cores run at 2.8 GHz, while the high performance X925 cores reach 3.9 to 4 GHz. As covered before, one of GB10’s clusters has 8 MB of L3, while the other has 16 MB. GB10 will provide a look at A725’s core architecture, though implementation choices will obviously influence performance. For comparison, I’ll use older cores from Arm’s 7-series line as well as Intel’s recent Skymont E-Core.

Acknowledgements

A massive thanks to Dell for sending over two of their Pro Max with GB10 for testing. In our testing the Dell Pro Maxs were quite quiet even when under a Linpack load which speaks to Dell’s thermal design keeping the GB10 SoC within reasonable levels.

Overview

Cortex A725 is a 5-wide out-of-order core with reordering capacity roughly on par with Intel’s Skylake or AMD’s Zen 2. Its execution engine is therefore large enough to dip deep into the benefits of out-of-order execution, but stops well short of high performance core territory. The core connects to the rest of the system via Arm’s DynamIQ Shared Unit 120 (DSU-120) via 256-bit read and write paths. Arm’s DSU-120 acts as a cluster level interconnect and contains the shared L3 cache.

In Arm tradition, A725 offers a number of configuration options to let implementers fine tune area, power, and performance. These options include:

Branch Predictor

Cortex A710’s branch predictor already showed capabilities on par with older Intel high performance cores, and A725 inherits that strength. In a test where branch direction correlates with increasingly long random patterns, A725 trades blows with A715 rather than being an overall improvement. With a single branch, latencies start going up once the random pattern exceeds 2048 for a single branch, versus 3072 on A710. With 512 branches in play, both A710 and A725 see higher latencies once the pattern length exceeds 32. However, A725 shows a gradual degradation compared to the abrupt one seen on A710.

Arm’s 7-series cores historically used branch target buffers (BTBs) with very low cycle count latencies, which helped make up for their low clock speed. A725’s BTB setup is fine on its own, but regresses compared to A710. Each BTB level is smaller compared to A710, and the larger BTB levels are slower too.

A725’s taken branch performance trades blows with Skymont. Tiny loops favor A725, because Skymont can’t do two taken branches per cycle. Moderate branch counts give Skymont a win, because Intel’s 1024 entry L1 BTB has single cycle latency. Getting a target out of A725’s 512 entry L1 BTB adds a pipeline bubble. Then, both cores have similar cycle count latencies to a 8192 entry L2 BTB. However, that favors Skymont because of its higher clock speed.

Returns, or branch-with-link instructions in Arm terms, are predicted on A725 using a 16 entry return stack. A call+return pair has a latency of two cycles, so each branch likely executes with single cycle latency.

In SPEC CPU2017, A725 holds the line in workloads that are easy on the branch predictor while sometimes delivering improvements in difficult tests. 541.leela is the most impressive case. In that workload. A725 manages a 6.61% decrease in mispredicts per instruction compared to Neoverse N2. Neoverse N2 is a close relative of Cortex A710, and was tested here in Azure’s Cobalt 100. A725’s gains are less clear in SPEC CPU2017’s floating point workloads, where it trades blows with Neoverse N2. Still, it’s an overall win for A725 because SPEC’s floating point workloads are less branchy and put more emphasis on core throughput.

Comparing with Skymont is more difficult because there’s no way to match the instruction stream due to ISA differences. It’s safe to say the two cores land in the same ballpark, and both have excellent branch predictors.

Instruction Fetch and Decode

After the branch predictor has decided where to go, the frontend has to fetch instructions and decode them into the core’s internal format. Arm’s cores decode instructions into macro-operations (MOPs), which can be split further into micro-operations (uOPs) in later pipeline stages. Older Arm cores could cache MOPs in a MOP cache, much like AMD and Intel’s high performance CPUs. A725 drops the MOP cache and relies exclusively on a conventional fetch and decode path for instruction delivery. The fetch and decode path can deliver 5 MOPs per cycle, which can correspond to 6 instructions if the frontend can fuse a pair of instructions into a single MOP. With a stream of NOPs, A725’s frontend delivers five instructions per cycle until test sizes spill out of the 32 entry instruction TLB. Frontend bandwidth then drops to under 2 IPC, and stays there until the test spills out of L3.

For comparison, A710 could use its MOP cache to hit 10 IPC if each MOP corresponds to a fused instruction pair. That’s easy to see with a stream of NOPs, because A710 can fuse pairs of NOPs into a single MOP. A725’s decoders can no longer fuse pairs of adjacent NOPs, though they still support other fusion cases like CMP+branch.

Arm’s decision to ditch the MOP cache deserves discussion. I think it was long overdue, because Arm cores already took plenty of measures to mitigate decode costs. Chief among these is a predecode scheme. Instructions are partially decoded when they’re filled into the instruction cache. L1I caches on cores up to the Cortex A78 and Cortex X1 stored instructions in a 36 or 40-bit intermediate format. It’s a compromise between area-hungry micro-op caching and fully decoding instructions with every fetch. Combining predecode and micro-op caching is overkill, especially on low clocked cores where timing should be easier. Seeing A710 hit ridiculous throughput in an instruction bandwidth microbenchmark was fun, but it’s a corner case. Arm likely determined that the core rarely encountered more than one instruction fusion candidate per cycle, and I suspect they’re right.

A725’s Technical Reference Manual suggests Arm uses a different predecode scheme now, with 5 bits of “sideband” predecode data stored alongside each 32-bit aarch64 instruction. Sideband bit patterns indicate a valid opcode. Older Arm CPUs used the intermediate format to deal with the large and non-contiguous undefined instruction space. A725’s sideband bits likely cover the same purpose, along with aiding the decoders for valid instructions.

Rename and Allocate

MOPs from the frontend go to the rename/allocate stage, which can handle five MOPs per cycle. Besides doing classical register renaming to remove false dependencies, modern CPUs can pull a variety of tricks to expose additional parallelism. A725 can recognize moving an immediate value of 0 to a register as a zeroing idiom, and doesn’t need to allocate a physical register to hold that instruction’s result. Move elimination is supported too, though A725 can’t eliminate chained MOVs across all renamer slots like AMD and Intel’s recent cores. In a test with independent MOVs, A725 allocates a physical register for every instruction. Arm’s move elimination possibly fails if the renamer encounters too many MOVs in the same cycle. That’s what happens on Intel’s Haswell, but I couldn’t verify it on A725 because Arm doesn’t have performance events to account for move elimination efficacy.

Memory renaming is another form of move elimination, where the CPU carries out zero-cycle store forwarding if both the load and store use the same address register. Intel first did this on Ice Lake, and AMD first did so on Zen 2. Skymont has memory renaming too. I tested for but didn’t observe memory renaming on A725.

Out-of-Order Execution Engine

Arm historically took a conservative approach to growing reordering capacity on their 7-series cores. A larger reordering window helps the core run further ahead of a stalled instruction to extract instruction level parallelism. Increasing the reordering window can improve performance, but runs into diminishing returns. A725 gets a significant jump in reorder buffer capacity, from 160 to 224 entries. Reordering capacity is limited by whichever structure fills first, so sizing up other structures is necessary to make the most of the ROB size increase.

A725 gets a larger integer register file and deeper memory ordering queues, but not all structures get the same treatment. The FP/vector register file has more entries, but has fewer renames available for 128-bit vector results. FP/vector register file entries are likely 64-bit now, compared to 128-bit from before. SVE predicate registers take a slight decrease too. Vector execution tends to be area hungry and only benefits a subset of applications, so Arm seems to be de-prioritizing vector execution to favor density.

Instruction Scheduling and Execution

A725’s execution pipe layout looks a lot like A710’s. Four pipes service integer operations, and are each fed by a 20 entry scheduling queue. Two separate ports handle branches. I didn’t measure an increase in scheduling capacity when mixing branches and integer adds, so the branch ports likely share scheduling queues with the ALU ports. On A710, one of the pipes could only handle multi-cycle operations like madd. That changes with A725, and all four pipes can handle simple, single cycle operations. Integer multiplication remains a strong point in Arm’s 7-series architecture. A725 can complete two integer multiplies per cycle, and achieves two cycle latency for integer multiplies.

Two pipes handle floating point and vector operations. Arm has used a pair of mostly symmetrical FP/vector pipes dating back to the Cortex A57. A725 is a very different core, but a dual pipe FPU still looks like a decent balance for a density optimized core. Each FP/vector pipe is fed by a 16 entry scheduling queue, and both share a non-scheduling queue with approximately 23 entries. A single pipe can use up to 18 entries in the non-scheduling queue after its scheduler fills.

Compared to A710, A725 shrinks the FP scheduling queues and compensates with a larger non-scheduling queue. A non-scheduling queue is simpler than a scheduling one because it doesn’t have to check whether operations are ready to execute. It’s simply an in-order queue that sends MOPs to the scheduler when entries free up. A725 can’t examine as many FP/vector MOPs for execution ready-ness as A710 can, but retains similar ability to look past FP/vector operations in the instruction stream to find other independent operations. Re-balancing entries between the schedulers and non-scheduling queue is likely a power and area optimization.

Load/Store

A725 has a triple AGU pipe setup for memory address calculation, and seems to feed each pipe with a 16 entry scheduling queue. All three AGUs can handle loads, and two of them can handle stores. Indexed addressing carries no latency penalty.

Addresses from the AGU pipes head to the load/store unit, which ensures proper ordering for memory operations and carries out address translation to support virtual memory. Like prior Arm cores, A725 has a fast path for forwarding either half of a 64-bit store to a subsequent 32-bit load. This fast forwarding has 5 cycle latency per dependent store+load pair. Any other alignment gives 11 cycle latency, likely with the CPU blocking the load until the store can retire.

A725 is more sensitive to store alignment than A710, and seems to handle stores in 32 byte blocks. Store throughput decreases when crossing a 32B boundary, and takes another hit if the store crosses a 64B boundary as well. A710 takes the same penalty as A725 does at a 64B boundary, and no penalty at a 32B boundary.

Address Translation

Arm grew the L1 DTLB from 32 entries in A710 to 48 entries in A725, and keeps it fully associative. DTLB coverage thus increases from 128 to 192 KB. Curiously, Arm did the reverse on the instruction side and shrank A710’s 48 entry instruction TLB to 32 entries on A725. Applications typically have larger data footprints than code footprints, so shuffling TLB entries to the data side could help performance. Intel’s Skymont takes the opposite approach, with a 48 entry DTLB and a 128 entry ITLB.

A725’s L2 TLB has 1536 entries and is 6-way set associative, or can have 1024 entries and be 4-way set associative in a reduced area configuration. The standard option is an upgrade over the 1024 entry 4-way structure in A710, while the area reduced one is no worse. Hitting the L2 TLB adds 5 cycles of latency over a L1 DTLB hit.

Intel’s Skymont has a much larger 4096 entry L2 TLB, but A725 can make up some of the difference by tracking multiple 4 KB pages with a single TLB entry. Arm’s Technical Reference Manual describes a “MMU coalescing” setting that can be toggled via a control register. The register lists “8x32” and “4x2048” modes, but Arm doesn’t go into further detail about what they mean. Arm is most likely doing something similar to AMD’s “page smashing”. Cores in AMD’s Zen line can “smash” together pages that are consecutive in virtual and physical address spaces and have the same attributes. “Smashed” pages act like larger pages and opportunistically extend TLB coverage, while letting the CPU retain the flexibility and compatibility benefits of a smaller page size. “8x32” likely means A725 can combine eight consecutive 4 KB pages to let a TLB entry cover 32 KB. For comparison, AMD’s Zen 5 can combine four consecutive 4 KB pages to cover 16 KB.

Core-Private Caching

A725 has a 64 KB 4-way set associative data cache. It’s divided into 16 banks to service multiple accesses, and achieves 4 cycle load-to-use latency. The L1D uses a pseudo-LRU replacement scheme, and like the L1D on many other CPUs, is virtually indexed and physically tagged (VIPT). Bandwidth-wise, the L1D can service three loads per cycle, all of which can be 128-bit vector loads. Stores go through at two per cycle. Curiously, Arm’s Technical Reference Manual says the L1D has four 64-bit write paths. However, sustaining four store instructions per cycle would be impossible with only two store AGUs.

The L2 cache is 8-way set associative, and has two banks. Nvidia selected the 512 KB option, which is the largest size that supports 9 cycle latency. Going to 1 MB would bump latency to 10 cycles, which is still quite good. Because L2 misses incur high latency costs, Arm uses a more sophisticated “dynamic biased” cache replacement policy. L2 bandwidth can reach 32 bytes per cycle, but curiously not with read accesses. With either a read-modify-write or write-only access pattern, A725 can sustain 32 bytes per cycle out to L3 sizes.

Arm’s optimization guide details A725’s cache performance characteristics. Latencies beyond L2 rely on assumptions that don’t match GB10’s design, but the figures do emphasize the expensive nature of core to core transfers. Reaching out to another core incurs a significant penalty at the DSU compared to getting data from L3, and takes several times longer than hitting in core-private caches. Another interesting detail is that hitting in a peer core’s L2 gives better latency than hitting in its L1. A725’s L2 is strictly inclusive of L1D contents, which means L2 tags could be used as a snoop filter for the L1D. Arm’s figures suggest the core first checks L2 when it receives a snoop, and only probes L1D after a L2 hit.

Performance: SPEC CPU2017

A725’s SPEC CPU2017 performance brackets Intel’s Crestmont as tested in their Meteor Lake mobile design. Neoverse N2 scores slightly higher. Core performance is heavily dependent on how it’s implemented, and low clock speeds in GB10’s implementation mean its A725 cores struggle to climb past older density optimized designs.

Clock speed tends to heavily influence core-bound bound workloads. 548.exchange2 is one example. Excellent core-private cache hitrates mean IPC (instructions per cycle) doesn’t change with clock speed. A725 manages a 10.9% IPC increase over Neoverse N2, showing that Arm has improved their core architecture. However, Neoverse N2’s 3.4 GHz clock speed translates to a 17% performance advantage over A725. Crestmont presents a more difficult comparison because of different executed instruction counts. 548.exchange2 executed 1.93 trillion instructions on the x86-64 cores, but required 2.12 trillion on aarch64 cores. A725 still achieves higher clock normalized performance, but Cresmont’s 3.8 GHz clock speed puts it 14.5% ahead. Other high IPC, core bound workloads show a similar trend. Crestmont chokes on something in 538.imagick though.

On the other hand, clock speed means little in workloads that suffer a lot of last level cache misses. DRAM doesn’t keep pace with clock speed increases, meaning that IPC gets lower as clock speed gets higher. A725 remains very competitive in those workloads despite its lower clock speed. In 520.omnetpp, it matches the Neoverse N2 and takes a 20% lead over Crestmont. A725 even gets close to Skymont’s performance in 549.fotonik3d.

Final Words

A725 only gains entries in the most important out-of-order structures compared to A710. Elsewhere, Arm preferred to rebalance resources, or even cut back on them. It’s impressive to see Arm trim the fat on a core that didn’t have a lot of fat to begin with, and it’s hard to argue with the results. A725’s architecture looks like an overall upgrade over A710 and Neoverse N2. I suspect A725 would come out top if both ran at the same clock speeds and were supported by identical memory subsystems. In GB10 though, A725’s cores don’t clock high enough to beat older designs.

Arm’s core design and Nvidia’s implementation choices contrast with AMD and Intel’s density optimized strategies. Intel’s Skymont is almost a P-Core, and only makes concessions in the most power and area hungry features like vector execution. AMD takes their high performance architecture and targets lower clock speeds, taking whatever density gains they can get in the process. Now that Arm’s cores are moving into higher performance devices like Nvidia’s GB10, it’ll be interesting to see how their density-optimized strategy plays out against Intel and AMD’s.

Again, a massive thank you to Dell for sending over the two Pro Max with GB10s for testing.

If you like the content then consider heading over to the Patreon or PayPal if you want to toss a few bucks to Chips and Cheese, also consider joining the Discord.

Here at CES 2026, AMD showed off their upcoming Venice series of server CPUs and their upcoming MI400 series of datacenter accelerators. AMD has talked about the specifications of both Venice and the MI400 series at their Advancing AI event back in June of 2025, but this is the first time AMD has shown off the silicon for both of product lines.

Starting with Venice, the first thing to notice is the packaging of the CCDs to the IO dies is different. Instead of using the organic substrate of the package to run the wires between the CCDs and the IO dies that AMD has used since EPYC Rome, Venice appears to be using a more advanced form of packaging similar to Strix Halo or MI250X. Another change is that Venice appears to have two IO dies instead of the single IO die that the prior EPYC CPUs had.

Venice has 8 CCDs each of which have 32 cores for a total of up to 256 cores per Venice package. Doing some measuring of each of the dies, you get that each CCD is approximately 165mm² of N2 silicon. If AMD has stuck to 4MB of L3 per core than each of these CCDs have 32 Zen 6 cores and 128MB of L3 cache along with the die to die interface for the CCD <-> IO die communications. At approximately 165mm² per CCD, that would make a Zen 6 core plus the 4MB of L3 per core about 5mm² each which is similar to Zen 5’s approximately 5.34mm² on N3 when counting both the Zen 5 core and 4MB of L3 cache.

Moving to the IO dies, they each appear to be approximately 353mm² for a total of just over 700mm² of silicon dedicated for the IO dies. This is a massive increase from the approximately 400mm² that the prior EPYC CPUs dedicated for their IO dies. The two IO dies appear to be using an advanced packaging of some kind similar to the CCDs. Next to the IO dies appear to be 8 little dies, 4 on each side of the package, which are likely to either be structural silicon or deep trench capacitor dies meant to improve power delivery to the CCDs and IO dies.

Shifting off of Venice and on to the MI400 accelerator, this is a massive package with 12 HBM4 dies and “twelve 2 nanometer and 3 nanometer compute and IO dies”. It appears as if there are two base dies just like MI350. But unlike MI350, there appears to also be two extra dies on the top and bottom of the base dies. These two extra dies are likely for off-package IO such as PCIe, UALink, etc.

Calculating the die sizes of the base dies and the IO dies, the die size of the base die is approximately 747mm² for each of the two base dies with the off-package IO dies each being approximately 220mm². As for the compute dies, while the packaging precludes any visual demarcation of the different compute dies, it is likely that there are 8 compute dies with 4 compute dies on each base die. So while we can’t figure out the exact die size of the compute dies, the maximum size is approximately 180mm². The compute chiplet is likely in the 140mm² to 160mm² region but that is a best guess that will have to wait to be confirmed.

The MI455X and Venice are the two SoCs that are going to be powering AMD’s Helios AI Rack but they aren’t the only new Zen 6 and MI400 series products that AMD announced at CES. AMD announced that there would be a third member of the MI400 family called the MI440X joining the MI430X and MI455X. The MI440X is designed to fit into the 8-way UBB boxes as a direct replacement for the MI300/350 series.

AMD also announced Venice-X which is likely is going to be a V-Cache version of Venice. This is interesting because not only did AMD skip Turin-X but if there is a 256 core version of Venice-X, then this would be the first time that a high core count CCD will have the ability to support a V-Cache die. If AMD sticks to the same ratio of base die cache to V-Cache die cache, then each 32 core CCD would have up to 384MB of L3 cache which equates to 3 Gigabytes of L3 cache across the chip.

Both Venice and the MI400 series are due to launch later this year and I can’t wait to learn more about the underlying architectures of both SoCs.

If you like the content then consider heading over to the Patreon or PayPal if you want to toss a few bucks to Chips and Cheese, also consider joining the Discord.