LLVM Divination of GFX1251’s Differences

Hello you fine Internet folks, this article is a sequel to the Scrying the AMD GFX1250 LLVM Tea Leaves article where we are going to look at the differences between GFX1250 and GFX1251.

So if you want to know what GFX1250 is changing compared to the prior generation of CDNA along with RDNA4, I would recommend reading that article first then coming back to this article.

The Shared Foundation

Starting with what is shared between both GFX1250 and GFX1251, they both inherit LLVM’s common GFX12.5 target which includes Wave32-only execution, 64-bit literals, a 32-bank LDS, packed FP32 operations, FP8 conversion support, and 1,024 addressable VGPR. Both targets also have 320 KiB of addressable LDS along with the same GFX12.5 SWMMAC instruction sets and the same WMMA and transcendental co-execution hazards.

The scheduling model is similarly close as well, with the notable exceptions being 64-bit math and WMMA AKA matrix math. Outside of those two cases, LLVM shows both GFX1250 and GFX1251 as having the same latency for 32-bit, LDS, and scalar memory operations along with the same modeled branch and VMEM costs among other similarities.

Getting Serious About FP64

The biggest difference between GFX1250 and GFX1251 is the 64-bit paths on GFX1251 getting a significant improvement. LLVM gives GFX1251 the same ‘FullRate64Ops’ feature as prior CDNA-based accelerators which means that the compiler expects the hardware to have full-rate operations for most FP64 instructions.

LLVM’s complete GFX12.5 schedule

Packing FP64 in like Sardines

Moving to packed arithmetic, GFX1251 adds v_pk_add_f64, v_pk_mul_f64, v_pk_fma_f64, v_pk_min_num_f64, and v_pk_max_num_f64 where each instruction operates on a v2f64 or two 64-bit values per active SIMD lane. Because a VGPR contains 32 bits per lane, every packed FP64 operand occupies four VGPRs meaning that the three sources consume 12 VGPRs and the result consumes four.

v_pk_fma_f64 v[4:7], v[8:11], v[12:15], v[16:19]

Across a Wave32, one instruction represents 64 FP64 FMAs operations which equates to 128 FP64 FLOPS.

The packed integer side follows the same 128-bit register layout. GFX1251 adds v_pk_add_nc_u64, v_pk_sub_nc_u64, and v_pk_lshl_add_u64, each operating on two unsigned 64-bit values per lane. Those instructions are useful for paired address calculations and counters, though LLVM does not expose a packed U64 instruction set as broad as the FP64 one.

Importantly, LLVM’s optimizer does know these instructions exist because the GFX1251 cost model advertises a 128-bit fixed vector width when packed FP64 or U64 operations are available, permits a vectorization factor of two for 64-bit elements, and cuts the estimated cost of work in half. In other words, this is not just an ISA feature for hand-written assembly, LLVM is being taught to combine pairs of 64-bit operations so that regular compiled code can use it.

FP64 Matrix Math

The v_wmma_f64_16x16x4_f64 instruction for FP64 Matrix Math is a GFX1251-only instruction appropriately enough in the LLVM feature set labeled GFX1251GEMMInsts. Notably, despite the feature set implying multiple instructions in this set, as of the time of writing the only instruction in this feature set is v_wmma_f64_16x16x4_f64.

v_wmma_f64_16x16x4_f64 v[8:23], v[0:3], v[4:7], v[8:23]

Per wave, the two matrix inputs consume four VGPRs each, while the accumulator and result occupy 16 VGPRs. Across a wave, the instruction updates a 16x16 FP64 output tile over a K dimension of four. That works out to 1,024 FP64 FMA operations or 2,048 FLOPS per wave with LLVM scheduling it as a four-pass WMMA instruction with a 16-cycle latency.

Trading Off Low-Precision WMMA Math

The tradeoff for all of this FP64 compute is a reduction of low precision matrix math. LLVM enables GFX125xLowestRateWMMA for GFX1251 and places many instructions into latency classes that are two to four times as long as GFX1250’s.

There is an unmerged patch that reduces the latency of FP8/BF8 and FP4 operations on GFX1250 from 8 cycles to 4 cycles along with a reduction of several mixed FP6/FP8 cases from 16 cycles to 8 cycles.

The slower instruction rate also makes GFX1251’s dependency hazards longer. LLVM groups GFX12.5 WMMA instructions into rate categories and counts the number of unrelated VALU instructions or `v_nop` slots required between dependent operations. The slowest GFX1251 category can require up to 17 intervening slots before a dependent WMMA operation and 16 before a dependent VALU operation, whereas the faster categories require only 9 slots before a dependent WMMA operation and 8 slots before a dependent VALU operation.

Conclusion

GFX1250 and GFX1251 share enough features that calling them members of the same GFX12.5 family makes sense. They use the same wave size, expose the same architectural VGPR range and LDS capacity, and look almost identical to LLVM for ordinary 32-bit and memory operations which points to GFX1250 and GFX1251 using the same base architecture.

However, it is very clear that AMD has decided to specialize the two GFX12.5 architectures far more than even the MI300 family. The MI300 series had the MI300A and MI300X which while specializing the SoCs, with MI300A having 3 Zen 4 CCDs plus 6 CDNA3 XCDs onboard compared to MI300X having 8 CDNA3 XCDs, reused the same XCDs between them. The MI430X and MI455X are very clearly not sharing the same XCDs but have specialized the XCDs for either HPC or ML workloads.

GFX1250 is the better target for low-precision matrix math. Depending on the instruction, GFX1251 takes two to four times as many modeled cycles and can require much longer spacing before dependent operations. GFX1251 instead gets full-rate FP64, packed FP64 and U64 arithmetic, 64-bit DPP, and support for FP64 matrix math.

That makes GFX1251 much more interesting in the broader HPC space. A large portion of traditional HPC still depends on FP64 for numerical stability and reproducibility, including computational fluid dynamics, weather and climate modeling, molecular dynamics, finite-element analysis, seismic processing, quantum chemistry, and other scientific workloads. These applications do not only need a high peak FP64 number, they also need FP64 arithmetic, conversions, reductions, address calculations, and dense linear algebra to work well together. GFX1251’s full-rate FP64 path, packed 64-bit instructions, 64-bit DPP support, and FP64 WMMA instruction all point to a HPC focus.

FP64 matrix operations are especially relevant because DGEMM sits underneath a large amount of scientific computing, directly through BLAS and LAPACK or indirectly through larger simulation and solver stacks. A 16x16x4 FP64 tile does not by itself establish application performance, but it gives GFX1251 a dedicated way to accelerate the double-precision matrix math used by workloads such as HPL and dense numerical solvers. Meanwhile, packed FP64 and 64-bit DPP should help with the less tidy vector math and reductions which make up a good portion of HPC compute. That makes the overall FP64 story both broader and stronger than adding one impressive matrix opcode.

In the end what we are seeing with the GFX12.5 family is a story of different architectures for different markets. While GFX1250 is very much focused on the insanely lucrative ML market, GFX1251 is focused on the still very important HPC market where high precision compute still reigns supreme.