Reverse-mode automatic differentiation (AD) computes exact gradients with respect to thousands of inputs in roughly the time of a single function evaluation. If you write numerical C++ for finance, scientific computing, or ML and you care about performance, this talk will show you where AD overhead actually comes from and what to do about it.
A straightforward operator-overloading implementation records every arithmetic operation onto a tape and replays it backward to propagate derivatives. On real numerical code this easily introduces 50-100x overhead, and most of that has nothing to do with calculus. It is heap allocation on every operation. It is cache misses walking the tape backward. It is branch mispredictions at memory chunk boundaries. It is recording tape entries for operations where none of the operands are even being differentiated.
We work through a series of C++ techniques that bring this down to around 4x on production code in finance. The two that matter most: a chunked arena allocator with cache-line alignment and branch-prediction hints that turns the recording hot path into a pointer bump and a store, and expression templates that collapse an entire right-hand side into one tape push with all derivatives accumulated in a compile-time-sized stack buffer. We pull up compiler output to show the abstraction really does compile away.
After the big two, we get into the smaller wins that are still worth knowing about: skipping zero adjoints during the backward sweep to exploit sparsity, picking derivative formulas at compile time that reuse the forward result instead of recomputing it, and demoting active-times-passive operations from binary to unary to halve their tape footprint. We also show how a should-record check propagated through the expression tree lets the tape skip entire statements when no operand is active, and how aggressive inlining lets the compiler eliminate these checks entirely for passive subexpressions, so the cost of asking is zero when the answer is no. We benchmark each one in isolation on four numerical workloads ranging from 8 to 161 sensitivities, so you can see what each technique is actually worth.
Practical HPC — Forcing the Compiler's Hand with Modern C++
Achieving near-peak FLOPs on modern CPUs requires exploiting SIMD, register files, and cache hierarchies — yet the levers that matter most (loop unrolling, vectorization width, tile sizes, register tiling) are largely outside the programmer's direct control. Autovectorization is fragile, #pragma unroll is advisory and non-portable, and the information the compiler needs most — loop bounds, problem shapes, working-set sizes — is typically only available at runtime. The result is a familiar gap between theoretical peak and delivered performance, bridged in practice only by hand-written intrinsics or inline assembly, fragile macro forests, or external code generators that sacrifice maintainability and type safety.
Modern C++ is, sometimes surprisingly, an excellent language for closing that gap without leaving the host language. We walk through the concrete features that make this possible and how to use them in practice — templates and constexpr to promote runtime values into compile-time constants, if constexpr and parameter packs for shape-specialized kernels, generic and template lambdas for loop bodies that are unrolled and inlined by construction, and concepts to turn silent performance cliffs into compile-time errors. Together, these give the programmer precise control over what the compiler emits: compile-time dispatch that specializes kernels per shape and target, guaranteed static unrolling that unlocks ILP and register tiling without relying on optimizer heuristics, and hardware-aware specialization parameterized on register count, SIMD width, and cache sizes.
Looking forward, C++26 and static reflection push this approach considerably further, turning today's disciplined metaprogramming into something closer to first-class, in-language code generation — and squarely aligning the language with the goal of making inline assembly unnecessary, even at the highest performance tiers.
These patterns are powerful but verbose and reappear across kernels, so we briefly introduce POET (Performance Optimized Excessive Templates), a header-only library usable from C++17 onward that packages the most common cases as ready-to-use building blocks, alongside xsimd for portable SIMD.
As a concrete case study, we walk through a real scientific-computing kernel — a workload of the kind that has historically demanded hand-tuned assembly — and show how these abstractions get within a small fraction of theoretical peak throughput while keeping the source readable, portable, and maintainable, with no inline assembly, no compiler builtins, and no external code-generation step. Scientific computing does not have to choose between performance and engineering quality, and with the right abstractions, development becomes dramatically more effective.
