Reverse-mode automatic differentiation (AD) computes exact gradients with respect to thousands of inputs in roughly the time of a single function evaluation. If you write numerical C++ for finance, scientific computing, or ML and you care about performance, this talk will show you where AD overhead actually comes from and what to do about it.
A straightforward operator-overloading implementation records every arithmetic operation onto a tape and replays it backward to propagate derivatives. On real numerical code this easily introduces 50-100x overhead, and most of that has nothing to do with calculus. It is heap allocation on every operation. It is cache misses walking the tape backward. It is branch mispredictions at memory chunk boundaries. It is recording tape entries for operations where none of the operands are even being differentiated.
We work through a series of C++ techniques that bring this down to around 4x on production code in finance. The two that matter most: a chunked arena allocator with cache-line alignment and branch-prediction hints that turns the recording hot path into a pointer bump and a store, and expression templates that collapse an entire right-hand side into one tape push with all derivatives accumulated in a compile-time-sized stack buffer. We pull up compiler output to show the abstraction really does compile away.
After the big two, we get into the smaller wins that are still worth knowing about: skipping zero adjoints during the backward sweep to exploit sparsity, picking derivative formulas at compile time that reuse the forward result instead of recomputing it, and demoting active-times-passive operations from binary to unary to halve their tape footprint. We also show how a should-record check propagated through the expression tree lets the tape skip entire statements when no operand is active, and how aggressive inlining lets the compiler eliminate these checks entirely for passive subexpressions, so the cost of asking is zero when the answer is no. We benchmark each one in isolation on four numerical workloads ranging from 8 to 161 sensitivities, so you can see what each technique is actually worth.
