Are Mutexes Slow?

Jon Gjengset; 2026

Have you heard these before?

Mutexes are slow.

Reader-writer locks are best for read-heavy workloads.

Lock-free data structures are always faster.

Our test: read a shared counter

let counter = Arc::new(Mutex::new(0u64));

loop {
    let mut guard = counter.lock().unwrap();
    std::hint::black_box(*guard);
}

Part 1: The Mutex

More threads = WORSE performance

Part 2: The reader-writer lock

RwLock barely beats Mutex

Interlude: CPU caches

Each core has private caches

┌─────────────────────────────────────────────┐
│                    RAM                      │
└─────────────────────────────────────────────┘
                      ↑
┌─────────────────────────────────────────────┐
│              L3 Cache (shared)              │
└─────────────────────────────────────────────┘
         ↑                           ↑
┌─────────────────┐         ┌─────────────────┐
│   L1/L2 Cache   │         │   L1/L2 Cache   │
│     Core 0      │         │     Core 1      │
└─────────────────┘         └─────────────────┘

What happens when two cores cache the same memory and one wants to write?

Cache coherence: the MESI protocol

Hardware tracks the state of each cache line.

State Meaning
Modified Only copy, dirty
Exclusive Only copy, clean
Shared Multiple caches have it
Invalid No valid copy

Reader-writer read lock requires a WRITE

pub fn read(&self) -> RwLockReadGuard {
    self.reader_count.fetch_add(1, ...);
}

Two readers acquiring a read lock

Step Core 0 S Core 1 S note
0 E I
1 fetch_add(1) M I No cross-core traffic needed!
2 M wants to lock I Core 1 sends write request
3 writeback I fetch_add(1) M Core 0 sends value to Core 1
4 wants to unlock I M Core 0 sends write request
5 fetch_sub(1) M writeback I Core 1 sends value to Core 0

Cache line ping-pong is expensive

Core 0: acquire (write)
        → invalidate Core 1

Core 1: acquire (write)
        → invalidate Core 0

Core 0: release (write)
        → invalidate Core 1
Operation Latency
L1 hit ~1 ns
L2 hit ~4 ns
L3 hit ~13 ns
Cross-core ~30+ ns
Main memory ~100 ns

Each invalidation is ~30+ ns - approaching main memory latency!

Short critical sections hurt more

Long work:

lock          10 cycles
work        1000 cycles
unlock        10 cycles
cache miss   100 cycles
─────────────────────────
overhead      ~11%

Short work:

lock          10 cycles
work          10 cycles
unlock        10 cycles
cache miss   100 cycles
─────────────────────────
overhead      ~92%

Part 4: The left-right data structure

What if readers never wrote to shared state?

Let's keep two copies of the data

One for all the readers, and one for the writer (if any).

┌──────────────────┐     ┌──────────────────┐
│    Left copy     │<-+  │    Right copy    │
│                  │  |  │  (writer-owned)  │
└──────────────────┘  |  └──────────────────┘
                      |
                (read pointer)

Let's keep two copies of the data

Swap the pointer when there are updates.

┌──────────────────┐     ┌──────────────────┐
│    Left copy     │  +->│    Right copy    │
│  (writer-owned)  │  |  │                  │
└──────────────────┘  |  └──────────────────┘
                      |
                (read pointer)

left-right reads scale linearly

Rare writes is the key

A debugging story: the 4-core cliff

While benchmarking left-right 0.11.6, I hit a mysterious performance anomaly:

ReadersOps/secExpected
1213M213M
2426M426M
3628M639M
457M852M
8945M1.7B

11x slower than expected at exactly 4 cores.

The culprit: false sharing

Each reader has its own epoch counter,
but they ended up on the same cache line!

The fix:

#[repr(align(64))]
struct PaddedEpoch(AtomicUsize);

The lesson: Even "lock-free" code can suffer from cache coherence.

Part 5: Choosing Wisely

left-right is NOT a drop-in replacement

  • 2x memory usage (two copies)
  • Readers see stale data during publish
  • Single writer only (need external mutex)
  • Operations must be deterministic
  • Writer waits for all readers to exit

Every solution has trade-offs

Primitive Read Scaling Write Cost Consistency
Mutex None Low Linearizable
Reader-writer lock For long critical sections Low Linearizable
left-right Excellent High Eventual

Questions to ask yourself

  1. What's my read/write ratio?
  2. How long is my critical section?
  3. How many threads will contend?
  4. Can I tolerate stale reads?
  5. Do I need linearizability?

The real lesson: it's not about locks

Coordination is expensive because of cache coherence, not because of locks.

Understanding cache topology and shared writes
is the key to scalable concurrent code.

Further reading

Questions?

Benchmarks & slides: https://github.com/jonhoo/are-mutexes-slow

Speaker notes: - Set the stage: common wisdom says "mutexes are slow" - We'll see that the answer is more nuanced - Understanding WHY is the key takeaway

Speaker notes: - These statements sound reasonable - We'll test them empirically - Spoiler: reality is more complicated

Speaker notes: - Simplest possible concurrent operation - Assume that there are _no_ writes for now - Vary: thread count, read/write ratio, critical section length ("critical section" = the code between lock() and unlock()) - Measure: operations per second

Speaker notes: - 1 thread: ~287M ops/sec (uncontended!) - 2+ threads: drops to ~21-29M ops/sec - That's a 10x+ SLOWDOWN just from adding a second thread - Not just "doesn't scale" - actively gets slower

Speaker notes: - Still the same declining trend - "Multiple readers can hold the lock simultaneously" - If 99% reads, RwLock should help... right? - RwLock reads scale slightly better than mutex - But still plateau quickly - something else is going on

Speaker notes: - CPUs don't cache individual bytes - they cache 64-byte "lines" - If Core 0 and Core 1 both have the same cache line cached - And Core 0 wants to write to it - We need a protocol to keep the caches consistent - That protocol is MESI

Speaker notes: - MESI tracks whether a cache line is shared across cores - If Core 0 modifies a cache line that Core 1 has cached, Core 1's copy is now stale. We must invalidate it. - To write when Shared: must invalidate ALL other copies - Wait for acknowledgment from every core - THEN you can write - This is the key insight: writes to shared data cause expensive cross-core communication

Speaker notes: - fetch_add is a WRITE operation - Even though you're "just reading" the data, you're writing to the lock - Every reader fights over the same cache line containing reader_count

Speaker notes: - Walk through each step - Each read causes **two** cache line transfers at high contention - This is "cache line ping-pong" - Each transition costs ~30+ ns - Technically: bus snooping

Speaker notes: - An uncontended mutex is ~17 ns - cheap! - Contention is the problem, not the lock itself - Both Mutex and RwLock do 2 atomics per lock/unlock cycle - Mutex: only one thread holds lock → unlock is uncontended (you have exclusive access to lock state) - RwLock: multiple readers hold lock simultaneously → all contend on reader_count for both acquire AND release - While you hold a read lock, other readers are still doing fetch_add/fetch_sub on the same cache line - This means RwLock readers experience MORE contention than Mutex users for short critical sections

Speaker notes: - For short critical sections: all time is spent waiting for the M - For long critical sections: multiple concurrent read locks still help over Mutex! - Our counter benchmark is worst case: minimal work - This is why real-world code often doesn't show these problems - But high-contention hot paths do

Speaker notes: - We've seen the problem: cache line ping-pong via MESI - Caused by every reader writing to the same cache line (reader_count) - fetch_add is an atomic read-modify-write operation - Can we make each reader write only to its OWN cache line (not shared)

Speaker notes: - Readers access one copy, writer modifies the other - Swap atomically when ready

Speaker notes: - How do we know no readers are left over in the left copy? - Each reader has per-thread counter (on its OWN cache line), which they tick each time they read (through the pointer) - Readers DO write (to their counter/epoch), but not to SHARED state - Writer has to wait until all epochs have "ticked" at least once, indicating that the reader must have observed the pointer swap. - No cache ping-pong between readers, only with writer! - Lots of nuances, like idle readers and applying updates to both copies - Reads become "wait-free" = readers are guaranteed to complete, never block

Speaker notes: - RwLock: ~19-60M reads/sec, doesn't scale (actually gets worse!) - left-right: 206M at 1 reader → 2.86B at 14 readers - At 14 readers: left-right is ~150x faster than RwLock

Speaker notes: - Real workloads have both reads and writes - left-right advantage grows with read/write ratio - At 9:1 (90% reads), left-right is dramatically faster - But: left-right has higher write cost (must update both copies) - Recall: reads contend on reader_count, but writes don't (writers just wait for reader_count → 0) - **9R:1W**: 90% of operations contend on reader_count - **3R:1W**: 75% of operations contend on reader_count - **Counterintuitive observation:** RwLock gets *worse* the more readers there are, because reads (not writes) cause the contention

Speaker notes: - Perfect linear scaling 1-3 cores - Catastrophic drop at exactly 4 cores - Not CCD crossing (that's at core 8 on my 7950X3D) - Not SMT (using physical cores only)

Speaker notes: - If epochs allocated adjacently, they share cache lines → ping-pong - Why 4 specifically? Allocator/slab layout quirks - Now scales linearly: 14 readers = 14x throughput - Contributed upstream (to myself) - "Lock-free" doesn't mean "contention-free" - This is why it's important to know about this stuff!

Speaker notes: - Not linearizable - this matters! - "Linearizable" = operations appear instant and globally ordered (if you write, then read, you see your write) - Instead, left-right is "eventual" = you might see stale data temporarily - Memory doubles (two full copies) - Good for: config, caches, lookup tables - Bad for: financial transactions, anything needing consistency

Speaker notes: - There's no "best" - only appropriate for your workload - left-right: 1000:1+ read/write, staleness OK - Mutex: simple, works, often good enough Aside: Sharded reader-writer locks (drwlock, BRAVO) exist too - Similar cache behavior to left-right (per-thread counters) - But readers can still block if a writer is present - Linearizable, unlike left-right

Speaker notes: - Most code: just use Mutex - Longer read sections: reader-writer lock - Hot path with many readers: consider alternatives - Always measure before optimizing

Speaker notes: - std::atomic operations still cause cache line invalidation - You're trading lock overhead for atomic overhead - Real scalability comes from avoiding shared writes entirely

Speaker notes: - Share the repo so people can run benchmarks on their own hardware - Results vary by CPU architecture, core count, etc.