🏁 Debugging Race Conditions

alias SIZE = 2
alias BLOCKS_PER_GRID = 1
alias THREADS_PER_BLOCK = (3, 3)
alias dtype = DType.float32
alias layout = Layout.row_major(SIZE, SIZE)


fn shared_memory_race(
    output: LayoutTensor[mut=True, dtype, layout],
    a: LayoutTensor[mut=False, dtype, layout],
    size: Int,
):
    """Fixed: sequential access with barriers eliminates race conditions."""
    row = thread_idx.y
    col = thread_idx.x

    shared_sum = tb[dtype]().row_major[1]().shared().alloc()

    # Only thread 0 does all the accumulation work to prevent races
    if row == 0 and col == 0:
        # Use local accumulation first, then single write to shared memory
        local_sum = Scalar[dtype](0.0)
        for r in range(size):
            for c in range(size):
                local_sum += rebind[Scalar[dtype]](a[r, c])

        shared_sum[0] = local_sum  # Single write operation

    barrier()  # Ensure thread 0 completes before others read

    # All threads read the safely accumulated result after synchronization
    if row < size and col < size:
        output[row, col] = shared_sum[0]

Understanding what went wrong

The race condition problem pattern

The original failing code had this critical line:

shared_sum[0] += a[row, col]  # RACE CONDITION!

This single line creates multiple hazards among the 4 valid threads:

1. Thread (0,0) reads shared_sum[0] (value: 0.0)
2. Thread (0,1) reads shared_sum[0] (value: 0.0) ← Read-after-write hazard!
3. Thread (0,0) writes back 0.0 + 0
4. Thread (1,0) writes back 0.0 + 2 ← Write-after-write hazard!

Why the test failed

• Multiple threads corrupt each other’s writes during the += operation
• The += operation gets interrupted, causing lost updates
• Expected sum of 6.0 (0+1+2+3), but race conditions resulted in 0.0
• The barrier() comes too late - after the race condition already occurred

What are race conditions?

Race conditions occur when multiple threads access shared data concurrently, and the result depends on the unpredictable timing of thread execution.

Key characteristics:

• Non-deterministic behavior: Same code can produce different results on different runs
• Timing-dependent: Results depend on which thread “wins the race”
• Hard to reproduce: May only manifest under specific conditions or hardware

GPU-specific dangers

Massive parallelism impact:

• Warp-level corruption: Race conditions can affect entire warps (32 threads)
• Memory coalescing issues: Races can disrupt efficient memory access patterns
• Kernel-wide failures: Shared memory corruption can affect the entire GPU kernel

Hardware variations:

• Different GPU architectures: Race conditions may manifest differently across GPU models
• Memory hierarchy: L1 cache, L2 cache, and global memory can all exhibit different race behaviors
• Warp scheduling: Different thread scheduling can expose different race condition scenarios

Strategy: Single writer pattern

The key insight is to eliminate concurrent writes to shared memory:

1. Single writer: Only one thread (thread at position (0,0)) does all accumulation work
2. Local accumulation: Thread at position (0,0) uses a local variable to avoid repeated shared memory access
3. Single shared memory write: One write operation eliminates write-write races
4. Barrier synchronization: Ensures writer completes before others read
5. Multiple readers: All threads safely read the final result

Step-by-step solution breakdown

Step 1: Thread identification

if row == 0 and col == 0:

Use direct coordinate check to identify thread at position (0,0).

Step 2: Single-threaded accumulation

if row == 0 and col == 0:
    local_sum = Scalar[dtype](0.0)
    for r in range(size):
        for c in range(size):
            local_sum += rebind[Scalar[dtype]](a[r, c])
    shared_sum[0] = local_sum  # Single write operation

Only thread at position (0,0) performs all accumulation work, eliminating race conditions.

Step 3: Synchronization barrier

barrier()  # Ensure thread (0,0) completes before others read

All threads wait for thread at position (0,0) to finish accumulation.

Step 4: Safe parallel reads

if row < size and col < size:
    output[row, col] = shared_sum[0]

All threads can safely read the result after synchronization.

Important note on efficiency

This solution prioritizes correctness over efficiency. While it eliminates race conditions, using only thread at position (0,0) for accumulation is not optimal for GPU performance - we’re essentially doing serial computation on a massively parallel device.

Coming up in Puzzle 11: Pooling: You’ll learn efficient parallel reduction algorithms that leverage all threads for high-performance summation operations while maintaining race-free execution. This puzzle teaches the foundation of correctness first - once you understand how to avoid race conditions, Puzzle 11 will show you how to achieve both correctness AND performance.

Verification

pixi run compute-sanitizer --tool racecheck mojo solutions/p10/p10.mojo --race-condition

Expected output:

========= COMPUTE-SANITIZER
out shape: 2 x 2
Running race condition example...
out: HostBuffer([6.0, 6.0, 6.0, 6.0])
expected: HostBuffer([6.0, 6.0, 6.0, 6.0])
✅ Race condition test PASSED! (racecheck will find hazards)
========= RACECHECK SUMMARY: 0 hazards displayed (0 errors, 0 warnings)

✅ SUCCESS: Test passes and no race conditions detected!