š§ Detective Work: First Case
Overview
This puzzle presents a crashing GPU program where your task is to identify the
issue using only (cuda-gdb) debugging tools, without examining the source
code. Apply your debugging skills to solve the mystery!
Prerequisites: Complete Mojo GPU Debugging Essentials to understand CUDA-GDB setup and basic debugging commands. Make sure youāve run:
pixi run -e nvidia setup-cuda-gdb
This auto-detects your CUDA installation and sets up the necessary links for GPU debugging.
Key concepts
In this debugging challenge, youāll learn about:
- Systematic debugging: Using error messages as clues to find root causes
- Error analysis: Reading crash messages and stack traces
- Hypothesis formation: Making educated guesses about the problem
- Debugging workflow: Step-by-step investigation process
Running the code
First, examine the kernel without looking at the complete code:
def add_10(
output: Pointer[Scalar[dtype], MutAnyOrigin],
a: Pointer[Scalar[dtype], MutAnyOrigin],
):
var i = thread_idx.x
output[unsafe_offset=i] = a[unsafe_offset=i] + 10.0
To experience the bug firsthand, run the following command in your terminal
(pixi only):
pixi run -e nvidia p09 --first-case
Youāll see output like this when the program crashes:
First Case: Try to identify what's wrong without looking at the code!
CUDA call failed: CUDA_ERROR_MISALIGNED_ADDRESS (misaligned address)
To get more accurate error information, set MODULAR_DEBUG=device-sync-mode.
#0 0x00007cc268dc4d1e (.../.pixi/envs/nvidia/lib/libKGENCompilerRTShared.so+0x112d1e)
...
#12 0x00007cc26a4fc328 AsyncRT_DeviceContext_release (.../.pixi/envs/nvidia/lib/libAsyncRTMojoBindings.so+0x45328)
#13 0x00007cc21280df9d
.../.pixi/envs/nvidia/bin/mojo: error: execution crashed
To get a symbolicated stack trace, compile your program using `mojo build` with debug info enabled (e.g., `-debug-level=line-tables`) and execute it separately.
Your task: detective work
Challenge: Without looking at the code yet, what would be your debugging strategy to investigate this crash?
Start with:
pixi run -e nvidia mojo debug --cuda-gdb --break-on-launch problems/p09/p09.mojo --first-case
Tips
- Read the crash message carefully - a
CUDA_ERROR_*_ADDRESSerror means the GPU tried to read or write through an address it canāt use - Check the breakpoint information - Look at the function parameters shown when CUDA-GDB stops
- Inspect all pointers systematically - Use
printto examine each pointer parameter - Compare the addresses - Do
outputandalook like they point into the same kind of memory? - Let the kernel run - A value the debugger prints through a pointer doesnāt prove the GPU can read it. Continue and see where the kernel stops
- Apply the systematic approach - Like a detective, follow the evidence from symptom to root cause
- Compare valid vs invalid patterns - If one pointer works and another doesnāt, focus on the broken one
š” Investigation & Solution
Step-by-step investigation with CUDA-GDB
Launch the debugger
pixi run -e nvidia mojo debug --cuda-gdb --break-on-launch problems/p09/p09.mojo --first-case
Examine the breakpoint information
When CUDA-GDB stops, it immediately shows valuable clues:
(cuda-gdb) run
CUDA thread hit application kernel entry function breakpoint, p09_add_10_...
<<<(1,1,1),(4,1,1)>>> (output=0x302000000, a=0x7ffff68575f0)
at /home/ubuntu/workspace/mojo-gpu-puzzles/problems/p09/p09.mojo:38
38 var i = thread_idx.x
š First Clue: The function signature shows
(output=0x302000000, a=0x7ffff68575f0)
outputpoints into the region where this programās GPU buffers liveapoints somewhere else entirely. On Linux,0x7fff...is the top of a processās CPU address space, where its stack and shared libraries live, not where GPU allocations come from
The exact value of a changes from run to run. What stays the same is that it
doesnāt look like output.
Systematic variable inspection
(cuda-gdb) next
39 output[unsafe_offset=i] = a[unsafe_offset=i] + 10.0
(cuda-gdb) print i
$1 = {0}
(cuda-gdb) print output
$2 = (!kgen.scalar<f32> * @register) 0x302000000
(cuda-gdb) print a
$3 = (!kgen.scalar<f32> * @register) 0x7ffff68575f0
Evidence Gathering:
- ā
Thread index
i=0is valid - ā
Result pointer
0x302000000is a GPU buffer address - ā Input pointer
0x7ffff68575f0isnāt
A trap: the debugger can read what the GPU canāt
(cuda-gdb) print output[0]
$4 = {0}
(cuda-gdb) print a[0]
$5 = {-1.34493767e+33}
The debugger happily prints a value through a, because it reads memory from
the host side. That value is garbage, and it doesnāt mean the GPU can read the
same address. Donāt treat a printed value as proof that a pointer is good.
Confirm the problem
Let the kernel run:
(cuda-gdb) continue
Continuing.
CUDA Exception: Warp Illegal Address
The exception was triggered at PC 0x7fffc127d5f0 p09_add_10_... (p09.mojo:39)
Thread 1 "mojo" received signal CUDA_EXCEPTION_14, Warp Illegal Address.
Smoking Gun: The GPU faults on line 39, the line that reads through a.
output was never the problem.
Root cause analysis
The Problem: Now if we look at the code
for --first-case, we see that the host code never allocates any GPU memory
for the input:
# A `DeviceBuffer` with 0 elements. Nothing is allocated, so there's no
# GPU address to give the kernel.
var input_buf = ctx.enqueue_create_buffer[dtype](0)
Why This Crashes:
ctx.enqueue_create_buffer[dtype](0)creates aDeviceBufferwith zero (0) elements.- Since there are no elements for which to allocate memory, the buffer has no GPU address.
- The kernel still receives something for
a, but it isnāt a GPU address. In this release itās a leftover CPU-side value like0x7ffff68575f0. - When the kernel evaluates
a[unsafe_offset=i], the GPU faults. Which CUDA error you see at the top depends on the bad address, which is why it can differ between Mojo versions.
The fix
Replace the empty buffer with a proper allocation:
# Wrong: allocates nothing, so the kernel gets no usable address
var input_buf = ctx.enqueue_create_buffer[dtype](0)
# Correct: allocate and initialize actual GPU memory for safe processing
var input_buf = ctx.enqueue_create_buffer[dtype](SIZE)
input_buf.enqueue_fill(0)
Key debugging lessons
Pattern Recognition:
- Pointers into the same kind of memory look alike. Here the GPU buffer is
0x302000000, and an address that looks nothing like it deserves suspicion 0x0is a null pointer, and0x7fff...on Linux is CPU-side memory. Neither belongs in a kernel argument
Debugging Strategy:
- Read crash messages - They often hint at the problem type
- Check function parameters - CUDA-GDB shows them at breakpoint entry
- Inspect all pointers - Compare addresses to identify invalid ones
- Let the kernel fault - The line where it stops names the bad access; a value the debugger prints proves nothing about the GPU
- Trace back to allocation - Find where the problematic pointer was created
š” Key Insight: This type of invalid pointer bug is extremely common in GPU programming. The systematic CUDA-GDB investigation approach you learned here applies to debugging many other GPU memory issues, race conditions, and kernel crashes.
Next steps: from crashes to silent bugs
Youāve learned crash debugging! You can now:
- Systematically investigate GPU crashes using error messages as clues
- Identify invalid pointer bugs through pointer address inspection
- Use CUDA-GDB effectively for memory-related debugging
Your next challenge: Detective Work: Second Case
But what if your program doesnāt crash? What if it runs perfectly but produces wrong results?
The Second Case presents a completely different debugging challenge:
- No crash messages to guide you
- No obvious pointer problems to investigate
- No stack traces pointing to the issue
- Just wrong results that need systematic investigation
New skills youāll develop:
- Logic bug detection - Finding algorithmic errors without crashes
- Pattern analysis - Using incorrect output to trace back to root causes
- Execution flow debugging - When variable inspection fails due to optimizations
The systematic investigation approach you learned here - reading clues, forming hypotheses, testing systematically - forms the foundation for debugging the more subtle logic errors ahead.