🔍 Detective Work: Second Case
Overview
Building on your crash debugging skills from the First Case, you’ll now face a completely different challenge: a logic bug that produces incorrect results without crashing.
The debugging shift:
- First Case: Clear crash signals (a CUDA address error) guided your investigation
- Second Case: No crashes, no error messages - just subtly wrong results that require detective work
This intermediate-level debugging challenge covers investigating
algorithmic errors using TileTensor operations, where the program runs
successfully but produces wrong output - a much more common (and trickier)
real-world debugging scenario.
Prerequisites: Complete Mojo GPU Debugging Essentials and Detective Work: First Case to understand CUDA-GDB workflow and systematic debugging techniques. Make sure you run the setup:
pixi run -e nvidia setup-cuda-gdb
Key concepts
In this debugging challenge, you’ll learn about:
- TileTensor debugging: Investigating structured data access patterns
- Logic bug detection: Finding algorithmic errors that don’t crash
- Loop boundary analysis: Understanding iteration count problems
- Result pattern analysis: Using output data to trace back to root causes
Running the code
First, examine the kernel without looking at the complete code:
def process_sliding_window[
Engine: TensorEngine,
](
output: TileTensor[
mut=True, dtype, VectorLayout, MutAnyOrigin, Engine=Engine
],
a: TileTensor[
mut=False, dtype, VectorLayout, ImmutAnyOrigin, Engine=Engine
],
) where (Engine.element_size == 1):
var thread_id = thread_idx.x
# Each thread processes a sliding window of 3 elements
var window_sum = Scalar[dtype](0.0)
# Sum elements in sliding window: [i-1, i, i+1]
for offset in range(ITER):
var idx = Int(thread_id) + offset - 1
if 0 <= idx < SIZE:
window_sum += rebind[Scalar[dtype]](a[idx])
output[thread_id] = window_sum
To experience the bug firsthand, run the following command in your terminal
(pixi only):
pixi run -e nvidia p09 --second-case
You’ll see output like this - no crash, but wrong results:
This program computes sliding window sums for each position...
Input array: [0, 1, 2, 3]
Computing sliding window sums (window size = 3)...
Each position should sum its neighbors: [left + center + right]
Actual result: HostBuffer([0.0, 1.0, 3.0, 5.0])
Expected: [1.0, 3.0, 6.0, 5.0]
[FAIL] Test FAILED - Sliding window sums are incorrect!
Check the window indexing logic...
Your task: detective work
Challenge: The program runs without crashing but produces consistently wrong results. Without looking at the code, what would be your systematic approach to investigate this logic bug?
Think about:
- What pattern do you see in the wrong results?
- How would you investigate a loop that might not be running correctly?
- What debugging strategy works when you can’t inspect variables directly?
- How can you apply the systematic investigation approach from First Case when there are no crash signals to guide you?
Start with:
pixi run -e nvidia mojo debug --cuda-gdb --break-on-launch problems/p09/p09.mojo --second-case
GDB command shortcuts (faster debugging)
Use these abbreviations to speed up your debugging session:
| Short | Full | Usage Example |
|---|---|---|
r | run | (cuda-gdb) r |
n | next | (cuda-gdb) n |
c | continue | (cuda-gdb) c |
b | break | (cuda-gdb) b 63 |
p | print | (cuda-gdb) p thread_id |
q | quit | (cuda-gdb) q |
All debugging commands below use these shortcuts for efficiency!
Tips
- Pattern analysis first - Look at the relationship between expected and actual results (what’s the mathematical pattern in the differences?)
- Focus on execution flow - Count loop iterations when variables aren’t accessible
- Use simple breakpoints - Complex debugging commands often fail with optimized code
- Mathematical reasoning - Work out what each thread should access vs what it actually accesses
- Missing data investigation - If results are consistently smaller than expected, what might be missing?
- Host output verification - The final results often reveal the pattern of the bug
- Algorithm boundary analysis - Check if loops are processing the right number of elements
- Cross-validate with working cases - Why does thread 3 work correctly but others don’t?
💡 Investigation & Solution
Step-by-step investigation with CUDA-GDB
Phase 1: Launch and initial analysis
Step 1: Start the debugger
pixi run -e nvidia mojo debug --cuda-gdb --break-on-launch problems/p09/p09.mojo --second-case
Step 2: analyze the symptoms first
Before diving into the debugger, examine what we know:
Actual result: [0.0, 1.0, 3.0, 5.0]
Expected: [1.0, 3.0, 6.0, 5.0]
🔍 Pattern Recognition:
- Thread 0: Got 0.0, Expected 1.0 → Missing 1.0
- Thread 1: Got 1.0, Expected 3.0 → Missing 2.0
- Thread 2: Got 3.0, Expected 6.0 → Missing 3.0
- Thread 3: Got 5.0, Expected 5.0 → ✅ Correct
Initial Hypothesis: Each thread is missing some data, but thread 3 works correctly.
Phase 2: Entering the kernel
Step 3: Observe the breakpoint entry
Based on the real debugging session, here’s what happens:
(cuda-gdb) r
Starting program: .../mojo run -O0 --debug-level=full problems/p09/p09.mojo --second-case
This program computes sliding window sums for each position...
Input array: [0, 1, 2, 3]
Computing sliding window sums (window size = 3)...
Each position should sum its neighbors: [left + center + right]
[Switching focus to CUDA kernel 0, grid 1, block (0,0,0), thread (0,0,0), device 0, sm 0, warp 0, lane 0]
CUDA thread hit application kernel entry function breakpoint, p09_process_sliding_window_...
<<<(1,1,1),(4,1,1)>>> ()
at /home/ubuntu/workspace/mojo-gpu-puzzles/problems/p09/p09.mojo:52
52 a: TileTensor[
Step 4: Navigate to the main logic
The entry breakpoint stops in the kernel’s signature, and stepping with n
from there leaves the kernel without stopping in its body. Set a breakpoint on
the first line of the loop body instead, and continue to it:
(cuda-gdb) b 63
Breakpoint 1 at 0x7fffc127eb50: file problems/p09/p09.mojo, line 63.
(cuda-gdb) c
Continuing.
CUDA thread hit Breakpoint 1, p09_process_sliding_window_...
<<<(1,1,1),(4,1,1)>>> (output=..., a=...)
at /home/ubuntu/workspace/mojo-gpu-puzzles/problems/p09/p09.mojo:63
63 var idx = Int(thread_id) + offset - 1
Step 5: Test variable accessibility - crucial discovery
(cuda-gdb) p thread_id
$1 = <unavailable>
❌ Problem: thread_id is not accessible.
(cuda-gdb) p window_sum
$2 = <optimized out>
❌ Problem: window_sum is not accessible.
(cuda-gdb) p a[0]
Attempt to take address of value not located in memory.
❌ Problem: Direct TileTensor indexing doesn’t work.
(cuda-gdb) p a
$3 = {_storage = 0x302000000, layout = {_shape = {m0 = {<No data fields>}},
_stride = {m0 = {<No data fields>}}}}
(cuda-gdb) p a._storage[0]@4
Cannot perform pointer math on incomplete type "!kgen.none", try casting to a known type, or void *.
(cuda-gdb) p ((float*)a._storage)[0]@4
$4 = {0, 1, 2, 3}
🎯 BREAKTHROUGH: p a shows that a TileTensor keeps its data behind a
_storage pointer. The debugger doesn’t know the pointer’s element type, so
cast it to float* (the puzzle’s dtype is float32). Then
((float*)a._storage)[0]@4 shows the full input array! This is how we can
inspect TileTensor data.
Phase 3: The critical loop investigation
Step 6: Set up loop monitoring
The breakpoint on line 63 fires once at the start of every loop iteration, so counting its hits counts the iterations. The loop variable is still readable:
(cuda-gdb) p offset
$5 = {0}
🔍 We’re now inside the loop body, on its first iteration. Let’s count iterations manually.
Step 7: First loop iteration (offset = 0)
(cuda-gdb) n
64 if 0 <= idx < SIZE:
(cuda-gdb) n
62 for offset in range(ITER):
First iteration complete: Loop went from line 63 → 64 → back to 62. For
thread 0, idx is -1, so it skipped the sum. The loop continues.
Step 8: Second loop iteration (offset = 1)
(cuda-gdb) n
CUDA thread hit Breakpoint 1, p09_process_sliding_window_...
63 var idx = Int(thread_id) + offset - 1
(cuda-gdb) p offset
$6 = {1}
(cuda-gdb) n
64 if 0 <= idx < SIZE:
(cuda-gdb) n
65 window_sum += rebind[Scalar[dtype]](a[idx])
(cuda-gdb) n
64 if 0 <= idx < SIZE:
(cuda-gdb) n
62 for offset in range(ITER):
Second iteration complete: This time it went through the if-block (line 65).
Step 9: testing for third iteration
(cuda-gdb) n
67 output[thread_id] = window_sum
CRITICAL DISCOVERY: The loop exited after only 2 iterations! It went directly to line 67 instead of hitting our breakpoint at line 63 again.
Conclusion: The loop ran exactly 2 iterations and then exited.
Step 10: Complete kernel execution and context loss
(cuda-gdb) n
46 def process_sliding_window[
(cuda-gdb) n
[Switching to Thread 0x7ffff7bc8a40 (LWP 13617)]
0x00007fffefc2be19 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
(cuda-gdb) p output
No symbol "output" in current context.
(cuda-gdb) p offset
No symbol "offset" in current context.
🔍 Context Lost: After kernel completion, we lose access to kernel variables. This is normal behavior.
Phase 4: Root cause analysis
Step 11: Algorithm analysis from observed execution
From our debugging session, we observed:
- Loop Iterations: Only 2 iterations (offset = 0, offset = 1)
- Expected: A sliding window of size 3 should require 3 iterations (offset = 0, 1, 2)
- Missing: The third iteration (offset = 2)
Looking at what each thread should compute:
- Thread 0: window_sum = a[-1] + a[0] + a[1] = (boundary) + 0 + 1 = 1.0
- Thread 1: window_sum = a[0] + a[1] + a[2] = 0 + 1 + 2 = 3.0
- Thread 2: window_sum = a[1] + a[2] + a[3] = 1 + 2 + 3 = 6.0
- Thread 3: window_sum = a[2] + a[3] + a[4] = 2 + 3 + (boundary) = 5.0
Step 12: Trace the actual execution for thread 0
With only 2 iterations (offset = 0, 1):
Iteration 1 (offset = 0):
idx = thread_id + offset - 1 = 0 + 0 - 1 = -1if 0 <= idx < SIZE:→if 0 <= -1 < 4:→ False- Skip the sum operation
Iteration 2 (offset = 1):
idx = thread_id + offset - 1 = 0 + 1 - 1 = 0if 0 <= idx < SIZE:→if 0 <= 0 < 4:→ Truewindow_sum += a[0]→window_sum += 0
Missing Iteration 3 (offset = 2):
idx = thread_id + offset - 1 = 0 + 2 - 1 = 1if 0 <= idx < SIZE:→if 0 <= 1 < 4:→ Truewindow_sum += a[1]→window_sum += 1← THIS NEVER HAPPENS
Result: Thread 0 gets window_sum = 0 instead of window_sum = 0 + 1 = 1
Phase 5: Bug confirmation
Looking at the problem code, we find:
comptime ITER = 2 # ← BUG: Should be 3!
for offset in range(ITER): # ← Only 2 iterations: [0, 1]
var idx = Int(thread_id) + offset - 1 # ← Missing offset = 2
if 0 <= idx < SIZE:
window_sum += rebind[Scalar[dtype]](a[idx])
🎯 ROOT CAUSE IDENTIFIED: ITER = 2 should be ITER = 3 for a sliding
window of size 3.
The Fix: Change comptime ITER = 2 to comptime ITER = 3 in the source
code.
Key debugging lessons
When Variables Are Inaccessible:
- Focus on execution flow - Count breakpoint hits and loop iterations
- Use mathematical reasoning - Work out what should happen vs what does happen
- Pattern analysis - Let the wrong results guide your investigation
- Cross-validation - Test your hypothesis against multiple data points
Professional GPU Debugging Reality:
- Variable inspection often fails due to compiler optimizations
- Execution flow analysis is more reliable than data inspection
- Host output patterns provide crucial debugging clues
- Source code reasoning complements limited debugger capabilities
TileTensor Debugging:
- Even with TileTensor abstractions, underlying algorithmic bugs still manifest
- Focus on the algorithm logic rather than trying to inspect tensor contents
- Use systematic reasoning to trace what each thread should vs actually accesses
Key Insight: This type of off-by-one loop bug is extremely common in GPU programming. The systematic approach you learned here - combining limited debugger info with mathematical analysis and pattern recognition - is exactly how professional GPU developers debug when tools have limitations.
Next steps: from logic bugs to coordination deadlocks
You’ve learned logic bug debugging! You can now:
- ✅ Investigate algorithmic errors without crashes or obvious symptoms
- ✅ Use pattern analysis to trace wrong results back to root causes
- ✅ Debug with limited variable access using execution flow analysis
- ✅ Apply mathematical reasoning when debugger tools have limitations
Your final challenge: Detective Work: Third Case
But what if your program doesn’t crash AND doesn’t finish? What if it just hangs forever?
The Third Case presents the ultimate debugging challenge:
- ❌ No crash messages (like First Case)
- ❌ No wrong results (like Second Case)
- ❌ No completion at all - just infinite hanging
- ✅ Silent deadlock requiring advanced thread coordination analysis
New skills you’ll develop:
- Barrier deadlock detection - Finding coordination failures in parallel threads
- Multi-thread state analysis - Examining all threads simultaneously
- Synchronization debugging - Understanding thread cooperation breakdowns
The debugging evolution:
- First Case: Follow crash signals → Find memory bugs
- Second Case: Analyze result patterns → Find logic bugs
- Third Case: Investigate thread states → Find coordination bugs
The systematic investigation skills from both previous cases - hypothesis formation, evidence gathering, pattern analysis - become crucial when debugging the most challenging GPU issue: threads that coordinate incorrectly and wait forever.