📊 Benchmarking - Performance Analysis and Optimization

Overview

After learning elementwise, tiled, manual vectorization, and Mojo vectorize patterns, it’s time to measure their actual performance. Here’s how to use the built-in benchmarking system in p23.mojo to scientifically compare these approaches and understand their performance characteristics.

Key insight: Theoretical analysis is valuable, but empirical benchmarking reveals the true performance story on your specific hardware.

Running benchmarks

To execute the comprehensive benchmark suite:

pixi run p23 --benchmark

pixi run -e amd p23 --benchmark

pixi run -e apple p23 --benchmark

uv run poe p23 --benchmark

Your output will show performance measurements for each pattern (the run below is a B200 with MAX 26.5.0 / Mojo 1.0.0 — read the ranking, not the absolute times):

SIZE: 1024
simd_width: 4
Running P23 GPU Benchmarks...
SIMD width: 4
--------------------------------------------------------------------------------
Testing SIZE=16, TILE=4
Running elementwise_16_4
Running tiled_16_4
Running manual_vectorized_16_4
Running vectorized_16_4
--------------------------------------------------------------------------------
Testing SIZE=128, TILE=16
Running elementwise_128_16
Running tiled_128_16
Running manual_vectorized_128_16
--------------------------------------------------------------------------------
Testing SIZE=128, TILE=16, Vectorize within tiles
Running vectorized_128_16
--------------------------------------------------------------------------------
Testing SIZE=1048576 (1M), TILE=1024
Running elementwise_1M_1024
Running tiled_1M_1024
Running manual_vectorized_1M_1024
Running vectorized_1M_1024
| name                      | met (ms)              | iters |
| ------------------------- | --------------------- | ----- |
| elementwise_16_4          | 0.0045024             | 10    |
| tiled_16_4                | 0.0043072             | 10    |
| manual_vectorized_16_4    | 0.0041248             | 10    |
| vectorized_16_4           | 0.0040704             | 10    |
| elementwise_128_16        | 0.0040926999999999995 | 10    |
| tiled_128_16              | 0.0041567             | 10    |
| manual_vectorized_128_16  | 0.0042144             | 10    |
| vectorized_128_16         | 0.0042014999999999995 | 10    |
| elementwise_1M_1024       | 0.0054303             | 10    |
| tiled_1M_1024             | 0.2601023             | 10    |
| manual_vectorized_1M_1024 | 0.5849376000000001    | 10    |
| vectorized_1M_1024        | 0.1486304             | 10    |

Benchmarks completed!

Benchmark configuration

The benchmarking system uses Mojo’s built-in benchmark module:

from std.benchmark import Bench, BenchConfig, Bencher, BenchId, keep
from max.benchmark import bencher_iter_custom

var bench_config = BenchConfig(max_iters=10, num_warmup_iters=1)

• max_iters=10: Up to 10 iterations for statistical reliability
• num_warmup_iters=1: GPU warmup before measurement
• Check out the benchmark documentation

Benchmarking implementation essentials

Core workflow pattern

Each benchmark follows a streamlined pattern:

@parameter
def benchmark_pattern_parameterized[test_size: Int, tile_size: Int](mut b: Bencher) raises:
    var bench_ctx = DeviceContext()
    # Setup: Create buffers and initialize data
    @parameter
    def pattern_workflow(ctx: DeviceContext) raises:
      # Compute: Execute the algorithm being measured

    bencher_iter_custom(b, pattern_workflow, bench_ctx)
    # Prevent optimization: keep(out.unsafe_ptr())
    # Synchronize: ctx.synchronize()

Key phases:

1. Setup: Buffer allocation and data initialization
2. Computation: The actual algorithm being benchmarked
3. Prevent optimization: Critical for accurate measurement
4. Synchronization: Ensure GPU work completes

Critical: The keep() function keep(out.unsafe_ptr()) prevents the compiler from optimizing away your computation as “unused code.” Without this, you might measure nothing instead of your algorithm! This is essential for accurate GPU benchmarking because kernels are launched asynchronously.

Why custom iteration works for GPU

Standard benchmarking assumes CPU-style synchronous execution. GPU kernels launch asynchronously, so we need:

• GPU context management: Proper DeviceContext lifecycle
• Memory management: Buffer cleanup between iterations
• Synchronization handling: Accurate timing of async operations
• Overhead isolation: Separate setup cost from computation cost

Test scenarios and thread analysis

The benchmark suite tests three scenarios to reveal performance characteristics:

Thread utilization summary

All four patterns touch the same number of elements. Elementwise, manual and vectorize move SIMD_WIDTH elements per operation, so their op counts agree. The tiled pattern is the exception: elementwise launches it with a width of 1, so each of its tile_size loop iterations moves a single element and it issues SIMD_WIDTH times as many memory operations for the same data.

Performance characteristics by problem size

Small problems (SIZE=16):

• Launch overhead dominates (~0.004ms baseline)
• Thread count differences don’t matter
• Tiled/vectorize show slightly lower overhead

Medium problems (SIZE=128):

• Still overhead-dominated (~0.004ms for all)
• Performance differences nearly disappear
• Transitional behavior between overhead and computation

Large problems (SIZE=1M):

• Real algorithmic differences emerge
• Impact of uncoalesced loads becomes apparent
• Clear performance ranking appears

What the data shows

Based on empirical benchmark results across different hardware:

Performance rankings (large problems)

Key performance insights

For simple memory-bound operations: Maximum parallelism (elementwise) outperforms complex memory optimizations at scale.

Why elementwise wins:

• 262,144 threads provide excellent latency hiding
• Simple memory patterns achieve good coalescing
• Minimal overhead per thread
• Scales naturally with GPU core count

Why Mojo vectorize holds up:

• Automatic full-width loads without hand-written index arithmetic
• Balanced approach between parallelism and memory locality
• Good thread utilization without excessive complexity

Why tiled falls behind:

• Width-1 loads issue four times as many memory operations for the same data
• Uncoalesced access: adjacent threads read tiles a full tile_size apart, so a warp’s loads never merge into one transaction
• Locality within a tile doesn’t make up for that lost coalescing

Why manual vectorization comes last:

• Only 256 threads limit parallelism
• Complex indexing adds computational overhead
• Cache pressure from large chunks per thread
• Diminishing returns for simple arithmetic

Framework intelligence:

• Automatic iteration count adjustment, bounded by max_iters
• Statistical reliability across different execution times
• Handles thermal throttling and system variation

Interpreting your results

Reading the output table

| name                     | met (ms)           | iters |
| elementwise_1M_1024      | 0.0054303          | 10    |

• met (ms): Execution time for a single iteration
• iters: Number of iterations performed
• Compare within problem size: Same-size comparisons are most meaningful

Making optimization decisions

Choose patterns based on empirical evidence:

For production workloads:

• Large datasets (>100K elements): Elementwise typically optimal
• Small/startup datasets (<1K elements): Tiled or vectorize for lower overhead
• Development speed priority: Mojo vectorize for automatic optimization
• Avoid manual vectorization: Complexity rarely pays off for simple operations

Performance optimization workflow:

1. Profile first: Measure before optimizing
2. Test at scale: Small problems mislead about real performance
3. Consider total cost: Include development and maintenance effort
4. Validate improvements: Confirm with benchmarks on target hardware

Advanced benchmarking techniques

Custom test scenarios

Modify parameters to test different conditions:

# Different problem sizes
bench.bench_function[benchmark_elementwise_parameterized[1024, 32]](
    BenchId("elementwise_1024_32")
)
bench.bench_function[benchmark_elementwise_parameterized[64, 8]](
    BenchId("elementwise_64_8")
)

# Different tile sizes
bench.bench_function[benchmark_tiled_parameterized[256, 8]](
    BenchId("tiled_256_8")
)
bench.bench_function[benchmark_tiled_parameterized[256, 64]](
    BenchId("tiled_256_64")
)

Hardware considerations

Your results will vary based on:

• GPU architecture: SIMD width, core count, memory bandwidth
• System configuration: PCIe bandwidth, CPU performance
• Thermal state: GPU boost clocks vs sustained performance
• Concurrent workloads: Other processes affecting GPU utilization

Best practices summary

Benchmarking workflow:

1. Warm up GPU before critical measurements
2. Run multiple iterations for statistical significance
3. Test multiple problem sizes to understand scaling
4. Use keep() consistently to prevent optimization artifacts
5. Compare like with like (same problem size, same hardware)

Performance decision framework:

• Start simple: Begin with elementwise for memory-bound operations
• Measure don’t guess: Theoretical analysis guides, empirical data decides
• Scale matters: Small problem performance doesn’t predict large problem behavior
• Total cost optimization: Balance development time vs runtime performance

Next steps

With benchmarking skills:

• Profile real applications: Apply these patterns to actual workloads
• Advanced GPU patterns: Explore reductions, convolutions, and matrix operations
• Multi-GPU scaling: Understand distributed GPU computing patterns
• Memory optimization: Dive deeper into shared memory and advanced caching

💡 Key takeaway: Benchmarking transforms theoretical understanding into practical performance optimization. Use empirical data to make informed decisions about which patterns work best for your specific hardware and workload characteristics.

Looking ahead: when you need more control

The functional patterns in Part VI provide excellent performance for most workloads, but some algorithms require direct thread communication:

Algorithms that benefit from warp programming:

• Reductions: Sum, max, min operations across thread groups
• Prefix operations: Cumulative sums, running maximums
• Data shuffling: Reorganizing data between threads
• Cooperative algorithms: Where threads must coordinate closely

Performance preview:

In Part VII, we’ll revisit several algorithms from Part III and show how warp operations can:

• Simplify code: Replace complex shared memory patterns with single function calls
• Improve performance: Eliminate barriers and reduce memory traffic
• Enable new algorithms: Unlock patterns impossible with pure functional approaches

Coming up next: Part VII: Warp-Level Programming - starting with a dramatic reimplementation of Puzzle 14’s prefix sum.