torch.compile Programming Model

Created On: Jul 28, 2025 | Last Updated On: Jul 29, 2025

The torch.compile programming model:

1. Clarifies some internal behaviors of torch.compile so that one can better predict compiler behavior on user code and
2. Provides ways for one to take more fine-grained control over torch.compile.

By understanding the torch.compile programming model, one can systematically unblock themselves when encountering issues with torch.compile.

• Dynamo Core Concepts
  • Dynamo Tracing
  • Graph Breaks
  • Guards
  • Recompilations
  • Dynamic Shapes
• Working with Graph Breaks
  • Use fullgraph=True to Identify and Eliminate Graph Breaks
    • Strategy 1: Rewrite the unsupported code to use features supported by Dynamo
    • Strategy 2: Pure functions can always be compiled via an escape hatch.
    • Strategy 3: Don’t compile the code
  • Common Graph Breaks
    • Incorrect Code
    • Data-dependent operations
    • Printing and logging
  • Use torch._dynamo.nonstrict_trace
  • Custom Operators
  • Working with fullgraph=False
    • Where to apply torch.compile?
      • compile(model) vs model.compile()
    • Disabling and Suppressing Errors
    • Toggling error_on_graph_break
      • error_on_graph_break(False) example
      • error_on_graph_break(True) example
      • error_on_graph_break nesting behavior
      • Interaction with fullgraph
      • Summary of fullgraph=True/False vs error_on_graph_break
    • Nested Graph Breaks
    • Skipped Functions
      • Graph Break in a Loop
      • Graph Break in a Context Manager
      • Graph Break in a Try Block
      • Hitting a Recompilation Limit
      • Compiler Errors
      • Dealing with Skipped Functions
• Non-strict Tracing Programming Model
  • Pure Functions
    • Example 1: No explicit input (e.g. accesses global tensor)
    • Example 2: Side effect (printing)
    • Example 3: Side effect (input list mutation)
    • No direct data_ptr manipulation
  • Specialization and Constants
• Dealing with Recompilations
  • Is Dynamic Shapes Enabled?
  • Wrapping Constants with Tensors
  • Changing the Cache Size Limit
  • Graph Breaking to Reduce Recompilation Costs
• tlparse / TORCH_TRACE
  • TORCH_LOGS
  • tlparse vs. TORCH_LOGS
• Reporting Issues
  • Ablation
  • Bisecting
  • Creating a reproducer