Authors: Michael Katz (IBM), Harsha Kokel (IBM), Christian Muise (Queen's University), Shirin Sohrabi (IBM), Sarath Sreedharan (Colorado State University)
Venue: ICML 2026 Position Paper Track
📌 About This Guide
This guide summarizes practical recommendations from our ICML 2026 position paper, Make Planning Research Rigorous Again! Automated planning has developed more than six decades of methodology for defining planning problems, building planners, validating solutions, and evaluating systems.
As LLM-based planning becomes increasingly popular, many evaluations risk repeating pitfalls that the planning community has already encountered: unclear task semantics, benchmark contamination, mismatched baselines, unreliable validation, and incomplete reporting.
Our position: LLM-based planning is promising, but progress will be clearer, faster, and more trustworthy if insights, tools, benchmarks, and evaluation protocols from automated planning are incorporated correctly.
Who this guide is for
This guide is intended for researchers building LLM-based planners, reviewers evaluating LLM-planning papers, and practitioners designing benchmarks for planning, acting, tool use, or reasoning about actions and change.
Highest-priority recommendations
Clearly specify the planning problem and assumptions.
Use independently checkable evaluation whenever possible.
Compare against strong, relevant baselines with matching guarantees.
Evaluate on diverse benchmarks with difficulty scaling and novelty controls.
Report full experimental budgets, including time, memory, LLM calls, tool calls, retries, and repairs.
Counter-arguments — and our responses
The paper explicitly addresses three opposing positions. Here is a brief summary of each and how we respond:
LLM planning is a different problem.
In open-world, natural-language settings the state/action space is underspecified and transition dynamics are not explicitly provided, so enforcing classical evaluation norms may mis-measure progress.
Our response: Identify a well-defined variant that makes correctness checkable, then select baselines accordingly.
Utility beats guarantees; success is what matters.
Unsound methods paired with a validator and repeated until a valid solution is found can yield a practically reliable system—what matters is empirical performance under matched resource budgets.
Our response: Valid solutions may be extremely sparse due to combinatorial growth, so the expected number of samples required can explode, making generate-and-validate an ineffective general remedy. Report acceptance rates and budgets, and apply novelty controls before making broad planning claims.
Contamination is inevitable; use harder, more robust evaluations instead.
Focusing on "clean" static test sets can be brittle; fresh instance generation, distribution shifts, and interactive tasks are more robust to memorization.
Our response: We agree evaluations must be robust to contamination, but "harder" should not mean less well-defined or less diagnostic. "Harder" benchmarks often introduce ill-defined validation and capability entanglement, making it unclear what is being measured. The planning community already provides mature protocols for robust, diagnostic evaluation—these should be adopted rather than replaced.
Probabilistic planning: Action dynamics is probabilistic
Non-deterministic planning (FOND): Action dynamics is non-deterministic
Temporal planning: Actions have durations and temporal constraints
Numeric planning: Actions can affect numeric state variables
HTN planning: Hierarchical decomposition of high-level tasks into sub-tasks
Planning with preferences: Soft and hard constraints to optimize
Planning Languages
PDDL (Planning Domain Definition Language): the de facto standard representation language, introduced with the International Planning Competition (IPC) in 1998; works for classical planning, numeric, preferences, conformant, FOND, temporal, and more; separates knowledge into a PDDL domain and PDDL problem
RDDL/PPDDL: for probabilistic planning tasks
HDDL: for hierarchical planning
ANML: for timeline-based planning
Best Practices
Define Your State and Action Spaces Formally
â–¶
Identify a well-defined variant of your problem with mathematically precise state and action spaces. Why: Without formal definitions, you cannot verify correctness, compare methods fairly, or build upon results. How: Specify states, actions with preconditions/effects, goals, and transition functions explicitly—even if the state space is infinite, it must be well-defined.
Match Your Tools to Your Problem Structure
â–¶
Before using planning tools, verify your problem has sequential structure where action order matters. Why: Problems like SAT and Sudoku can be represented as planning but are better solved by different search procedures. How: Ask: Does the order of decisions matter? Are there action dependencies? If not, use CSP/SAT solvers as baselines instead of planners.
Problems, Algorithms, and Complexity
Understanding computational complexity and algorithmic guarantees is crucial for fair comparisons and meaningful results.
Best Practices
Always Validate Unsound Methods
â–¶
If your method doesn't guarantee correctness, pair it with an independent sound validator. Why: Unsound methods may produce invalid solutions. How:
Use established validators like VAL for PDDL
Separate validation from generation code
Report the complexity of your procedure, e.g., if repeat until passed validation, report how many times would run on expectation
Document what guarantees your method provides
Compare Methods with Matching Guarantees
â–¶
Only compare planners that solve the same computational problem (e.g., cost-optimal vs. cost-optimal). Why: Different problems have different complexity classes and require different algorithms. How:
Clearly state which problem you're solving
Select baselines targeting the same problem
If comparing across types, explicitly justify why
Report the guarantees each method provides
Understand and Justify The Planner Choices
â–¶
For cost-optimal planning, use A* only with admissible heuristics. For non-optimal planning, you rarely want to use A*. Why: A* with inadmissible heuristics loses optimality guarantees; A* for satisficing wastes effort proving optimality. How: Match algorithm to problem: A* + admissible heuristic for optimal; non-optimal search (e.g., GBFS, wA*) for satisficing; fast methods for agile.
Validate Regressed States Carefully
â–¶
When using regression, validate that generated states satisfy domain constraints and are reachable. Why: Regression creates spurious states that may be invalid or unreachable. How:
Check states satisfy domain constraints
Use mutex reasoning to detect invalid states
Verify reachability when possible
Always validate final plans forward from initial state
The Data
High-quality, uncontaminated data with proper validation is essential for reliable evaluation.
Best Practices
Match Benchmark Difficulty to Your Problem Type
â–¶
Select IPC instances appropriate for your planning problem type (optimal/satisficing/agile). Why: Instances challenging for optimal planning may be trivial for satisficing. How:
Use the autoscale tool to generate appropriate difficulty levels
Test on diverse domains
Report coverage, quality, and scaling behavior
Understand what each domain tests
Generate Fresh Instances to Avoid Contamination
â–¶
Use instance generators to create fresh, unseen test data. Why: Public benchmarks and internet-scraped data may be in training sets, inflating performance. How:
Use PDDL generators (50+ domains available)
Create novel domains if need to guarantee unseen domains
Document data provenance
Avoid scraping from public websites
Report contamination risks honestly
Provide Independent, Deterministic Validators
â–¶
Create sound, independent validators for your benchmarks—avoid LLM-as-judge for planning. Why: Planning requires precise logical reasoning; LLMs may miss constraint violations. How:
Separate validation logic from solver
Check action applicability, state transitions, and goal achievement
Make validation auditable
Manually verify samples from datasets
Use Structure-Aware Train-Test Splits
â–¶
Partition data by problem size or plan length, not randomly; verify test instances are structurally different. Why: Generators may produce isomorphic instances; random splits cause data leakage. How:
Split by object count or plan length (e.g., train: 3-7 objects, test: 7-20)
Verify test problems require different reasoning, cannot be solved using the same simple procedure
Check that complexity actually increases (e.g., plan length not artificially capped)
Tools From The Planning Community
The planning community has developed sophisticated tools—using them correctly is critical.
Best Practices
Fully Specify Your Planner Configuration
â–¶
Always state the exact planner configuration and which computational problem it solves. Why: "Fast Downward" alone is insufficient—it could be optimal, satisficing, or agile. How:
Report:
Planner name and version
Search algorithm
Heuristic function
Problem type being solved (optimal/satisficing/agile)
Any non-standard configurations
Preserve Static Information for LLM-Based Planners
â–¶
When using grounding tools, preserve static information (maps, types) that LLMs may need. Why: Grounders remove information irrelevant to symbolic search but crucial for LLMs. How:
Understand what your grounding tool is doing, what is preserved, what is removed
Manually preserve static predicates if needed
Consider using lifted representations
Test whether your LLM needs this information
On Evaluating Planners
Rigorous evaluation requires appropriate metrics, diverse benchmarks, and complete experimental details.
Best Practices
Provide Complexity Analysis and Formal Guarantees
â–¶
Document your method's computational complexity, soundness, completeness, and optimality properties. Why: Understanding complexity guides baseline selection and result interpretation. How: State the problem's complexity class, prove or clearly state your method's guarantees, clarify the target guarantees if full characterization is unavailable, and use this to justify baseline choices.
Choose Metrics Matching Your Problem Type
â–¶
Use coverage for optimal planning; use IPC score for satisficing/agile planning. Why: Coverage doesn't capture plan quality differences in non-optimal settings. How:
For satisficing: report IPC score (c*/c where c* is best known cost and c is your plan cost)
For agile: use time-based IPC score
For optimal: coverage is appropriate
Always report on diverse domains
Separate Validation from Generation Code
â–¶
Implement validation as an independent module, not embedded in task code. Why: Embedded validation may miss errors like using disallowed actions. How:
Only report optimality metrics if your method guarantees optimal solutions. Why: Validating optimality in classical setting is PSPACE-complete; occasional optimal solutions don't imply guarantees. How:
If your method is unsound or uses inadmissible heuristics, don't measure optimality
Report plan quality but not how many times it happened to find an optimal solution
Compare only to methods with matching guarantees
Compare Search Effort Only with Matching Algorithms
â–¶
Compare search effort (nodes expanded/generated) only when using the same search algorithm. Why: Different algorithms have different expansion strategies; comparisons across algorithms are meaningless. How:
Use same search algorithm
Ensure algorithms provide same guarantees
Remember: A* is optimally efficient—claims of "better than A*" mean some guarantee was sacrificed
Evaluate on Diverse, Existing Benchmarks
â–¶
Test on multiple existing benchmark domains with varying difficulty; avoid introducing dataset and method together. Why: Single-domain or co-developed datasets risk evaluation bias. How:
Use established planning benchmarks if possible (e.g., IPC benchmarks)
Test on diverse set of domains
Show scaling with difficulty
If new datasets used, justify why existing ones insufficient
Have independent validation
Report Complete Experimental Details
â–¶
Specify all experimental details: hardware, time/memory limits, LLM calls, tool versions. Why: Incomplete specifications prevent reproducibility and fair comparison. How:
Report:
Hardware specs
Time and memory limits
Number and cost of LLM calls
Planner configurations
All tool versions
Random seeds if applicable
Resources
The following links are drawn from the paper and grouped by how readers are likely to use them.
planutils: practical entry point for running planners and planning tools.
Unified Planning: library for modeling, manipulating, and solving planning problems.
Fast Downward: widely used planning system. Always report the exact configuration used.
Pyperplan: useful for education and prototyping. The paper cautions that it should not be used for experimental comparison.
Forbid Iterative: diverse planner available as a python package, also useful for obtaining problem landmarks and various graph-based representations of planning problems.
K* planner: State-of-the-art top-k/top-quality planner, available as a python package.
SymK planner: symbolic top-k planner that supports many PDDL features.
VAL and INVAL: independent PDDL plan validators, useful for independently checking PDDL plans.
Benchmarks, generators, and parsing
autoscale: useful for generating benchmark instances with controlled difficulty.
PDDL generators: useful for fresh instance generation and novelty controls.