Skip to content

Fix sub_table ordering for nested inlined comprehensions (PEP 709)#7480

Merged
youknowone merged 1 commit intoRustPython:mainfrom
LegNeato:fix/nested-comprehension-underscore
Mar 22, 2026
Merged

Fix sub_table ordering for nested inlined comprehensions (PEP 709)#7480
youknowone merged 1 commit intoRustPython:mainfrom
LegNeato:fix/nested-comprehension-underscore

Conversation

@LegNeato
Copy link
Contributor

@LegNeato LegNeato commented Mar 22, 2026

When an inlined comprehension's first iterator expression contains nested scopes (such as a lambda), those scopes' sub_tables appear at the current position in the parent's sub_table list. The previous code spliced the comprehension's own child sub_tables (e.g. inner inlined comprehensions) into that same position before compiling the iterator, which shifted the iterator's sub_tables to wrong indices.

Move the splice after the first iterator is compiled so its sub_tables are consumed at their original positions.

Fixes nested list comprehensions like:

    [[x for _, x in g] for _, g in itertools.groupby(..., lambda x: ...)]

Disclosure: I used AI to develop the patch though I was heavily involved.

Summary by CodeRabbit

Bug Fixes

  • Fixed nested comprehension scoping and iteration semantics for complex cases involving lambda expressions and underscore variables.

When an inlined comprehension's first iterator expression contains
nested scopes (such as a lambda), those scopes' sub_tables appear at the
current position in the parent's sub_table list. The previous code
spliced the comprehension's own child sub_tables (e.g. inner inlined
comprehensions) into that same position before compiling the iterator,
which shifted the iterator's sub_tables to wrong indices.

Move the splice after the first iterator is compiled so its sub_tables
are consumed at their original positions.

Fixes nested list comprehensions like:
```python
    [[x for _, x in g] for _, g in itertools.groupby(..., lambda x: ...)]
```

Disclosure: I used AI to develop the patch though I was heavily
involved.
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 22, 2026

📝 Walkthrough

Walkthrough

Reorders compilation sequence in compile_inlined_comprehension to compile the outermost iterator expression before splicing comprehension sub-tables into the parent symbol table. Adds three runtime tests for nested comprehensions with lambda iterators, underscore variables, and basic nested cases.

Changes

Cohort / File(s) Summary
Comprehension Compilation Reordering
crates/codegen/src/compile.rs
Moved compilation of the outermost iterator (generators[0].iter) to execute before splicing comp_table.sub_tables into the parent symbol table, ensuring nested scopes created during iterator compilation are consumed before symbol-table splicing.
Nested Comprehension Tests
extra_tests/snippets/syntax_comprehension.py
Added three test functions: test_nested_comp_with_lambda() for lambda-based iterators, test_nested_comp_underscore() for throwaway variables, and test_simple_nested_comp() for basic nested comprehension cases. All tests include immediate invocation and assertions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • RustPython/RustPython#7438: Modifies compile.rs to change how nested-scope sub\_tables are consumed during optimized-out asserts to maintain symbol-table ordering alignment.
  • RustPython/RustPython#7300: Addresses the root comprehension sub\_tables desynchronization issue by preventing spurious sub\_table creation during annotation scanning.
  • RustPython/RustPython#7412: Modifies inlined-comprehension compilation in compile.rs, specifically the handling and order of outermost iterator compilation and sub\_tables splicing.

Suggested reviewers

  • ShaharNaveh

Poem

🐰 A rabbit's ode to order restored:

First iterate, then splice with care,
Symbol tables dance through the air,
Lambda lambdas and underscores too,
Nested comprehensions work anew! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main fix: reordering sub_table operations for nested inlined comprehensions, directly matching the primary change in the PR.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

You can disable the changed files summary in the walkthrough.

Disable the reviews.changed_files_summary setting to disable the changed files summary in the walkthrough.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
extra_tests/snippets/syntax_comprehension.py (1)

50-81: Ensure this snippet suite is executed in debug mode in CI/test docs.

These additions are in extra_tests/snippets/**; keeping execution on cargo run (debug) avoids unnecessary compile-time overhead for this suite.

As per coding guidelines, "extra_tests/snippets/**: Use debug mode (cargo run) for extra_tests/snippets tests as compilation is faster".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@extra_tests/snippets/syntax_comprehension.py` around lines 50 - 81, Update
CI/test documentation and the test-runner config so the extra_tests/snippets
suite runs in debug mode (via cargo run) instead of a release/compiled
invocation; specifically ensure any CI job or test script that executes tests
under the directory extra_tests/snippets uses cargo run (debug) to invoke these
snippet files (including test_nested_comp_with_lambda,
test_nested_comp_underscore, test_simple_nested_comp) to avoid unnecessary
compile-time overhead. Adjust the relevant CI step or README/test docs to call
cargo run for that path and confirm the change is applied where
extra_tests/snippets is referenced.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@extra_tests/snippets/syntax_comprehension.py`:
- Around line 50-81: Update CI/test documentation and the test-runner config so
the extra_tests/snippets suite runs in debug mode (via cargo run) instead of a
release/compiled invocation; specifically ensure any CI job or test script that
executes tests under the directory extra_tests/snippets uses cargo run (debug)
to invoke these snippet files (including test_nested_comp_with_lambda,
test_nested_comp_underscore, test_simple_nested_comp) to avoid unnecessary
compile-time overhead. Adjust the relevant CI step or README/test docs to call
cargo run for that path and confirm the change is applied where
extra_tests/snippets is referenced.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 51cda9eb-baf9-4d88-bc93-6fa42c52ff77

📥 Commits

Reviewing files that changed from the base of the PR and between dfcb07c and ecd1052.

📒 Files selected for processing (2)
  • crates/codegen/src/compile.rs
  • extra_tests/snippets/syntax_comprehension.py

Copy link
Member

@youknowone youknowone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for contributing! You don't need to worry about AI develop as long as you are in the loop.

@youknowone youknowone merged commit 2180f53 into RustPython:main Mar 22, 2026
18 checks passed
@LegNeato LegNeato deleted the fix/nested-comprehension-underscore branch March 22, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants