Skip to content

perf: Prune matched right rows in nested loop semi and anti joins - #25413

Open
lyne7-sc wants to merge 1 commit into
apache:mainfrom
lyne7-sc:perf/nlj-right-existence-pruning
Open

lyne7-sc wants to merge 1 commit into
apache:mainfrom
lyne7-sc:perf/nlj-right-existence-pruning

Conversation

@lyne7-sc

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • Closes #.

Rationale for this change

For RightSemi and RightAnti nested loop joins, each right row only needs one match to determine whether it belongs in the result. Currently, matched right rows continue participating in comparisons with subsequent left rows.

Skipping these comparisons can significantly reduce execution time when matches are found early.

What changes are included in this PR?

  • Add RightExistenceProbe to track matches and right-side candidates.
  • Compact the right-side filter columns when at least half the candidates have matched and more left rows remain.
  • Stop probing the current right batch when all its rows have matched.
  • Preserve matches in original right-row coordinates for output and spill handling.

What is the testing strategy for this PR?

  • Add unit tests for match tracking and candidate compaction.
  • Add SQL coverage in nested_loop_join_existence.slt for RightSemi/RightAnti joins, including NULLs, repeated matches, and multiple left rows per probe.
  • Extend nested_loop_join_spill.slt to cover RightSemi/RightAnti joins across spilled left chunks.

Are there any user-facing changes?

No. This is an internal performance optimization.

Benchmark results

Case Scenario main This PR Time change
Q15 Right Semi, early matches 307.659 0.365 -99.88%
Q16 Right Anti, early matches 306.225 0.338 -99.89%
Q18 Right Semi, no matches 7.964 7.728 -2.95%
Q19 Right Anti, no matches 8.100 7.770 -4.07%
Q20 Right Semi, staged compaction 56.082 7.182 -87.19%
Q21 Right Anti, staged compaction 56.057 7.115 -87.31%
Q22 Right Semi, repeated 40% matches (4,096 left × 8,192 right) 12.185 12.410 +1.84%
Q23 Right Anti, final-left-row matches with wide Utf8 0.305125 0.309042 +1.28%

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 17, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.74359% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.33%. Comparing base (606ae0f) to head (b9fb069).
⚠️ Report is 31 commits behind head on main.

Files with missing lines Patch % Lines
...fusion/physical-plan/src/joins/nested_loop_join.rs 89.74% 8 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25413      +/-   ##
==========================================
+ Coverage   81.94%   82.33%   +0.38%     
==========================================
  Files        1135     1137       +2     
  Lines      428278   432064    +3786     
  Branches   428278   432064    +3786     
==========================================
+ Hits       350952   355737    +4785     
+ Misses      56369    54822    -1547     
- Partials    20957    21505     +548     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants