1. Preventing O(2k) Catalyst Plan Blowup in HITS
In iterative Spark DataFrame loops (especially HITS, which chains authority and hub joins every iteration),
un-truncated lineage causes Catalyst's logical plan DAG to expand exponentially and exhaust driver heap inside
AdaptiveSparkPlanExec.explainString().
- Truncating lineage via
localCheckpoint(eager=True)every 1–2 iterations keeps plan depth strictlyO(1). - Disabling AQE (
spark.sql.adaptive.enabled=false) removes per-iteration plan re-optimization overhead.