SparkExecution

Inspect result.root_run.spark_executions for actions observed during collection, their source locations, and available executed query plans.

dataclass linescope.model.SparkExecution[source]

Snapshot one observed action and its available executed query plans.

Attributes

id : str Unique identifier used by action views and source references. name : str Observed Spark action or query name. location : SourceLocation | None Exact project line that triggered the action, when available. stats : SparkExecutionStats Available action duration and cumulative executor measurements. operators : list[SparkOperator] Observed physical operator trees, preferring the final AQE plan. executed_plan : str | None Executed physical plan text, or None when unavailable. initial_plan : str | None Initial physical plan text before AQE, when exposed by Spark. optimized_plan : str | None Optimized logical plan text, or None when unavailable. parsed_plan : str | None Parsed logical plan text, or None when unavailable. analyzed_plan : str | None Analyzed logical plan text, or None when unavailable. stages : list[dict[str, Any]] Available stage summaries and task metric distributions. jobs : list[int] Spark job identifiers associated with this action. status : str | RunStatus Action outcome normalized to RunStatus at construction. warnings : list[str] Diagnostics explaining unavailable or partial action metadata. metadata : dict[str, Any] Collection context and plan availability diagnostics.

See Also

ProfileRun

Represent a profiling scope and any separately executed child notebooks.

SparkExecutionStats

Keep wall duration separate from cumulative executor work.

SparkOperator

Represent a physical operator and the metrics Spark makes available.