SparkExecutionStats

An action's stats. Driver duration and cumulative task work stay separate.

dataclass linescope.model.SparkExecutionStats[source]

Keep wall duration separate from cumulative executor work.

Attributes

wall_time_ns : int | None

Driver-observed action duration in nanoseconds, if available.

executor_time_ns : int | None

Cumulative executor task time, separate from driver duration.

rows : int | None

Reported row count, or None when unavailable.

bytes_read : int | None

Reported input bytes, or None when unavailable.

shuffle_read_bytes : int | None

Reported shuffle bytes read, or None when unavailable.

shuffle_write_bytes : int | None

Reported shuffle bytes written, or None when unavailable.

peak_memory_bytes : int | None

Reported peak executor memory, or None when unavailable.

spill_bytes : int | None

Reported spilled bytes, or None when unavailable.