PySpark Time Series Pipelines
Today's post covers the following:
- Basic pipeline conversion of timestamp to unix time
- Lag feature combination pipelines
- Aggregation based statistics pipelines
July, 2025¶
PySpark Pivoting
Today's post covers the following:
- Basic pivot operation
- Pivot with multiple aggregations
- Conditional pivoting
- Pivoting with specified column values
PySpark Aggregations
Todays post covers the following:
- Aggregations without grouping
- Aggregations with grouping
- Filtering after grouping
PySpark Data Filtration
Todays post covers the following:
- Filtration by column value (one or multiple conditions)
- String related filtration using like / contains
- Missing data filtration
- List based filtration using isin
- General data clearning operations
PySpark Pipelines
Todays post covers the following:
- Missing data treatment classification pipeline
- Feature scaling using ScandardScaler classification pipeline
- TF-IDF corpus classification pipeline
- PCA dimensionality reduction classification pipeline
PySpark Select, Drop, Rename Columns
Todays post covers the following:
- Reading CSV & Parquet formats
- Column selection
- Renaming columns
- Adding columns
- Dropping columns