dongwooshin

Urban traffic pattern analysis project processing over 10 years of Seoul subway ridership data using modern data lake technology stack. Transformed 73,692 records with Delta Lake and Apache Superset to visualize rush hour commute patterns and transfer station hub effects.

Subway data pipeline automation project transitioning from Jupyter Notebook development environment to Airflow production environment. Implemented enhanced security using Kubernetes Secret and workflow construction based on scheduling through SparkKubernetesOperator.

This blog introduces a project using the PAASUP DIP to implement a modern data pipeline that processes and analyzes over 10 years of Seoul subway ridership data through Apache Spark, Delta Lake, and PostgreSQL, combined with Apache Superset dashboards for multi-dimensional data visualization.