Introduction In the previous posts of the PaasUp DIP modern data pipeline series (#1~3), we implemented data pipelines purely using open-source libraries (pyspark, delta-spark, boto3, etc.) without any framework. This was a crucial learning process for understanding basic principles and direct implementation. However, in actual production environments, framework adoption

Analyzing Urban Transportation Patterns with Seoul Subway Hourly Boarding and Alighting Data Project Overview The Seoul subway network is South Korea's largest urban railway system, serving over 7 million citizens daily. This project implements the 'Rush Hour Pattern Analysis through Hourly Data Acquisition' task that was

Introduction In the previous blog post (Modern Data Pipeline with PaasUp DIP: Subway Passenger Analytics Project #1), we implemented a data pipeline by creating a Spark Session in a Jupyter Notebook environment. In the second part of this series, we'll explore implementing and executing data pipelines in an

Seoul Subway Usage Analysis Project Introduction In the first part of this series , the subway of modern cities is one of the most important public transportation systems. By analyzing the usage patterns of subways used by countless citizens daily, we can understand the flow of the city and citizens'