개발자가 되는 방법

다음의 영상을 제가 필요한 내용 위주로 요약한 글입니다.
What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline (2023)

• ETL : extract, transform, load 데이터의 추출, 가공, 적재

기본작동 방식 과 아키텍처

What is Data Pipeline

Data pipeline automates data supply(including data processing) to data consumers

  (point A)		(point C, D , E, ... )		(point B)
Data Producers			Data Pipeline		Data Comsumers

Data Consumers' needs

• Data Science
• Machine Learning
• Business Analytics
• Reporting

Diff between traditional ETL and Data Pipeline

Data pipeline 이 더 넓은 개념이며, ETL은 Data pipeline mechanism 세부 개념이다.

2 types of data pipeline

1. Real Time Data Pipeline
2. Batch Data Pipeline
3. Lambda Architecture (Real Time + Batch)

Lambda Architecture

Data Pipeline Architecture Example