Data Fundamentals 0.5 – Understanding Data Pipelines

Welcome Back

In the previous article, we explored how data is stored and organized.

You learned about:

• Files
• Databases
• Tables
• Rows and columns
• Primary keys
• Foreign keys
• Schemas
• Data warehouses
• Data lakes
• Lakehouses

You now understand where data lives.

But a modern data platform needs more than storage.

Data is constantly being created, changed, and moved.

A customer places an order.

A sensor sends a measurement.

An application creates a log.

A user clicks a button.

A company receives information from an external API.

All of this data needs to travel from the systems where it is created into systems where it can be analyzed.

That is where data pipelines come in.

What Is a Data Pipeline?

A data pipeline is a process that moves data from one location to another and prepares it for use.

A simple example:

Source System

↓

Extract Data

↓

Transform Data

↓

Store Data

↓

Analytics

A pipeline is the connection between the different parts of a data ecosystem.

Without pipelines, data would remain isolated in individual systems.

A sales application might know what customers bought.

A marketing platform might know what emails were opened.

A website might know what pages visitors viewed.

A data pipeline brings these pieces together so the company can understand the complete picture.

A Real-World Example

Imagine an online store.

A customer buys a product.

Several things happen:

Step 1 — The transaction is created

The order is stored in the company’s operational database.

Example:

OrderID: 50001

CustomerID: 1001

ProductID: 501

Amount: 499

At this point, the data is useful for running the website.

But it is not yet ready for analytics.

Step 2 — Data is extracted

A pipeline collects the new order data.

It might extract data from:

• Databases
• APIs
• Files
• Applications

Step 3 — Data is transformed

The raw data might need cleaning.

Examples:

• Fix incorrect formats
• Remove duplicates
• Calculate new fields
• Combine multiple sources

Step 4 — Data is loaded

The cleaned data is stored in an analytical system.

For example:

• Data warehouse
• Lakehouse

Step 5 — Data is used

Now analysts can answer questions:

• How much did we sell this month?
• Which products perform best?
• Which customers buy most often?

The pipeline made this possible.

The Data Pipeline Lifecycle

A typical modern data pipeline looks like this:

Data Sources

↓

Ingestion

↓

Raw Storage

↓

Transformation

↓

Data Modeling

↓

Data Quality

↓

Serving Layer

↓

Consumption

Let’s explore each step.

1. Data Sources

Every pipeline starts somewhere.

These are called data sources.

Examples:

Databases

A company database might contain:

• Customers
• Orders
• Payments
• Products

Example:

SQL Server
PostgreSQL
MySQL
Oracle

APIs

Many systems provide data through APIs.

Examples:

• Weather APIs
• Financial APIs
• Social media APIs
• Government data APIs

The QuakeFlow project will use this approach.

The earthquake data source will be an API providing information about earthquake events.

Files

Companies still exchange enormous amounts of data through files.

Examples:

• CSV exports
• Excel files
• JSON files
• Parquet files

Applications

Modern applications generate huge amounts of data.

Examples:

• Website clicks
• Mobile app events
• System logs

2. Data Ingestion

Data ingestion is the process of collecting data from sources and bringing it into your data platform.

Think of ingestion as the “entry point” of the pipeline.

Example:

USGS API

↓

Python Script

↓

Raw JSON Files

The goal is usually not to transform everything immediately.

A common modern approach is:

First capture the data. Then process it.

This gives you the ability to:

• Reprocess data
• Debug problems
• Keep historical records

Batch Ingestion

The most common type of ingestion is batch processing.

Batch means data is collected at intervals.

Examples:

Every night:

00:00

Extract yesterday's sales

Every hour:

10:00

Load new customer data

Every 15 minutes:

10:15

Refresh marketing data

Batch processing works well when data does not need to be available instantly.

Streaming Ingestion

Streaming means data is processed continuously as it arrives.

Examples:

• Fraud detection
• Stock trading
• IoT sensors
• Real-time monitoring

Instead of:

Collect data

↓

Process data

Streaming works like:

New Event

↓

Process Immediately

Streaming systems often use technologies like:

• Apache Kafka
• Azure Event Hubs
• Amazon Kinesis

Batch vs Streaming

For beginners, batch pipelines are usually the best place to start.

That is also where QuakeFlow begins.

3. Raw Data Storage

After ingestion, data is often stored in a raw layer.

This is sometimes called:

• Raw zone
• Landing zone
• Bronze layer

The purpose is simple:

Keep the original data.

Example:

The API returns:

{
 "id": "abc123",
 "magnitude": 4.8,
 "location": "California",
 "time": "2026-07-18"
}

The raw layer stores this exactly as received.

Why Keep Raw Data?

You might wonder:

“Why not clean the data immediately?”

Keeping raw data provides several advantages.

Reprocessing

If your transformation logic changes, you can rebuild everything from the original data.

Debugging

If something goes wrong, you can compare:

Original data

↓

Transformed data

Data History

You preserve what the source system actually provided.

4. Data Transformation

Raw data is rarely ready for reporting.

Transformation prepares data for use.

Examples:

Cleaning

Before:

Country

DK
Denmark
Danish

After:

Country

Denmark

Removing Duplicates

Before:

CustomerID

1001

1001

1002

After:

CustomerID

1001

1002

Joining Data

Combining information from multiple sources.

Example:

Customers:

CustomerID
Name

Orders:

CustomerID
Amount

Combined:

Customer
Order Amount

Creating Business Logic

Example:

A company defines:

“Active customer = customer who purchased within the last 90 days.”

The transformation creates this calculation.

ETL vs ELT

Two important approaches in data engineering are:

• ETL
• ELT

ETL: Extract Transform Load

Traditional approach:

Extract

↓

Transform

↓

Load

Data is cleaned before entering the target system.

Example:

Database

↓

Python Transformation

↓

Data Warehouse

This approach was common when storage and computing power were expensive.

ELT: Extract Load Transform

Modern cloud platforms often use:

Extract

↓

Load

↓

Transform

The raw data is loaded first.

Transformations happen inside the warehouse or lakehouse.

Example:

API

↓

Data Lake

↓

SQL Transformations

↓

Analytics Tables

Why ELT Became Popular

Modern cloud platforms provide:

• Cheap storage
• Powerful computing
• Scalable processing

Companies can keep raw data and transform it later.

Technologies like:

• Databricks
• Snowflake
• BigQuery

are designed around this approach.

5. Data Modeling

After data has been cleaned and transformed, we need to organize it in a way that makes sense for users.

This is called data modeling.

Data modeling is the process of designing how data should be structured.

A common beginner mistake is thinking:

“If the data exists in a database, we are finished.”

But storing data is not the same as making it useful.

A company might have hundreds of tables.

The challenge is creating a structure where people can easily answer business questions.

Why Do We Need Data Models?

Imagine you have these tables:

Orders

Customers

Products

An analyst wants to answer:

“How much revenue did we generate by product category last year?”

They need to combine information from multiple places.

A good data model makes this simple.

Analytical Data Models

One of the most common approaches in analytics is a star schema.

A star schema contains:

• Fact tables
• Dimension tables

Example:

              DimCustomer

                   |

                   |

DimProduct ---- FactSales ---- DimDate

                   |

                   |

              DimStore

Fact Tables

A fact table stores measurable business events.

Examples:

• Sales transactions
• Website visits
• Sensor readings
• Earthquake events

A fact table usually contains:

• Foreign keys
• Measurements
• Numbers

Example:

FactSales

Dimension Tables

Dimension tables describe the context around facts.

Examples:

DimCustomer

Who?

DimProduct

What?

DimDate

When?

DimLocation

Where?

Example:

DimCustomer

Why Does This Matter for Data Engineers?

Because the pipeline does not end when data arrives.

A Data Engineer must think about:

• How will people use this data?
• What questions need answering?
• What structure makes reporting easier?

The final design affects everyone downstream.

6. Data Serving

After transformation and modeling, data needs to be delivered to users.

This is called the serving layer.

The serving layer is where prepared data becomes useful.

Examples:

• Power BI
• Tableau
• Applications
• Machine learning models
• APIs

A typical flow:

Raw Data

↓

Transformations

↓

Data Model

↓

Power BI Dashboard

What Does a BI Developer See?

A BI Developer usually does not want to work directly with raw data.

They want clean, understandable tables.

Instead of:

Raw_Events_20260718_JSON

they want something like:

FactEarthquake

DimLocation

DimDate

This is why Data Engineering and BI work closely together.

The Data Engineer prepares the foundation.

The BI Developer builds the experience.

7. Data Quality

A pipeline that runs successfully is not necessarily a good pipeline.

The data also needs to be correct.

This is called data quality.

Imagine a pipeline runs every night.

The process completes successfully.

But today’s sales numbers are 90% lower than yesterday.

Something is wrong.

The pipeline worked technically.

The data is still incorrect.

Common Data Quality Problems

Missing Data

Example:

Expected:

Received:

Duplicate Data

Example:

One order appears twice.

Invalid Values

Example:

Age:

-5

or:

Country = "Unknown123"

Unexpected Changes

Imagine yesterday:

Orders: 150,000

Today:

Orders: 500

The pipeline may have a problem.

Data Quality Checks

Professional pipelines include automated checks.

Examples:

Row Count Check

“Did we receive approximately the expected number of records?”

Duplicate Check

“Are primary keys unique?”

Completeness Check

“Are required fields populated?”

Validity Check

“Are values within acceptable ranges?”

A simple example:

If orders_today < orders_yesterday * 0.5

Send alert

8. Orchestration: Managing Pipelines

A pipeline is rarely just one script.

A real data platform may have hundreds of processes.

Example:

Extract Customers

↓

Extract Orders

↓

Clean Data

↓

Update Warehouse

↓

Refresh Dashboard

Something needs to control the order.

This is called orchestration.

What Does an Orchestrator Do?

An orchestration tool can:

Schedule Jobs

Example:

Run every night at 02:00.

Manage Dependencies

Example:

Do not update the dashboard until the warehouse is ready.

Retry Failures

Example:

The API fails.

Try again after 10 minutes.

Monitor Execution

See:

• What succeeded
• What failed
• How long jobs took

Common Orchestration Tools

Examples:

• Apache Airflow
• Azure Data Factory
• Databricks Workflows
• AWS Step Functions

You do not need to master these as a beginner.

But understanding their purpose is important.

9. Monitoring and Logging

Professional pipelines need visibility.

A Data Engineer needs to answer:

• Did it run?
• Did it fail?
• Why did it fail?
• How long did it take?
• Did the data look correct?

Logging

A pipeline might record:

2026-07-18 02:00

Started ingestion

Received 15,432 records

Transformation completed

Loaded warehouse table

Completed successfully

If something fails:

2026-07-18 02:15

ERROR:

API connection failed

Error Handling

Failures happen.

Examples:

• API unavailable
• File missing
• Database connection lost
• Source format changed

Good pipelines handle failures gracefully.

Instead of:

Pipeline crashed

A professional system:

Pipeline failed

↓

Logged error

↓

Alert sent

↓

Retry attempted

↓

Engineer notified

Building Reliable Pipelines

A beginner pipeline might look like:

load_data()
transform_data()
save_data()

A production pipeline needs much more:

Extract

↓

Validate

↓

Transform

↓

Test

↓

Load

↓

Monitor

↓

Document

The difference between a script and a data engineering system is reliability.

The Complete QuakeFlow Architecture

Now we can combine everything.

This is the pipeline we will build in the practical phase.

              USGS Earthquake API

                       |

                       |

                       ↓

              Python Ingestion

                       |

                       |

                       ↓

                Raw JSON Data

                       |

                       |

                       ↓

              Data Storage Layer

                       |

                       |

                       ↓

             SQL Transformations

                       |

                       |

                       ↓

             Analytical Data Model

                       |

                       |

                       ↓

              Power BI Dashboard

Step-by-Step QuakeFlow Journey

Step 1: Extract

Python connects to the earthquake API.

It downloads earthquake events.

Step 2: Store Raw Data

The original JSON response is saved.

Why?

Because we want the ability to:

• Reprocess data
• Debug problems
• Keep history

Step 3: Transform

We clean and prepare the data.

Examples:

• Standardize dates
• Handle missing values
• Calculate fields
• Remove duplicates

Step 4: Model

We create analytical tables.

Example:

FactEarthquake

DimLocation

DimDate

Step 5: Analyze

Power BI connects to the final model.

Users can explore:

• Earthquake frequency
• Magnitude trends
• Geographic patterns
• Time-based analysis

Why Understanding Pipelines Matters

Many beginners start with tools:

“I want to learn Airflow.”

“I want to learn Databricks.”

“I want to learn Spark.”

But tools only make sense when you understand the problem they solve.

A Data Engineer is not valuable because they know a specific tool.

They are valuable because they understand:

• Where data comes from
• How it moves
• How it should be stored
• How it should be transformed
• How to make it reliable

Tools change.

The principles remain.

Key Takeaways

After this article, you should understand:

• A data pipeline moves data from sources to users.
• Pipelines usually contain ingestion, transformation, storage, and serving.
• Data can arrive from databases, APIs, and files.
• ETL transforms before loading.
• ELT loads before transforming.
• Data modeling makes analytics easier.
• Fact tables store events.
• Dimension tables describe those events.
• Data quality ensures trustworthy results.
• Orchestration manages complex workflows.
• Monitoring and logging make pipelines reliable.
• Data Engineers build systems that make data useful.

What Comes Next?

You have now completed the Data Fundamentals phase.

You understand:

1. How the data world works
2. The different data careers
3. Whether Data Engineering fits you
4. How data is stored
5. How data moves

The next step is no longer theory.

It is time to build.

In the next part of the series:

Phase 1 — Building QuakeFlow

You will:

• Set up your development environment
• Create your first Data Engineering project
• Use Git
• Write your first Python ingestion pipeline
• Work with real data
• Build a complete end-to-end solution

The goal is simple:

Stop learning about Data Engineering. Start doing Data Engineering.