Data Engineering for Beginners (0.4): Understanding Data Storage

You’re on Part 0.4 of 5 in Foundations → the hands-on build (QuakeFlow) starts at Part 1.1

Welcome to the Series

In Part 0.1 you learned how data moves. In Part 0.2, who moves it. In Part 0.3, whether this is the work for you. Before any of that movement can happen, though, data has to actually live somewhere — and understanding where and how is what this article is about.

Where does data actually live, and how is it organized? In short: in files, databases, warehouses, or lakes — each suited to a different job — and inside those, information is broken into tables, rows, columns, and relationships that let a computer find and combine it reliably. Skip this and every tool you touch later — SQL, Spark, cloud platforms — feels like a pile of unrelated technologies instead of variations on the same handful of ideas.

Learning Path

Why Data Needs Structure

In.

Imagine a global seismic network logging one million ground-motion readings today. For each one, something needs to remember: which station recorded it, exactly when, at what magnitude and depth, and where. That’s already millions of individual facts from a single day — and multiplying that by years of continuous monitoring makes clear why “a folder of spreadsheets” stops working fast. Databases and structured storage exist specifically to make that volume searchable, updatable, and trustworthy.

The Vocabulary: Entity, Attribute, Record

An entity is something you want to store information about — an Earthquake, a Monitoring Station, a Region.

An attribute describes that entity — for an Earthquake: magnitude, depth, location, and time.

A record is one specific instance — one actual earthquake event, with its own values for every attribute:

Earthquake ID: 70001 · Magnitude: 5.6 · Depth (km): 12.3 · Place: Banda Sea

Once we get to tables, records become rows, and attributes become columns — same idea, different name.

Three Types of Data

Structured data fits neatly into rows and columns — an earthquake catalog table is a textbook example:

Semi-structured data has some organization but doesn’t fit a rigid grid — a USGS API response, for instance:

json

{
  "id": "us7000abcd",
  "mag": 5.6,
  "place": "Banda Sea"
}

{
  "id": "us7000abcd",
  "mag": 5.6,
  "place": "Banda Sea"
}

QuakeFlow will pull exactly this kind of JSON directly from a real earthquake API — this isn’t a hypothetical example, it’s the actual shape of your first real data source.

Unstructured data has no predefined format at all — think raw seismograph waveform recordings, satellite imagery of a fault line, or a written incident report. Most real organizations, including QuakeFlow eventually, end up working with a mix of all three.

Where Data Actually Gets Stored

Files — CSV, JSON, Parquet, and similar formats. Simple, portable, and ideal for raw ingestion, but with limited validation and no built-in relationships. CSV is usually the first format any beginner encounters, and it’s exactly where QuakeFlow starts too.

Databases — PostgreSQL, MySQL, SQL Server, SQLite, and similar systems built for fast queries, reliable updates, and enforced relationships between data. This is where structured earthquake records will actually live once QuakeFlow moves past raw files.

Data warehouses — databases specifically optimized for analytics rather than day-to-day transactions. Instead of “log this new reading,” a warehouse answers questions like “how has average magnitude in this region changed year over year?”

Data lakes — storage for large volumes of raw data in whatever format it arrives, with the philosophy of “store first, decide how to use it later.” A lake might hold raw JSON straight from the USGS feed, station sensor logs, and processed Parquet files side by side.

How Databases Organize Data Internally

Inside a relational database, information sits in a hierarchy:

Database → Schema → Table → Column → Row → Value

Database → Schema → Table → Column → Row → Value

Concretely:

EarthquakeDB
  └── staging
      └── earthquakes_raw
          └── magnitude
              └── 5.3

EarthquakeDB
  └── staging
      └── earthquakes_raw
          └── magnitude
              └── 5.3

Once this hierarchy clicks, SQL stops feeling arbitrary — every query is just navigating this structure.

Tables: One Table, One Thing

A well-designed table stores information about exactly one entity:

Keeping each table focused on a single entity is what keeps a growing system understandable instead of tangled.

Rows, Columns, and Data Types

Columns describe what is stored; rows are individual records:

Every column also has a data type, telling the database what kind of value to expect:

Choosing the right type isn’t pedantry — it directly affects performance and data quality, as you already saw in Part 1.2 when a magnitude value arrived as text and needed explicit conversion.

Primary Keys and Foreign Keys

A primary key uniquely identifies each row. EarthquakeID works because it’s guaranteed unique — unlike, say, Place, since dozens of earthquakes can share the same location name.

A foreign key connects tables together. If an Earthquakes table includes a StationID column, that column points back to a Stations table, letting the database know exactly which station recorded which event:

Stations

Earthquakes

Relationships Between Tables

One-to-many is the most common shape: one region can contain many earthquakes.

Many-to-many shows up too: one earthquake can be recorded by multiple stations, and one station records many earthquakes over time. This is usually resolved with a bridge table:

What Is a Schema?

A schema organizes related database objects — think of it as a folder inside a database:

seismic

• earthquakes
• stations
• regions

research

• models
• predictions

A full table reference typically looks like seismic.earthquakes — the table name prefixed with its schema.

Why Not Just Use One Giant Table?

Imagine cramming everything together:

The station and region details repeat on every single row. That’s wasted storage, and worse, if a station’s metadata changes, you now have to update it in a thousand places instead of one. Splitting related data into separate, connected tables is what avoids this.

Operational vs. Analytical Storage, Revisited

Part 0.1 introduced OLTP and OLAP through the lens of real-time seismic alerting versus historical research queries. The storage side of that split looks like this:

Operational (OLTP) tables are typically normalized — each entity gets its own table, kept lean for fast, reliable writes, with related information joined in only when needed.

Analytical (OLAP) tables are often denormalized on purpose — some repetition is accepted in exchange for fewer joins and faster large-scale queries, since read performance matters more than write efficiency here.

Warehouses, Lakes, and Lakehouses

A data warehouse takes data copied from operational systems and organizes it specifically for reporting — consistent definitions, historical depth, and query performance built in.

A data lake keeps raw data in its original form — raw USGS JSON, station logs, Parquet exports — useful specifically because you haven’t committed to a structure yet.

A lakehouse combines both: the flexibility of a lake with the reliability and transaction support of a warehouse. Technologies like Delta Lake, Apache Iceberg, and Apache Hudi exist specifically to make this combination work.

How It All Connects

How This Connects to QuakeFlow

QuakeFlow’s actual pipeline will move through nearly every concept in this article:

USGS Earthquake API
  → Python ingestion
    → Raw JSON storage
      → Database tables
        → SQL transformations
          → Analytical model
            → Power BI dashboard

USGS Earthquake API
  → Python ingestion
    → Raw JSON storage
      → Database tables
        → SQL transformations
          → Analytical model
            → Power BI dashboard

That’s not a coincidence — this Foundations phase exists specifically so that by the time you’re building each of those steps, you already understand why it’s shaped the way it is.

Key Takeaways

• Data lives in files, databases, warehouses, or lakes — each built for a different job
• Tables organize entities into rows and columns, with data types enforcing consistency
• Primary keys uniquely identify records; foreign keys connect tables together
• Schemas group related tables for organization and security
• Operational (OLTP) systems favor normalized, fast-write tables; analytical (OLAP) systems favor denormalized, fast-read tables
• Warehouses, lakes, and lakehouses each solve a different part of the storage problem
• Data Engineers are the ones who build and connect all of these pieces

What Comes Next

You now understand where data lives. The next question is how it actually gets from one place to another.

In Part 0.5, we cover what a data pipeline actually is — ETL vs. ELT, batch vs. streaming, ingestion, transformation, orchestration, monitoring, and data quality. Understand the storage, then understand the movement — that’s the last piece of Foundations before QuakeFlow itself begins.

Next up → Part 0.5: Understanding Data Pipelines — how raw data actually travels from source to dashboard.