Data Engineering for Beginners: Part 1.1 – Setting Up Your Data Engineering Environment

Welcome to QuakeFlow

You have now completed the Data Fundamentals section of the series. Great job everyone!

You will now hopefully understand the following:

  • How the data world works
  • The different data careers
  • What Data Engineers actually do
  • How data is stored
  • How data moves through pipelines

Until now, we have mostly been learning concepts.

That’s about to change.

Welcome to QuakeFlow.

QuakeFlow is a hands-on Data Engineering project that we’ll build step by step throughout this series.

Instead of learning Python, SQL, Git, data modeling, APIs, and other technologies as isolated tutorials, we’re going to use them to build one complete data platform.

By the end of the project, you’ll have built a pipeline that takes earthquake data from a real-world source and turns it into something that can be analyzed in Power BI.

But we’re not starting with APIs. We’re not starting with databases. We’re not starting with cloud platforms.

We’re starting with something much simpler:

Setting up a professional development environment.

What Is QuakeFlow?

QuakeFlow is a fictional data platform designed around earthquake data. Why Quakeflow? Well, I didn’t really want to start yet another project about some kind of online store tracking sales. There are plenty of other projects if you are interested in that. I am an educated geographer, so yes, we are going to work with earthquake data!

The project will eventually collect information about earthquakes, process it, store it, model it, and make it available for analytics.

You don’t need to understand every component yet. That’s what the rest of the series is for.

What We Are Building

By the end of QuakeFlow, we’ll have a system capable of answering questions such as:

  • How many earthquakes occurred?
  • Where do earthquakes occur most frequently?
  • How does earthquake activity change over time?
  • What was the largest earthquake in a given period?
  • Which regions experience the most activity?
  • How does earthquake magnitude vary geographically?

The final project will contain several different layers.

Data Sources

I really want to keep this project from overwhelming people. That’s why we are taking this in steps. Sure, there is a nice API to fetch data from, but I really want to focus on building a simple pipeline first, before introducing APIs. We will get there though. Anway, we start with a simple CSV file, but later we will use the USGS Earthquake API.

Ingestion

Python will collect the data.

Storage

We’ll store the raw data before transforming it. We will start with a simple PostgreSQL database.

Transformation

We’ll use SQL to prepare the data.

Modeling

We’ll create an analytical data model.

Data Quality

We’ll add checks to make sure the data is trustworthy.

Analytics

We’ll eventually connect the final model to Power BI.

But before any of that, we need somewhere to build it.

What You’ll Install

For the first part of QuakeFlow, we only need a few tools.

Required

  • Python, to build logic.
  • Visual Studio Code, to write code with.
  • Git, to save our work. Or fetch mine hehe 🙂

Later

We’ll introduce other technologies when we actually need them.

This is intentional.

You don’t need to install:

  • Docker
  • PostgreSQL
  • SQL Server
  • Databricks
  • Airflow
  • Azure
  • AWS

today.

Installing ten technologies before understanding why you need them is one of the easiest ways to make learning Data Engineering unnecessarily complicated.

We’ll introduce each tool when QuakeFlow needs it.

Step 1 – Install Python

Python will be one of the main programming languages we use throughout QuakeFlow.

We’ll use it for things such as:

  • Reading files
  • Calling APIs
  • Moving data
  • Automating tasks
  • Validating data
  • Building ingestion pipelines

If Python is already installed on your computer, you can check the version from a terminal.

python --version

On some systems you may need:

python3 --version

You should see something similar to:

Python 3.x.x

For this series, use a currently supported Python 3 version.

If you don’t have Python installed, download it from the official Python website, which you can find here:

https://www.python.org/downloads

When installing Python on Windows, make sure the option to add Python to your PATH is enabled.

Afterwards, you can run the python3 –version command again.

Step 2 – Install Visual Studio Code

We’ll use Visual Studio Code, usually called VS Code, as our code editor.

You could use another editor.

That’s completely fine.

The important thing is that you have an environment where you can:

  • Write Python
  • Edit SQL
  • Manage files
  • Use Git
  • Work with a terminal

VS Code is particularly useful for beginners because it combines these capabilities in one application.

After downloading VS Code from https://code.visualstudio.com/, install and open it.

Step 3 – Install the Python Extension

VS Code supports Python through an extension.

Open the Extensions panel and search for:

Python

Install the official Python extension from Microsoft.

This gives VS Code features such as:

  • Syntax highlighting
  • Code completion
  • Running Python
  • Debugging
  • Virtual environment integration

We’ll use these features later.

Step 4 – Install Git

Git is a version control system, which you learned about version control earlier in the Fundamentals section.

Now we’re going to actually use it.

Git allows you to track changes to your project, makes collaboration easier, and acts like a backup for your code. For all these reasons, Git is an essential professional skill for Data Engineers.

You will encounter Git constantly in real Data Engineering teams.

Check Git

Open a terminal and run:

git --version

You should see something similar to:

git version 2.x.x

If you don’t have Git installed, install it before continuing.

Step 5 – Create the QuakeFlow Project

Now we’re going to create the actual project.

Create a folder somewhere convenient. You can do this through your OS’ GUI, or you can use a terminal and run:

mkdir QuakeFlow

Open that folder in VS Code.

Your project currently contains nothing. That’s totally normal. We’re starting from scratch.

Step 6 – Create the Project Structure

Before writing code, let’s create a basic structure.

Our first version will look like this:

QuakeFlow/
data/
src/
tests/ 
README.md
.gitignore

It might look unnecessarily simple right now.

That’s intentional.

We’ll add more structure as the project grows.

What Does Each Folder Do?

data/

This is where we’ll keep datasets used by the project.

Later we’ll have different types of data here.

For example:

data/
raw/
processed/

We aren’t creating those folders yet unless we need them.

src/

This is where our Python code will live.

Eventually we’ll have things like:

src/
ingestion/
transformation/ 
validation/

Again, we’ll build this gradually.

tests/

This is where we’ll eventually put automated tests.

We’ll introduce testing later in the project.

For now, it’s simply part of our professional project structure.

README.md

The README explains the project.

It should eventually contain:

  • What QuakeFlow is
  • How to run it
  • Project architecture
  • Technologies used
  • How the data flows through the system

A good README is part of a professional project.

.gitignore

The .gitignore file tells Git which files it should not track.

For example, Python creates temporary files and virtual environments that generally shouldn’t be committed to Git.

We’ll configure this shortly.

Step 7 – Create a Virtual Environment

This is an important Python concept.

A virtual environment creates an isolated Python environment for your project.

Why do we need one?

Imagine you have two projects.

Project A requires:

pandas 2.x

Project B requires a different version.

Installing everything globally can eventually create conflicts.

A virtual environment keeps each project independent.

Creating the Virtual Environment

From the QuakeFlow project directory, run:

python -m venv .venv

This creates:

QuakeFlow/
.venv/
data/
src/
tests/
README.md
.gitignore

The .venv directory contains the isolated Python environment.

Activating the Environment

On Windows PowerShell:

.venv\Scripts\Activate.ps1

On Windows Command Prompt:

.venv\Scripts\activate

On macOS or Linux:

source .venv/bin/activate

Once activated, your terminal should indicate that the virtual environment is active.

You might see something like:

(.venv)

before your command prompt.

Why Is .venv Not Going Into Git?

This is where .gitignore becomes important.

We don’t want to commit the entire virtual environment to Git.

Add:

.venv/

to .gitignore.

Our project now looks like:

QuakeFlow/
.venv/
data/
src/
tests/
README.md
.gitignore

But Git will ignore .venv.

Step 8 – Create requirements.txt

Eventually, QuakeFlow will use Python packages.

For example:

  • pandas
  • requests

Instead of expecting another developer to guess which packages are required, we can record them in a dependency file.

Create:

requirements.txt

We don’t need to add anything yet.

We’ll add packages when we actually use them.

This is another example of something we’re deliberately not over-engineering at the beginning.

Step 9 – Write Your First Python Program

We’re finally going to write some code.

Create:

src/main.py

Add:

print("Welcome to QuakeFlow!")

Save the file.

Now run:

python src/main.py

You should see:

Welcome to QuakeFlow!

Congratulations.

You’ve just run the first piece of the QuakeFlow project.

It isn’t a data pipeline yet, but that’s ok! The purpose of this exercise is to confirm that your development environment works.

Step 10 – Initialize Git

Now we’ll turn the project into a Git repository.

From the root of the QuakeFlow folder:

git init

Git will now track changes to the project.

You can check the status:

git status

You should see files that Git has detected.

The .venv directory should not appear as something to commit if your .gitignore is configured correctly.

Step 11 – Make Your First Commit

Before committing anything, add the files:

git add .

Then check the status again:

git status

Now create your first commit:

git commit -m"Initial QuakeFlow project setup"

You have now created the first version of the project.

What Is a Commit?

A commit is essentially a saved point in the history of your project.

Think of it like:

Version 1
↓
Version 2
↓
Version 3

Each commit records changes. And it is possible to go back to these other version when the need arrives. This could be for example when you pushed code into production that actually had a major bug in it. Go back a commit and push to production, and you are fine.

Any future developers will able be able to look back and understand how your project evolved.

Step 12 – Optional: Create a GitHub Repository

Git and GitHub are related, but they are not the same thing.

Git is the version control system.

GitHub is a platform where Git repositories can be hosted and shared.

For your portfolio, GitHub is useful because employers can see your project.

Create a new repository called:

QuakeFlow

Then connect your local project to GitHub.

The exact commands depend on the repository URL GitHub gives you.

For example:

git remote add origin <your-repository-url>

Then:

git branch -M main
git push -u origin main

Your QuakeFlow project is now available in your GitHub repository.

Your Project So Far

At this point, you should have something similar to:

QuakeFlow/
.venv/
data/
src/│   
└── main.py
tests/
.gitignore
README.md
equirements.txt

And your Git history should contain your first commit.

That’s a solid starting point.

What We Haven’t Done Yet

Notice what we haven’t installed.

We haven’t installed:

  • PostgreSQL
  • SQL Server
  • Docker
  • Airflow
  • Spark
  • Databricks
  • Azure
  • AWS

And that’s completely intentional. At this stage, we don’t need them.

Data Engineering involves a huge ecosystem of technologies. A common mistake is trying to learn all of them simultaneously. Instead, QuakeFlow will introduce technologies when they solve an actual problem.

What Happens Next?

Our environment is ready.

But we still don’t have any data.

That’s the next step.

And we’re not going to start with an API.

Our first data source will be a simple CSV file.

This might seem almost too easy.

That’s exactly why we’re doing it.

Before we introduce HTTP requests, APIs, JSON, authentication, databases, and other concepts, we want you to understand the fundamental process of moving data through a pipeline.

The next article will introduce the first QuakeFlow dataset.

You’ll download a small CSV file, inspect it, load it with Python, and start working with real data.

The pipeline will begin simply:

CSV File
↓
Python
↓
Inspect Data

Then we’ll build on it.

Key Takeaways

You’ve now:

  • Installed Python
  • Installed VS Code
  • Installed Git
  • Created a Python virtual environment
  • Created the QuakeFlow project
  • Created a basic project structure
  • Written your first Python script
  • Initialized Git
  • Created your first commit

More importantly, you’ve created the foundation for the rest of the project.

From here, every article will add another piece.

What Comes Next

QuakeFlow 1.2 — Your First Data Source: Working With a CSV

We’ll introduce the first QuakeFlow dataset.

You’ll learn how to:

  • Understand the dataset
  • Inspect a CSV file
  • Read CSV data with Python
  • Explore rows and columns
  • Understand data types
  • Identify potential data quality problems
  • Start thinking like a Data Engineer

For the first time, we’ll be working with actual data.

Newsletter Updates

Enter your email address below and subscribe to our newsletter

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *


Trending now

Data Engineering for Beginners: Part 1.1 – Setting Up Your Data Engineering Environment