#024: 8 mini-projects to help you practice

Dec 17, 2022

A common challenge for data engineers is finding ways to practice.

The options are seemingly endless, which isn’t always a good thing.

So you aimlessly search online or become overwhelmed and stop trying.


To help give a little guidance, every Friday for the last 2 months I’ve posted a “mini-project” idea on Twitter & YouTube Community.

Each post shares a few steps and a general goal.

Some may take you 20 minutes, others an entire weekend.

And they are purposely a bit vague to encourage self-learning.


But today I’ve consolidated all 8 mini-projects and listed them below so you can see them in one place.

My hope is that perhaps even one of these will help nudge you in a new direction.



Project #1

Goal: See how startup scripts work on VMs

  1. Write bash script to create a directory
  2. Create virtual machine on a cloud platform
  3. Set startup script -> your bash script
  4. Start the VM
  5. Open a terminal & confirm directory exists

Project #2

Goal: Understand virtual environments & aliases

  1. Install python
  2. Create a virtual environment
  3. Install any package via pip
  4. Deactivate & reactivate, notice the difference
  5. Add alias to bash/zsh profile
  6. Reactivate w/ alias

Project #3

Goal: See a GitHub workflow in action

  1. Signup for GitHub
  2. Create a new repository
  3. Confirm/Install git locally
  4. Create a random SQL file
  5. Add remote refs to GH
  6. Add, commit, push file to GH rep
  7. Create new branch, repeat & merge

Project #4

Goal: Better understand unstructured data types

  1. List your 5 favorite songs
  2. Under each, list artist, album & year
  3. Under each artist, list hometown city
  4. Convert to XML
  5. Convert to JSON
  6. Convert to YAML

Project #5

Goal: Become familiar w/ no-code automation tools

  1. Create a Google Sheet
  2. Sign up for Zapier (free plan)
  3. Connect w/ any other tool
  4. Trigger event by adding a new row
  5. Experiment w/ other tiggers

Project #6

Goal: Learn different components of SQL

  1. Write a SQL query about a fun topic
  2. Convert some logic into a function
  3. Call function in query
  4. Turn query into Stored Procedure
  5. Run Stored Procedure on a schedule

Project #7

Goal: Learn how websites use databases

  1. Create a MySQL DB
  2. Create a local webpage w/ PHP
  3. Connect to DB w/ PHP
  4. Run SQL to print a record on page

(No graphics - just plain text w/ a value from the DB.)

Project #8

Goal: Learn how to remotely interact with (and automate) cloud storage

  1. Sign up for AWS
  2. Create a S3 Bucket
  3. Create a local CSV file
  4. Connect to AWS & load the file w/ Python

As you continue your journey in this wild world of data engineering, don’t forget to practice.

It will help you stay engaged, keep your skills sharp and be a reminder of why you were interested in data in the first place.

Get clarity on common tools & components of a modern data stack

Get started with The Starter Guide for Modern Data to help you cut through the noise & better understand common "modern" architectures.

You'll also get free weekly emails with helpful tips & tutorials.