
#024: 8 mini-projects to help you practice
Dec 17, 2022A common challenge for data engineers is finding ways to practice.
The options are seemingly endless, which isn’t always a good thing.
So you aimlessly search online or become overwhelmed and stop trying.
To help give a little guidance, every Friday for the last 2 months I’ve posted a “mini-project” idea on Twitter & YouTube Community.
Each post shares a few steps and a general goal.
Some may take you 20 minutes, others an entire weekend.
And they are purposely a bit vague to encourage self-learning.
But today I’ve consolidated all 8 mini-projects and listed them below so you can see them in one place.
My hope is that perhaps even one of these will help nudge you in a new direction.
Enjoy!
Project #1
Goal: See how startup scripts work on VMs
- Write bash script to create a directory
- Create virtual machine on a cloud platform
- Set startup script -> your bash script
- Start the VM
- Open a terminal & confirm directory exists
Project #2
Goal: Understand virtual environments & aliases
- Install python
- Create a virtual environment
- Install any package via pip
- Deactivate & reactivate, notice the difference
- Add alias to bash/zsh profile
- Reactivate w/ alias
Project #3
Goal: See a GitHub workflow in action
- Signup for GitHub
- Create a new repository
- Confirm/Install git locally
- Create a random SQL file
- Add remote refs to GH
- Add, commit, push file to GH rep
- Create new branch, repeat & merge
Project #4
Goal: Better understand unstructured data types
- List your 5 favorite songs
- Under each, list artist, album & year
- Under each artist, list hometown city
- Convert to XML
- Convert to JSON
- Convert to YAML
Project #5
Goal: Become familiar w/ no-code automation tools
- Create a Google Sheet
- Sign up for Zapier (free plan)
- Connect w/ any other tool
- Trigger event by adding a new row
- Experiment w/ other tiggers
Project #6
Goal: Learn different components of SQL
- Write a SQL query about a fun topic
- Convert some logic into a function
- Call function in query
- Turn query into Stored Procedure
- Run Stored Procedure on a schedule
Project #7
Goal: Learn how websites use databases
- Create a MySQL DB
- Create a local webpage w/ PHP
- Connect to DB w/ PHP
- Run SQL to print a record on page
(No graphics - just plain text w/ a value from the DB.)
Project #8
Goal: Learn how to remotely interact with (and automate) cloud storage
- Sign up for AWS
- Create a S3 Bucket
- Create a local CSV file
- Connect to AWS & load the file w/ Python
As you continue your journey in this wild world of data engineering, don’t forget to practice.
It will help you stay engaged, keep your skills sharp and be a reminder of why you were interested in data in the first place.
Set Your New Data Stack Up for Success with a Free Strategy Call
You’ve modernized your data architecture - now let’s ensure it’s setup for long-term success.
In this Free 30-Minute Strategy Call, we’ll:
- Review your current setup and progress
- Identify potential gaps or risks
- Provide actionable next steps to optimize your process
Whether you’re managing this effort solo or with a small team, you’ll leave with a clear roadmap to confidently build a structured, scalable, and maintainable data architecture.
For best results, please provide as much detail as possible.