Build a modern data architecture you can trust

Step 1: Design

Learn how to design a simple, yet modern end-to-end pipeline

The first step is to establish a clear understanding of common design patterns so you can feel confident picking a strategy or learning a tool.

Without a strategy, you may look busy but will feel completely overwhelmed.

But with one, you’ll have more clarity & your efforts will have more productive, compounding results.

Step 2: Setup

Gain hands-on experience setting up common data tools

With our design in hand, we'll then configure each core component of the data stack, establish roles/permissions where needed & ensure connectivity between them.

You'll learn how to setup & use tools such as: Postgres, Sling, dbt, Metabase, GitHub & Docker.

Step 3: Build

Confidently build a data architecture from scratch with best practices

In this step, we'll implement a single foundational end-to-end data pipeline that touches all 5 core components and establish Production vs Development environments.

This sets the standards & conventions for which all future work on the architecture will follow.

By keeping the scope small, you’re able to pay close attention to each layer while also showcasing quick wins.

(Yes, we'll cover data modeling too)

Step 4: Automate

Discover how to automate your development workflow

Next, it's time to create automated workflows that validate data quality before reaching Production & ensure our data models stay updated each day.

Automation makes the development cycle more efficient, reduces egos/emotions in code reviews & overall provides more control over the architecture. 

Once you get this right, it's hard to work on any other team without it in place.

Step 5: Maintain

Learn how to maintain & scale your architecture long-term

Lastly, we'll create assets that provide transparency into the design, organize important information & overall establish a culture of high quality development long-term.

The biggest cost in any architecture is maintenance, which compounds over time.

Being proactive will save countless hours later while enabling developers to have fun creating new features vs troubleshooting.

Resources (BONUS)

Access custom resources to prepare you for real implementations

As a student, you'll also get access to tons of resources to help you implement your Simple Stack faster & maintain it more effectively:

This includes (but not limited to):

• A Simple Stack Checklist (Excel)
• Pre-made workflow automation jobs (yaml)
• Pre-written documentation, style guides & more
• Data Architecture designs

Each of these are based on real-world implementations & will save you a ton of time!

We'll cover key topics data engineers & team leaders need to understand such as:

• How to think strategically about designing an end-to-end architecture that's simple & easy to follow
• How to create a foundational data model (facts & dimensions) so you can build pipelines with a clear strategy
• How to properly set database roles & permissions so you stay in control of user access
• How to automate testing & deployments so you can spend more time on development
• Best practices & tips from 10 years as a developer, consultant & educator so you avoid common mistakes 

In my experience, the best engineers are the ones that see the bigger picture of their work, not just the individual task.

And my goal with this Playbook is to help you start approaching data like an architect, not just another developer.

Imagine walking into your office, opening up your laptop and instead of feeling like you're treading water, you feel like you're solving a fun puzzle.

As a beginner, your focus is often on coding or individual tasks. But in my experience, the top performers approach it more abstractly (like a mental puzzle).

And with the right understanding you too can start to think higher level.

By learning & applying the information in this Playbook, you will:

• Feel less pressure to learn every tool or skill (you'll instead be able to think through scenarios and adapt)
• Become the go-to person on your team & have teammates wondering how you know so much
• Be able to outline any architecture in a few minutes
• Complete tasks much faster than others (work smarter, not harder)
• Identify design optimization opportunities & propose solutions
• Build your own stack from scratch
• Fast-track your career growth by learning to contribute at a higher-level, without wasting precious time trying to do it all alone

The Playbook course is focused on using all open-source, self-hosted tools. 

This avoids the need to create multiple accounts & keeps the experience consistent for everyone.

For this, we will create an end-to-end data playground using the following tools:

• Local Development - Hosted Virtual Machine (6 Month Access)
• Database - Postgresql
• Ingestion - Sling Data
• Transformation - dbt Core
• Version Control & CI/CD - GitHub
• Reporting - Metabase

Note: The intention here is not to say these are the BEST tools for any team or ones that you should always use. That is highly dependent on your scenario (as discussed more within the lessons). 

Like many others, I started my data career using drag-and-drop tools and traditional databases.

Then I joined a team using dbt, Snowflake and GitHub and felt overwhelmed with how to actually use it. It was all so different.

Fast-forward to today and I now help Fortune 100 companies to greenfield startups implement data architectures with modern workflows and cloud databases. 

I'm both a Microsoft & dbt Certified Developer and have reached millions of aspiring engineers through my YouTube Channel (@KahanDataSolutions).

But this took years of practice, trial and error and learning on the job.

This Playbook not only teaches you step-by-step how to implement a data architecture from scratch, but shares tips, tricks and lessons based on my real-life experience as a data consultant.

You'll fast-track your learning, avoid common mistakes, and feel confident contributing to any project.

I hope to see you inside!

While orchestration is an important topic, this course does not go overly deep into it.

In my opinion, dedicated orchestration tools (ex. Airflow, Prefect, Dagster, etc.) are excellent, but also not necessary when first getting started.

They are components to be added later after you've set strong foundations of your initial end-to-end stack.

Instead, you'll learn to establish baseline orchestration via GitHub Actions as daily cron jobs.

This is sufficient to keep the initial data models refreshed each day & is an approach I often use with real client implementations. 

Having said that, your playground stack is just that - it's yours!

I encourage you to take what we build in the course and continuously add to it.

This includes introducing dedicated orchestration tools (or anything else you can think of).

This will not cover how to manage cloud computing for big data or administering cloud providers (ex. AWS, Azure, GCP).

It also does not touch on AI or Machine Learning.

I also personally do not have experience in the field of Machine Learning/AI and therefore would not be the right person to teach you about it. 

These are all worthy topics that could have entire courses dedicated to them, but are outside the scope of this particular course.

Instead, the focus is setting the foundations of an end-to-end stack for internal analytics using common tools & workflows. 

To see the specific tools used, please refer to the "What tools will we be using?" question.

I recommend you have a have a solid baseline understanding of SQL, Git & general database design (schemas, tables, etc.).

Generally speaking, this is intended for more mid to senior-level engineers.

However, more junior-level engineers are welcome to join if they have some experience under their belt.

In short, no.

The data will be a very small amount stored across two sources:

1. PostgreSQL database 
2. Google Sheets

Both sources are managed outside of the Playbook course but access is given to each student to load it into their own environment.

Why not "big data"? 

I don't believe having more data helps to better understand the concepts.

If anything, it makes it messier and distracts from the material.

But you can still scale it up

By the end, you'll have created your very own playground stack where, on your own, you can:

• Add more/bigger data sources
• Add streaming
• Add whatever you want

It's your own personal playground and the sky is the limit. 

Absolutely - Although I recommend adding your own unique touch to your stack by adding new data sources, data models, reporting, etc.

Otherwise it'll look too generic.

You can certainly mimic the open-source stack we'll create as a proof-of-concept design at your company if you'd like.

In fact, it's very similar to one's I've seen at real clients. 

However, as mentioned in the course, the tools used are simply for example purposes and do not necessarily mean they are the "best" options for you.

There are a TON of great data architecture videos on YouTube created by both myself and other content creators. 

While you can learn pretty much anything from YouTube, you are going to spend significantly more time trying to put all the pieces together on your own.

Not to mention you may not be learning in the right order.

This Playbook will give you a step-by-step guided path for building data architectures in a tested way.

You'll also get access to MUCH more.

For example, pre-created resources, a private hosted workspace, custom data sources & guidance from me along the way.

In short, you will learn more in less time and in the right order.

All material is hosted on my website (KahanDataSolutions.com).

Once you enroll, you will be prompted to create a username/password which you will use to log in anytime.