How to Structure AI Projects for Data Engineering

ai Apr 02, 2026

Structuring an AI project for data engineering comes down to two things: a file that carries your standing instructions, and a layout underneath it that separates instructions from execution. In Claude Code that file is CLAUDE.md, a plain markdown file read at the start of every session, so you stop re-prompting the same context and start treating it as a baseline the agent always has. Underneath it I use what I call the APT framework: agents, playbooks and tools. Playbooks are the written instructions, the SOPs for how a thing gets done. Agents read the relevant playbook and decide what to do. Tools are the scripts, SQL files and API calls that actually run. None of that structure is built in, which is the point: you decide the layout, write it down, and tell the agent what to expect.

Key takeaways

A CLAUDE.md file is persistent instructions. It loads at the start of every session, so the baseline prompt stops being something you retype.
Markdown is the native format here. AI agents read plain language well, which is a shift if you are used to everything being code.
The APT framework splits a project into three layers: playbooks for instructions, agents for decisions, tools for execution.
Instructions like "pull main, create a feature branch, never commit directly to main" get followed every time. That is the practical difference from a style guide people skim once.
Everything in CLAUDE.md consumes context on every session, so keep it tight. A long file means more for the agent to hold in working memory.
Uploading a sample project is the fastest way to get the conventions you want. Let it read the project, write the playbook, then delete the sample.
You still need to understand the underlying tools. The framework only helps if you can tell whether the output follows your best practices.

What a CLAUDE.md file is

A CLAUDE.md file is a markdown file, and markdown turns out to be a surprisingly large part of working with these tools. Coming from a development background that feels odd at first, since we are used to everything being scripts and config.

But agents and language models read regular language well. Plain prose is the interface.

It is a prompt you only write once

When you use a chat tool, you already know a good prompt beats a vague one. Give it a role, a task and an expectation and the answer improves.

A CLAUDE.md file is that idea, made permanent. Instead of one well written prompt for one conversation, you write the baseline once and every future prompt sits on top of it.

Open a new session and it reads the file. Close it, open another, and it reads the file again. You never retype the groundwork.

Think of it as onboarding

The useful mental model is a developer joining your team. You would explain the expectations, the framework, the layout and where things live before handing over a ticket.

That is what goes in the file. Not the task, the context the task depends on.

The APT framework: agents, playbooks, tools

Underneath the file I use a structure I call the APT framework, for agents, playbooks and tools. It is adapted from a slightly different framework I came across, reshaped for data work.

None of it is built in. You decide the structure, then tell Claude Code that this is the structure and this is what to follow.

Layer one: playbooks

Playbooks are the instructions. Think of them as SOPs, the instruction manuals for the things your team does repeatedly.

How to build a dbt project. How to generate and load test data. Each one is a step by step description of what should happen and what the output should look like.

Layer two: agents

The agent is the decision maker, sitting in the middle and directing traffic. It reads the relevant playbook, understands the objective, and then looks to the tools layer to find out how to carry it out.

Playbooks are the instructions, the agent is the one reading them and working out what to do next.

Layer three: tools

Tools are execution. This is where the actual scripts live: bash, Python, SQL files, API calls.

At the bottom of the architecture description I also explain to the AI why the split exists. When it tries to do everything directly without instructions, the results drift. More instruction means more accuracy, more of the time.

project/
├── CLAUDE.md
├── playbooks/
│   ├── build-dbt-project.md
│   ├── generate-test-data.md
│   └── extract-and-load.md
├── agents/
└── tools/
    └── extract/
        └── replicate.sh

Conventions it actually follows

Here is the part that convinced me. In an earlier project I was confident the agent would create a branch and commit correctly without being told each time, and it did. The reason was a short list near the top of the file.

## Before any task

1. Activate the virtual environment
2. Pull the latest main
3. Create a feature branch. Never commit directly to main
4. Check for other relevant context

That is the same thing you would say to a new developer on day one. Always check out a branch before you start.

The difference is that this one gets followed. Plenty of developers know the rule and still skip it or forget it. The agent does not skip it, and when it does miss something you correct the file rather than the person.

The same file also describes the expected file structure, so the agent knows where things live and where new work belongs.

A style guide that gets read

I also keep a project guide describing what a dbt project should look like. Again, imagine telling a developer what the expected design is.

We have all worked somewhere with a style guide that nobody opens. Here you finally have a reader that follows it every time.

When something comes out slightly off, say the spacing in the generated code, you adjust the guide. It is continuous maintenance of a style guide, except the guide is actually applied.

Be mindful of context

One caution. Everything in this file loads at the start of every session, which means it takes up context.

Context is the working memory the agent is holding while it reasons. The longer your instructions, the more it has to carry before it even sees your request.

So keep it tight. My own file is longer than it should be and I will be trimming it. Treat length as a cost rather than a measure of thoroughness.

Expedite it with a sample project

Writing playbooks from a blank page is slow. There is a shortcut, and it is the one I would start with if you are setting this up today.

Upload a sample project that already uses the conventions you want. Tell the agent this is what you expect things to look like, and to follow it.

That is exactly what I did. It read the project, produced the playbook from what it saw, and then I deleted the sample. The conventions stayed behind in writing.

An example playbook and tool together

The clearest example in my project is data replication. The tool is a bash script that uses Sling to extract data and load it into Snowflake.

It could just as easily be Python. The point is that it is the thing that actually runs.

The playbook documents it

Sitting next to the script is a playbook that explains the process and documents how to use it. It is like having someone write up the process for you, once, properly.

# Playbook: extract and load

## What this does
## How to run it
## How to run a specific source
## How to add a new source
## Layout and where files live

Most of the time you are instructing the AI to run these things rather than running them yourself. The playbook is what gives it everything it needs for each specific case.

Scale that across a company

Now think about every process your team has. Ingestion, transformation, updates, formatting, the small recurring chores and the large ones.

Each becomes its own playbook, with the real code in tools, and a CLAUDE.md explaining the architecture so the agent knows how the pieces fit together.

Why you still need to understand the tools

This gets complicated quickly, which is the argument for learning how these agents and frameworks work rather than just pointing them at a repo.

It is also the argument for understanding the underlying tools. You need to be able to steer the output toward what you want to see and confirm it follows good practice. Without that, you cannot tell a good result from a plausible one.

It feels like getting started with dbt

This is open ended and it can feel overwhelming. The closest comparison I have is the first time I used dbt.

dbt gives you some sample models and some notes when you create a project, and then the design is up to you. I follow a three layer design, other teams do it differently, and both work.

What made dbt click was understanding the mechanics well enough to customize it. Same thing here, one layer up.

Instead of deciding what goes in each model directory, I am deciding what goes in each directory of the AI project: playbooks here, tools there, and a description of how they work together.

These ideas will keep evolving as the tools do. What matters is having a framework at all, rather than going in with no structure and hoping.

Key terms

CLAUDE.md

A markdown file of persistent instructions that Claude Code reads at the start of every session, so the project baseline does not have to be re-prompted.

APT framework

My layout for an AI project: agents, playbooks and tools, separating the decision maker from the written instructions and the code that runs.

Playbook

A written SOP for one process, describing the steps, the expected output and how to run the tool that performs it.

Tool

The executable layer of the project: bash scripts, Python, SQL files and API calls that a playbook points to.

Context

The working memory the agent carries during a session. Everything loaded at session start, including your instructions file, uses some of it.

Common questions

What should go in a CLAUDE.md file?

The context a task depends on rather than the task itself: what the project is, the expected file structure, the steps to take before any work begins, and where the conventions are written down. Treat it like onboarding notes for a new developer. Keep it short enough that it is cheap to load every session.

Why markdown instead of a config file?

Because these tools read plain language well. Markdown gives you headings and lists for structure while leaving the content as prose, which is what the agent is best at interpreting.

How long should my instructions file be?

Shorter than feels natural. Everything in it loads at the start of every session and occupies context, so length has a real cost. If a section is only relevant to one process, move it into a playbook the agent reads when it needs it.

Do I have to use this exact framework?

No. Agents, playbooks and tools is the split that made sense to me for data work, adapted from a framework I came across elsewhere. None of it is built in, so you can define your own layers as long as you write them down and tell the agent what to expect.

How do I get good conventions without writing them from scratch?

Upload a sample project that already follows them. The agent reads it, writes the playbook from what it finds, and you can delete the sample afterward. That turns an afternoon of writing into a few minutes of review.

Final takeaway

The teams I work with already have the conventions, they are just scattered across heads, pull request comments and a style guide nobody opens. Writing them into playbooks and an instructions file is the same documentation work, except this time something reads it on every run.

Additional Free Resources

Starter Guides & Checklists

Explore additional free resources built on the same patterns I use with real clients so you can build your own with structure and confidence. Topics include data architecture, modeling and more specifically for small data teams.

Browse Resources