The AI Workflow for Data Engineering

The AI workflow for data engineering is not copying code into a chat window. It is a coding agent working inside your repository: you describe what you want, it proposes a plan, you approve or correct it, and then it writes the models, runs the build, opens a pull request and lets your CI/CD pipeline deploy it. The walkthrough below uses Claude Code with a dbt project on Snowflake, Sling for replication and Evidence for the dashboard. The whole change, a new data mart plus a new dashboard page, took 20 to 30 minutes with no code written by hand. What made it work was knowing what good looked like and correcting the agent when it drifted.

Why this isn't ChatGPT

A lot of data people still associate AI with chat prompts. You paste in some code, you get a result back, maybe you get autocomplete.

Working with a coding agent is a totally different paradigm, and honestly a lot more powerful. It's hard to wrap your head around until you see it, which is why a walkthrough beats an explanation.

The setup here is Claude Code running inside Visual Studio Code, pointed at a real project.

The project, built without writing code

Over roughly the span of a work day, I built an end to end data platform. I did not write a single line of the code. I instructed and I observed.

• Source data. A set of CSV files in cloud storage, three sources and around 16 tables. Nothing complicated, just enough to work with.
• Snowflake. A setup script created a raw database, an analytics database, a small warehouse, roles and permissions.
• Sling. The tool used to replicate the source data into the warehouse.
• dbt. The transformation project, in its own repository, following a three layer data model.
• Evidence. The visualization tool serving the sales overview dashboard.

The documentation came out of the same session. I asked for a quick writeup of what we had just built, and in about a minute it produced an environment guide: what separates loading from transformation, the sources, the definitions, the role hierarchy and what each role can touch.

The repository README covers the systems, the three layer model, project structure, links to the tools, conventions and setup. Same story. I instructed, it wrote.

Asking for a change, and getting a plan first

The request was simple, and I gave it with voice to text: we have a dbt project with a mart around total sales, I want a customer specific tab so I can filter by customer.

Rather than building immediately, the agent thought it through and came back with a plan. That preview is the whole point.

Its plan was to add a page with a customer dropdown filter, querying the existing mart_sales_overview. It noted that all the data it needed already existed and no new dbt model was required.

That would have worked. It just wasn't what I wanted. So I replied that I did not want to reuse the existing mart, and asked for a new dedicated mart table so the data is also available in the database.

It read the project again and came back with a revised plan: a new dbt model, mart_customer_metrics, one row per customer, pulling from specific upstream models.

Plan mode versus letting it run

There are different ways to run this. You can let the agent go on its own, or you can run it in plan mode so it has to show you the plan and get approval.

Plan mode is what I used for the build. The preview is how you catch an approach you don't want before any files change.

Reviewing what it built

The output followed good conventions. Sources, models, tests, facts and dimensions in the warehouse, relationships I had worked through with it, cleanly structured queries.

If I walked in on a team and saw this project, I'd think they were organized and detail oriented. The difference is that it was built by an agent I was directing.

Correcting style once, not every time

One small thing bothered me: the padding between a column and its alias. So I said to use a single space instead.

-- what it wrote
select
    customer_id                 as customer_key,
    sum(order_total)            as lifetime_value

-- what I asked for
select
    customer_id as customer_key,
    sum(order_total) as lifetime_value

Then I told it to make a note to itself so it builds that way going forward. That is the pattern: every little thing it does wrong, you can turn into a standing instruction, and it stops happening.

From there it ran dbt build against the dev target, built the model and ran the tests.

Branch, pull request, deploy

Next I switched to bypass permissions mode to move faster. You would rarely want that in production, because it lets the agent act without asking you first.

I deliberately said nothing about Git, to see whether it would follow the workflow on its own. It noticed we were still on main and branched first.

That came from an instruction I had written once, which it now follows every time:

• Activate the virtual environment
• Pull the latest from main
• Create a feature branch

It staged the files, committed, pushed the branch and opened a pull request with a description and a test plan.

CI/CD did the rest

The pull request kicked off a workflow. I had previously had it build two GitHub workflow files: one for CI on pull requests, one for the production deploy on merge.

The CI run built the models in a test environment and ran the tests, including the unique and not null tests on the new customer mart.

It also flagged something I never mentioned: Evidence points at the production marts, and the new model was only in my dev schema, so the page would not work until the change was merged.

Where it got it wrong

After the merge, the model landed in production but the dashboard page still errored. I knew the real cause was that the tool needed its data source refreshed.

The agent went a different way. It rewrote where the customer dropdown pointed, which solved one problem but not the one I had. Because I was in bypass permissions mode, it just kept going and then told me everything looked good.

That moment is the whole argument for knowing what's going on. If you don't understand the problem, you accept the confident answer and move on.

I pushed back, it refreshed the sources, the cache updated and the page worked. Along the way the context filled up, which is its own thing worth understanding.

What the demo actually shows

We created a new data mart, added the sources, built the table, ran it through GitHub and deployed it to a working dashboard page. Twenty to thirty minutes, hiccups included, no code written by hand.

Two things stand out to me.

• Writing code is no longer scarce. It knows every language, it has the documentation, it is fast. If your only value add is that you can write code and others can't, that is not going to be enough.
• Knowing how code should be written is scarce. How tools work, how a strategy fits together, what good looks like. That is what lets you direct it and course correct.

Code based tools are especially well suited to this, because the agent can navigate the whole project and you can review what changed. dbt is a big one.

Think like a coach, not a player

Here's the analogy that keeps coming back to me. Imagine two basketball teams with identical players: same athletic ability, same potential. The only difference is the coach.

One coach has played for 20 years, has been coached, has been on the court through the ups and downs. The other watched a few videos and is winging it.

The inexperienced coach lets the players figure it out. The experienced one corrects a jump shot every single time: hand here, follow through like this, release it this way.

Once the players learn it, they never forget it. They shoot it the same way every time. Same with defense, screens, spacing, communication, all coached with specifics that only come from having been there.

Same tooling, same roster, completely different results. That's where our interaction with these tools is heading. We have to think like coaches rather than players, which means learning strategy and nuance, not just syntax.

There are real emotions in this. It can be unsettling to watch something develop faster than you do. It also opens up possibilities, because what you can build is now limited mostly by your imagination.

Related reading

• Create a dbt Project from Scratch w/ Claude Code
• How to Structure AI Projects for Data Engineering
• AI For Data Teams (How to Be Ready)
• Why a Good Data Model Makes Your Team More AI-Ready

Final takeaway

The speed in this workflow came from the parts that were already in place: conventions, a three layer model, tests, environments and a deploy pipeline. When I help a team adopt this, that groundwork is what we fix first, because an agent pointed at a messy project just makes the mess faster.