Why Scaling Your Data Architecture Feels Hard (and how to fix it)

Architecture that feels painful to add to usually comes down to three missing pieces, not a lack of skill or effort. There is no clear end to end flow for how data moves from source to report, so people build their own schemas and naming conventions instead of a shared one. The data model is inconsistent, with facts and dimensions created on a whim instead of following a real star schema, so nothing compounds into something reusable. And there is no workflow for moving a change from development to production, so every change feels manual and uncertain. Fix those three and the same architecture stops feeling like a fight every time a new request comes in.

Why no shared agreement on where data lives causes over-engineering

A data strategy is not the same thing as a clear flow. You can list your tools, your database, and dbt and still have no outline for how data actually moves between them.

Without that outline, each engineer decides for themselves. One builds tables in one spot, another spins up their own schema, and nobody agrees on where data is supposed to live.

That disagreement is what turns into over-engineering. With no shared convention, there's no cost to doing more, so people build with different layers, different directories, and different naming every time. It becomes one-off pipelines instead of a system, and the sprawl is what makes the architecture feel complex.

Why an audit starts with your landing zone, not your data model

When teams bring me in, the data model is usually the first thing they want to talk about. It matters, but it's not where I start.

The first thing I look at is how data gets from the source into the database in the first place. I call this the landing zone, and the question is simple: is there a clear, isolated convention for how source data lands, or is it just wherever it happened to go?

Setting a clear landing zone sets a precedent. It tells the rest of the team you're going to be organized and particular about the architecture, and that precedent carries straight into the next layer, your data modeling.

Mental clarity matters as much as the technical setup

A clear landing zone also makes a tool like dbt easier to use, because every source lives in a place you can mentally map. A lot of what makes architecture feel difficult isn't technical. It's the lack of clarity about what you're actually building.

That clarity is what lets a team move efficiently. Without it, this is exactly why it feels like you're starting over on every request. It isn't a sign you don't care about doing a good job. Smaller teams in particular get buried in requests and never get the time to step back and look at the whole flow.

How one distribution company fixed its architecture by isolating sources first

I worked with a distribution company earlier this year that wanted to move fast on the latest tools: streaming infrastructure, open source tools, and AI. Those goals were fine. The real constraint was that they had no clear process behind any of it.

The first win had nothing to do with new tools. We isolated their source data and added clear naming conventions, prefixing source systems so they stayed organized together. That gave the database real structure for the first time.

From there, we moved into modeling the data across the three layers, staging, warehouse, and marts. It felt like slowing down, but it was the step that let everything else they wanted to build actually hold up.

Group your pipeline by layer, not by source

A lot of teams build their pipeline around a specific source system instead of around layers. The better approach is to group everything into layers, usually a raw or staging layer, a warehouse layer, and a marts layer, so any new request has somewhere to flow into instead of becoming its own one-off path.

Why inconsistent facts and dimensions create wasted work

Data modeling is usually the first thing that comes to mind when people think about architecture problems, and for good reason. Most teams start with ad hoc scripts that get a deliverable out the door, then realize none of it is reusable because it's scattered across one-off queries.

When teams try to fix that without a consistent process, it tends to go one of two ways.

Ad hoc scripts that never compound

The first pattern is continuing to write one-off scripts that combine source data and get the result out. It's a short-term win for the stakeholder, but it adds nothing to the architecture's long-term growth. Nothing about it can be built on.

Facts and dimensions that exist in name only

The second pattern is calling things facts and dimensions without actually following a star schema. A fact table holds a business action and its metrics, like a sale or a shipment. A dimension is the descriptive context around it, like a customer or a location.

Without a clear idea of what belongs in each, new facts and dimensions get created on a whim, facts get joined together that shouldn't be, and the model stops making sense to anyone but the person who built it. That's what creates waste: a data model that looks structured but isn't reusable.

Done correctly, a fact and dimension model becomes a compounding asset. You build it once with the business in mind, and every future mart gets built by picking off existing metrics and dimensions instead of starting over.

How one manufacturing company is rebuilding a warehouse modeled around one deliverable

I'm currently working with a manufacturing company rebuilding their core data warehouse. They already had a lot of facts and dimensions in place, built with good intentions under tight deadlines, but not following a clear star schema.

The bigger issue was that most of it was modeled around one big deliverable: moving a report out of SQL Server and into Snowflake. It solved that one need, but the facts and dimensions were tied so closely to that single deliverable that they couldn't be reused for anything else in the business.

That's a common outcome when a team is under pressure to ship one thing. Nobody did anything wrong on purpose. The fix now is refactoring that model so it can grow and compound instead of staying locked to the one report it was built for.

How to trim unused tables out of your warehouse

If you're in this spot, start by looking at your facts and dimensions today. Are they following a consistent pattern, whether that's a star schema or something else you've chosen deliberately? Or are they just tables with a fact or dimension prefix that don't actually function that way?

You also have permission to trim the fluff. Most warehouses I look at have tables built for a use case that never materialized. If something isn't being used and there's no proven need for it, remove it.

• If it's backed by Git, the code is never gone. You can always bring it back.
• Trimming unused tables simplifies what the team has to maintain today.
• Reintroduce something later only once you've proven the need and it can follow your current conventions.

Why a workflow needs guardrails instead of manual deployments

The third area behind a difficult architecture is the workflow: how changes actually move from development into production. When this is missing, pushing a change through feels entirely manual, and testing is manual too.

That uncertainty shows up as people looking to each other for approval just to feel okay about shipping, or batching up a pile of changes to push through once because doing it one at a time feels too hard. This is especially common on teams still early in setting up their warehouse and architecture.

A good workflow should feel like a background process, not a wall. Development, a test or CI environment, and production should have guardrails: tests, automations, isolation. You move through those steps rather than getting stopped by them, and that matters even more now that AI lets people code and ship faster.

Simple guardrails still count

This doesn't need to be complicated to count as a real workflow. I've set up simple cron jobs and GitHub pipelines for teams of four people or fewer, sometimes for a single person.

You can use AI tools like Claude or ChatGPT to help build the automation itself, as long as you already know what you want it to do and where it fits in the bigger picture.

How one healthcare company automated testing and deployment

I worked with a healthcare company where the VP of operations, who wasn't deeply technical but knew enough SQL to be dangerous, needed automation around his changes. We introduced three things: automated tests that run when he opens a pull request on GitHub, automated deployment once a change is approved and merged, and an automated daily refresh.

None of it was complicated, and it included basic security roles. It let him introduce a change, know exactly what process it would go through, and get flagged if something broke, all without getting in his way.

What to look at in your own process

Next time a new request comes in, pay attention to the steps between intake and delivery. Write them out if you can. Where does it feel manual? Where are you repeating the same copy and paste work over and over?

You don't need to build all of it at once. Add one automation, see how it goes, then add the next. Start simple, prove the need, and only add complexity when it's earned.

The three things to fix: architecture, modeling, and workflow

These three areas work together. The end to end flow and the landing zone set the convention your data model follows. The data model is what compounds into something reusable instead of one-off scripts. The workflow is what lets you move changes through all of it without the process itself becoming the blocker.

None of this gets fixed overnight. But addressing these three incrementally is what gets an architecture to the point where new requests stop feeling like starting from scratch.

Related reading

• Fix Your Data Pipeline...From The Start
• Data Modeling Crash Course: The 3 Layered Model
• Why Bother With Naming Conventions?
• The Art of Data Team Workflows

Final takeaway

When I audit a team's architecture, I almost never start with the data model, even though that's what everyone wants to talk about first. I start with how data lands, because that's the convention everything else inherits, and the model and the workflow only hold up once that's in place.