Fix Your Data Pipeline...From The Start

The landing zone is where your source data first lands in your database, or wherever you store raw data, before any modeling touches it. Most teams rush past it because modeling and automation are more interesting, and that is exactly why so many warehouses turn into a mess. Three things make the landing zone worth real attention: it is the cheapest place to establish naming conventions that scale, it makes database permissions simple to scope, and it lines up directly with how your transformation project is organized. Whatever structure you set here trickles down through staging, marts and reporting. Spend an hour on it now or spend a quarter untangling it later.

What I mean by the landing zone

In data engineering, most people prefer to talk about data modeling or automation. The area I think is equally important, and far more often overlooked, is the landing zone.

The landing zone is where source data first arrives in your database, or in whatever storage holds your raw data. It is the output of ingestion and the input to everything else.

One rule keeps it useful: raw stays raw. The landing zone holds the data as it arrived, and every change to it happens in the layer above.

It is the foundation for the rest of your pipeline. Which is why skipping past it to get to the fun parts causes problems that show up much later, in places that look unrelated.

Reason 1: It is where conventions get established

The approach I see most often is that teams mimic whatever the source calls itself. The connector says the table is named a certain way, so that is what it is called in the database.

It feels easy at first. It is something you can fly through at the beginning of a project.

Why source names stop working

The problem shows up as you add sources. Names get mixed up, objects scatter across the database, and the collection stops making sense as a whole.

Then someone new joins the team, opens the database, and cannot tell what a given schema represents. In my experience that is one of the fastest ways to make a mess of a database.

Two conventions that scale

The landing zone is a great place to fix this, and it does not take much:

• A separate raw database. Call it raw, or whatever signals raw data only, and name the schemas inside it after each source. The grouping tells you what it is.
• A raw prefix. If everything lives in one database, prefix ingested objects with raw_ plus the source name so they group together on their own.

-- a database for raw data only, one schema per source
raw.salesforce.accounts
raw.stripe.charges

-- or one database, with a prefix doing the same job
analytics.raw_salesforce__accounts
analytics.raw_stripe__charges

Simple, but it adds real organization, and other people can see immediately what is going on.

The scaling part is the point. You know the convention, so every new source is rinse and repeat instead of another judgment call.

As my dad would say, this is easy to do and easier not to do. That is why it gets skipped, and why doing it with intention pays off.

Reason 2: It simplifies security

A clear, structured landing zone makes roles and permissions much easier to reason about.

If everything raw lives in a raw database, or in raw schemas, those are the only objects your loading tools need access to. There is nothing to think about beyond that boundary.

One role, many users

The approach I follow is a single role called loader, which acts as the bucket for those permissions. Each loading tool then gets its own user, and each user is assigned to that role.

create role loader;
grant usage on database raw to role loader;
grant create schema on database raw to role loader;

create user fivetran_svc;
grant role loader to user fivetran_svc;

create user airbyte_svc;
grant role loader to user airbyte_svc;

It keeps things organized, and it stops those tool users from accidentally reaching anything further down the pipeline.

All of that is possible because you were specific about the landing zone. The alternative is handing a tool an admin user because nothing in the database is organized enough to scope it properly.

For how the rest of the roles fit around this one, see The 4-Role Security Model for Your Data Warehouse.

Reason 3: It aligns with your transformation project

The third reason is that a clean landing zone lines up with the way your transformation project is already organized.

I use dbt a lot, and a dbt project has subdirectories for different sources. If your raw data is separated by source, those two structures mirror each other exactly.

models/
  staging/
    salesforce/
      stg_salesforce__accounts.sql
    stripe/
      stg_stripe__charges.sql

What the alternative costs

When the raw layout does not match anything, developers carry the mapping in their heads. Which schema holds the billing data, which prefix means what, who renamed that table last year.

That knowledge never gets written down, and it walks out the door with whoever leaves first. A convention is the cheapest documentation you will ever write.

The payoff is that a developer never has to guess. You are in the project, you see the source directory, and you know precisely where that data lives in the database.

No questions, no tribal knowledge, no asking the one person who set it up. It is clear and it makes development life easier.

It trickles down

Look at this from a higher level. When you are intentional about the landing zone, the mentality and the organization carry into the rest of the project.

Why would you establish conventions in one part of a project and not the rest? If you make the effort here, it applies to your data modeling, your marts layer and your reporting.

You are effectively setting the expectation for what the rest of the stack looks like. That is a lot of leverage for a decision that costs an afternoon.

My suggestion is to go look at your current landing zone and work out whether it follows any of this, or where you could introduce one of these ideas to improve the design you already have.

Related reading

• The First Step in a Data Architecture (Ingestion)
• The 4-Role Security Model for Your Data Warehouse
• Data Architecture w/ dbt (Putting It All Together)
• How to Build Modern Data Architectures Faster

Final takeaway

When I come into a warehouse that feels chaotic, the landing zone is almost always where it started. Decide how raw data is named and grouped before the first source loads, and the conventions, the permissions and the project structure all fall out of that one decision.