What Every Data Team Lead Should Know (Lessons From 30+ Teams)

After working with more than 30 data teams over the last 10 years, the things that hold a team back are almost never technical. They are judgment calls. When to automate something and when to just do it by hand. Whether the naming convention you pick in a rushed meeting is going to outlive everyone in the room. How much polish is worth chasing before you put something in front of a stakeholder. Most of leading a small data team comes down to knowing when to stop refining and get something out the door.

Why perspective matters more than the perfect solution

The first one is not technical advice. Most of them are not.

It is about keeping the end goal in view. That became much clearer to me once I went independent and stepped away from being on a data team day to day.

Most data teams are a cost center

A cost center is a team funded by the business rather than one that generates revenue directly. That is most of us, unless the data itself is the product.

We exist to help a business make better decisions and operate more efficiently. So the job is a balance between building the perfect data product and actually getting results out the door.

Conventions are guardrails, not handcuffs

I have written plenty about the value of good conventions and being particular about how you build. This is the other half of that.

Guardrails let you move fast without leaving the road. You can work freely between them, without going completely outside the lines and without operating with no process at all.

That tension is one of the hardest parts of leading. You can spend all day making something optimal, but if it never reaches the business, what did you actually accomplish?

The best leads are translators

The strongest leaders I have come across in this industry are very good at translation. They talk to the business, understand what it actually needs, and turn that into technical requirements.

They do it while still executing inside their own guardrails. Juggling those two sides is the job.

How to tell when you are over-engineering

Over-engineering is building more solution than the problem requires. It bleeds out of the first point, and I run into it constantly.

Automation is where it shows up most. As technical people it is enticing to custom build everything. It scratches an itch, and the people on your team often want to do it too.

The question to ask first

Here is the scenario. You get one odd request that needs strange logic to import a file. You do it once, or maybe once a year.

What happens a lot is that teams automate or standardize that process so they are ready if it comes up again. Sometimes that is right. Often it is not.

Ask whether building the automation takes longer than doing it by hand the one or two times it will happen. If the answer is yes, you are not being efficient.

And it is not only today's time. Custom solutions carry compounding maintenance for as long as they exist.

Where I see it most

• Automation of rare or one-off processes. The classic case of a pipeline built for something that runs twice a year.
• Reporting. A fully generalized report framework when three stakeholders wanted three numbers.
• Data ingestion. A hand-rolled connector for a source a vendor tool already handles.

If you do build it, document it

Sometimes the answer really is yes, build the thing. In that case, document it thoroughly.

A route I like is video. Use any screen recording tool, walk through what you are doing, and explain it out loud. That often lands better than text nobody reads.

Otherwise you have created new tribal knowledge around an ad hoc process. Two years later somebody opens it, has no idea what it is, and it sits there as tech debt.

Why small decisions stick around longer than you think

Now scale that up. Instead of a one-off import, picture a full data modeling discussion or a bigger project where the team feels rushed.

In those moments you make quick calls. A naming convention. Where something gets stored. A workaround for one awkward scenario.

They almost never get revisited

What I have seen over and over is that decisions like those rarely change. Not unless you are deliberate about it, with a review and refactor step actually built into your process.

So if you are going to set a convention in one scenario, assume it will stick around far longer than you expect.

Small calls compound

Over time these decisions pile up. They turn into tribal knowledge and scattered logic spread across the project.

You make a call today, a few years pass, people turn over, maybe you have moved on. The new team no longer understands why anything was done the way it was.

Let that happen enough times and starting over from scratch begins to look like the reasonable option. That is how migrations get born.

How to work with stakeholders without the friction

This one is about internal communication: talking to stakeholders, to the business, to the people you are doing all this work for.

A lot of teams carry friction in those relationships. There is always internal politics at a company, and things get passive aggressive. That is real.

But what I have found, personally and from watching leads who do this well, is that being proactive, empathetic and open goes a very long way.

The power outage

Where I live we recently had bad storms and lost power for a while. I did not care about the technical reason the power was out.

I cared that I could not turn on my lights, cook food, or keep my fridge running. I just wanted it to work.

Proactive updates would not have fixed the outage, but they would have lowered my stress. Knowing someone is on it and paying attention is most of what people want.

What your stakeholders actually want

They feel the same way about reports. They want the numbers to work, because that is how they do their jobs:

• A sales rep wants their numbers so they can go make more money.
• An operations team wants confidence they are making good decisions.
• The C-suite wants a clear understanding of the business.

How it gets there is not the point for them, and in my opinion it should not have to be.

None of this has much to do with data engineering. It is about being a professional in an organization, and it is easy not to do. Staying in a black box is easier.

Why a data team's work is never really done

Your job as a data team is never truly finished. There is always more to do.

The goal is to get established, with a process you can repeat efficiently. That is not the same as done. New data sources and new requests keep arriving.

In a way I hope that takes pressure off, whether you lead the team or work as an engineer on it. There is no finish line to miss.

The work comes in waves

You will have sprints, or a month, or a whole quarter where you are moving fast and there is a lot to build. Those stretches are where you get your reps in.

Then there is downtime. That is when you refactor, revisit old decisions, and spend time upskilling yourself or your team.

When perfectionism turns into procrastination

This is one I personally struggle with, and it runs underneath most of the points above.

We are striving toward the perfect architecture while also needing to get something delivered. Perfectionism is worth striving for. It keeps you getting better.

It becomes a problem when you are constantly reinventing or tweaking something and never actually shipping it.

AI made this easier to fall into

Oddly enough, I think this got worse with AI chatbots and coding assistants. Rather than getting something done, we spend time looking for validation that the decision we made was right.

I have spent longer coaxing a tool into giving me exactly what I wanted than it would have taken to write it myself. Sometimes I abandoned the tool, did it by hand, and finished sooner.

Ship the bare bones version

From watching other teams, other engineers, and myself, constant tweaking is often just procrastination. Being busy for the sake of looking busy.

You could have stopped a few levels earlier, shipped it, gotten real feedback, and iterated from there.

That ties back to stakeholders. If you treat them like customers and your data team like a business, you want the product in front of customers so you can iterate on what they actually want.

That beats guessing at exactly what they need, building it in isolation, and hoping. With an open line of communication you can do it without burning a bridge.

Related reading

• The Gift and Curse of Small Data Teams
• 3 Strategy Killers for Data Teams
• Are Strict Data Conventions Helpful or Overkill?
• Data Team Stakeholders: Friend or Foe?

Final takeaway

Across more than 30 teams, the pattern that separates the effective ones is not tooling. It is leads who keep the business outcome in view and ship something imperfect rather than polish something invisible. The judgment call you keep postponing is usually the one worth making today.