What Every Data Team Lead Should Know (Lessons From 30+ Teams)

After working with more than 30 data teams over the last 10 years, the things that hold a team back are rarely technical. They're judgment calls. When to automate something and when to just do it manually. Whether the naming convention you pick today is going to stick for longer than you expect. How much perfect is worth chasing before you ship. Most of it comes down to knowing when to stop spinning your wheels and get something out the door.

Key takeaways

• A data team is usually a cost center. The job is helping the business decide better and move faster, not shipping the most elegant solution.
• Before you automate, ask whether building the automation takes longer than doing the task manually the one or two times it will actually happen.
• Small decisions like naming conventions almost never get revisited. Assume whatever you pick today is permanent.
• Stakeholders don't care how the numbers get produced, only that they work. Proactive updates lower their stress more than technical explanations.
• The work has no finish line. It comes in waves, and the quiet stretches are when you refactor, document and upskill.
• A lot of the time, constantly tweaking something is just another form of procrastinating. Being busy for the sake of being busy. Get something bare bones out there, get real feedback, then iterate.

Why perspective matters more than the perfect solution

This first one isn't technical advice. Most of these aren't.

It's about keeping the end goal in mind. This became a lot clearer to me once I went independent and stepped away from being on a data team day to day.

At the end of the day, we're here to help a business make better decisions and operate more efficiently by giving them good data products and good insights. With that in mind, the job is a balance between building the perfect data product and actually getting results out the door.

I've talked before about the value of good conventions, having a strategy, and being particular about how you build things. This is the other half of that. As the leader of the team, your job is nuance and judgment.

Conventions give you guardrails. You can move freely between them to get what you need done, without going completely outside the lines or operating with no process at all.

That's one of the hardest parts of leading. Progress as a team, getting things done, and serving the business matter more than perfection on any one thing. You can spend all day trying to make something optimal, but if you never deliver it, what are you actually doing?

Most data teams are a cost center. We exist to help the business move forward and be productive.

The best leaders I've come across in this industry are very good at translating. They can talk to the business, understand what it actually needs, and turn that into technical requirements while still executing within their own guardrails. Juggling those two sides is the job.

How to tell when you're over-engineering

This bleeds over from the first point. Be careful about over-engineering a solution when you don't have to.

I run into this constantly, especially around automation. As technical people, it's enticing to custom build everything. It's exciting. It scratches an itch in your brain as a developer, and the people on your team often want to do it too.

Here's the scenario. You get one odd request that needs some strange logic to import a file. You're doing it once, or maybe once a year. It's rare.

What happens a lot is that teams automate or standardize that process so they'll be ready if it comes up again. Sometimes that's the right call. But often it isn't.

The question to ask first: will building the automation take longer than doing it manually the one or two times it will actually happen?

If the answer is yes, you're not being efficient. And it isn't only the time you spend today. Those solutions carry compounding maintenance behind them.

The areas where I see it most:

• Automation of rare or one-off processes
• Reporting
• Data ingestion

Sometimes the answer really is yes, build it. In that case, document it thoroughly.

A route I like is video. Use any screen recording tool, walk through what you're doing, and explain it out loud. That often lands better than text documentation nobody reads. Nobody enjoys documenting, and you can use AI to help with it, but there needs to be some reference.

Otherwise you've just created the beginnings of new tribal knowledge to manage an ad hoc process. In a couple of years somebody else looks at it, has no idea what it is, and it sits there as tech debt.

Why small decisions stick around longer than you think

Now scale that up. Instead of a one-off import, imagine a full data modeling discussion or a bigger project where you feel rushed.

In those moments you make quick decisions. A naming convention. Where something gets stored. A workaround for one scenario.

What I've seen over and over is that decisions like those rarely get changed. Not unless you're deliberate about it, unless you specifically plan to revisit and refactor. The vast majority of the time, it never changes.

So if you're going to set a convention or a practice in one scenario, assume it's going to stick around far longer than you expect.

Over time these small decisions compound. They turn into tribal knowledge and scattered logic. You make a call today, a few years pass, people turn over, maybe you've moved on, and the new team no longer understands why anything was done the way it was.

Let that happen enough times and you end up with so much scattered logic that starting over from scratch looks like the reasonable option.

How to work with stakeholders without the friction

This one is about internal communication. Talking to stakeholders, to the business, to the people you're actually doing all this work for.

For some reason a lot of teams carry friction in those relationships. There's always internal politics at a company, and things can get passive aggressive. That's real.

But what I've found, both personally and from watching leaders who do this well, is that being proactive, empathetic and open goes a very long way.

Your stakeholders are other people trying to get something done. They don't care about the specifics of how it gets accomplished, and in my opinion they shouldn't have to.

Where I live we recently had bad storms and lost power for a while. I didn't care about the technical reasons the power was out. I cared that I couldn't turn on my lights, cook food, or keep my fridge running. I just wanted it to work.

Your stakeholders feel the same way about reports. They want the numbers to work, because that's how they do their jobs:

• A sales rep wants their numbers so they can go make more money
• An operations team wants to make good decisions
• The C-suite wants a clear understanding of the business

How it gets there isn't the point for them.

Going back to the power outage, if I'd been getting proactive step by step updates, I'd still have wanted the power on. But I'd have felt better. Knowing someone is working on it and paying attention brings the stress down. That's what people want, to know their needs are being attended to.

None of this has much to do with data engineering specifically. It's about being a professional and working with people in your organization. But it's easy not to do. It's easier to stay in a black box, do your work, and not reach out.

One of the keys to a genuinely data driven company is getting everybody rowing in the same direction. You're all on the same team and you work for the same company. There's no reason to have a hostile relationship, or at least no reason to be the one causing it.

Why the work is never really done

Your job as a data team is never truly done. There's always going to be more to do.

The goal is to get established, with a process you can rinse and repeat efficiently. But that's not the same as finished. There are always new data sources and new requests. You don't hit a finish line and stop.

In a way I hope that takes some pressure off, whether you're leading the team or working as an engineer on it.

Things come in waves. You'll have sprints, or a month, or a whole quarter where you're moving quickly and there's a lot to build. Those are the moments where you're honing your craft and getting reps in.

Then there's downtime, and that's when you refactor, revisit decisions, and spend time learning and upskilling yourself or your team.

The technology keeps changing, so you want to stay on top of it. That's one of the genuinely good things about working in this industry. You're constantly growing and challenging yourself to stay good at what you do.

When perfectionism turns into procrastination

This is one I personally struggle with, and it runs underneath most of the points above.

We're striving toward the perfect architecture while also needing to get something done. Perfectionism is a good thing to strive for. It keeps you getting better and closer to the line.

It becomes a problem when you're constantly reinventing or tweaking something and never actually shipping it.

Oddly enough, I think this has gotten worse with AI chatbots and coding assistants. Rather than getting something done, we spend a lot of time looking for validation that what we're building or the decision we made is the right one. A computer will now validate your decision, and you can keep going like that indefinitely.

If you'd just done it and figured it out, you might have moved faster. I've had plenty of moments where I spent longer trying to tweak an AI tool into giving me exactly what I wanted than it would have taken to write it myself. There have been times I abandoned the tool entirely, did it by hand, and finished sooner.

That will probably improve over time. The point is being aware when you're doing too much, because at some stage you need to deliver the thing you're working on.

What I've found from watching other teams and engineers, and from watching myself, is that constantly tweaking and reworking is often just another form of procrastination. Being busy for the sake of looking busy.

You could have stopped a few levels earlier, shipped it, gotten real feedback, and iterated from there.

That ties back to stakeholders. If you treat them like customers and your data team like a business, which is how I look at it, then you want your product in front of customers so you can iterate on what they actually want. That beats guessing at exactly what they need, building it in isolation, and hoping.

Most of the time you're better off getting something bare bones in front of them and iterating. With an open line of communication and a good relationship, you can do that without burning a bridge, and it becomes a collaborative process everybody feels good about.

Key terms

Tech debt

Shortcuts and one-off solutions that work today but cost you time every time someone touches them later.

Tribal knowledge

Process that only lives in someone's head. When that person leaves, the team inherits something nobody can explain.

Cost center

A team funded by the business rather than generating revenue directly, which is why delivered results matter more than technical elegance.

Common questions

How do I know when automating something is worth it?

Compare the build time against the manual time. If writing the automation takes longer than doing the task by hand the one or two times it will actually come up, it isn't worth it. Factor in maintenance too, because that cost compounds for as long as the solution exists.

Is it bad to do something manually?

No, as long as you're not creating unnecessary tech debt and you document what you did. A manual process that's recorded and explained is much safer than an automated one nobody understands.

How strict should naming conventions be?

Strict enough that you'd be comfortable living with them for years, because you probably will. These decisions almost never get revisited unless revisiting them is deliberately built into your process.

What should I tell stakeholders when something breaks?

Tell them early, tell them plainly, and keep telling them. They don't need the technical reason. They need to know someone is on it and roughly when it will be working. Proactive updates lower stress even when the problem hasn't been fixed yet.

How do I know if my data team is doing a good job?

Look at whether you're consistently delivering things the business uses, not whether your architecture is optimal. A team that ships useful data products and communicates well is doing better than one with a perfect model nobody has seen yet.

Where this comes from

These are observations from working with more than 30 data teams over the last 10 years. Some of them you're probably already doing well. Others might be worth a second look.

If you're leading a data team and you're in the middle of a migration, or thinking about how you want to move your architecture forward and you'd like an outside opinion, that's what I do. Get in touch here and we can talk through what you're working on.