#70: How to Pick Tools for a Data Stack (considerations)

Picking tools for a data stack comes down to three considerations. Money, judged as total cost of ownership rather than sticker price, which means infrastructure cost plus the human resources cost of the people who manage the tool. Maintenance, which is really the build versus buy decision and turns on whether your data work is a competitive advantage for the business. And expectations, meaning the skill set your team has today and plans to have, the vendors you are already committed to, and the source systems you need to connect. My general advice is to start with a lean stack and only add a tool once you have proven you have outgrown what you already own.

Why the decision got harder

Every data team has to choose tooling at some point. Nobody gets to skip it. It matters most at the very beginning and again when you migrate to a new stack.

It has also become harder over the years. More tools, more opinions, more competing best practices, and the result is an overwhelming conversation.

It is still not a decision to rush. It shapes how effective your team is, how far you can scale, and what the whole thing costs.

Consideration 1: Money

Money is where everyone starts, which is fine. What I would change is the frame: look at total cost of ownership rather than the price of any single tool.

The way I think about it, total cost of ownership has two components.

Infrastructure cost

This is the cost of the tool itself, and it usually splits into licensing and usage.

• Licensing. Cloud tools commonly charge per user per month, sometimes in blocks of users.
• Usage. Processing fees. Separated storage and compute on a database, or volume and credit based billing on an ingestion tool.

This is the part that gets all the attention in a tooling conversation. It is also, I would argue, not the biggest part.

Human resources cost

The other half is paying people to manage, maintain and build with whatever you picked. For most teams this ends up larger than licensing and infrastructure combined.

Here is the trap. You go all self hosted open source to save on licensing. The saving is real, so the tool looks cheaper and the decision looks obvious.

What went up instead is your human resources cost. Your people, or you, now spend significantly more time setting it up, managing it and troubleshooting it.

People do not work for free. That time is racking up cost under the hood, and it is also time not spent building pipelines and delivering reports.

None of that means you should always pay for tools or never use open source. It means total cost of ownership has to include both halves.

Consideration 2: Maintenance

Maintenance overlaps with cost, but it deserves its own heading because it is the question you will live with daily. How much of your time is this tool going to ask for?

Stated plainly, this is the build versus buy decision. Build means self hosting open source. Buy means paying a licensing fee for a SaaS product that someone else hosts.

There is no universal right answer. In my experience across a lot of teams, buying usually turns out to be the better call, for a couple of reasons.

Is your data work a competitive advantage?

The first reason is whether the infrastructure you build is what gives the business an edge over its competitors, or whether it exists to produce insight that supports the business.

For most data teams, the data team is not the product. That is a hard thing to say as a data engineer, and it is still usually true.

We support the business. We are often not the reason the company's product is differentiated in the market. Sometimes we are, and when that is genuinely the case custom is worth it.

The honest risk is getting a little too cute. Building something unique and custom is fun, and fun is not the same as advantageous.

The tax analogy

Here is the comparison I keep coming back to. Think about your personal finances for a month.

You notice expenses: bills, subscriptions, the things you actively pay. But the largest single expense most people have is income tax, and you barely register it because it is withheld before the money reaches you.

Self hosted maintenance is withheld the same way. You do not feel it monthly because it is lumped into salaries you were paying anyway.

It never arrives as an invoice, so it never gets compared against one. It is still frequently the most expensive thing in the stack, and it is taken straight out of your capacity to deliver results.

If you work at a large, deeply technical company, or custom infrastructure genuinely is your advantage, building can be right. Most of us are on a small team or working alone, still trying to match what big tech companies do, and paying for the privilege.

Consideration 3: Expectations

The third consideration is the current and future state of your team. Three things sit under it.

Skill set

The tool should match what your team can actually do today. Be honest about that, and about whether their time is well spent on this at all.

Then look forward. If you want a tool your team cannot support yet, you are implicitly deciding to hire for it. That is a legitimate plan, as long as you have named it as one.

Existing vendors

Where you already are matters a lot. A full Microsoft shop is unlikely to pick BigQuery or another Google database, and a Google shop is unlikely to go the other way.

Staying inside one ecosystem makes billing simpler, management simpler and your life easier. Flip flopping without a reason is a cost you pay quietly.

The exception is strategy. If you plan to change vendors or deliberately want to stay vendor agnostic, that is a decision about where you want to be in a few years, not about this quarter's convenience.

Source systems

The last one is mostly about ingestion: how you extract and load data into your central landing zone.

Not every tool connects to every system, and the ones that do vary in performance and reliability. This is where the build versus buy question reappears as a custom connector question.

The practical approach is to cover roughly 90 percent of your important sources with one tool, then either build the remaining edge cases or pay someone who specializes in them. Make sure your most important sources land cleanly and accept that there will always be more work to do.

Start simple and add on

Picking a stack is personal to your team and your circumstances. If I had to give one piece of general advice, it is this: start with a bare minimum, lean set of tools and only add when you see the need.

Most tools do more out of the box than people use. It may not be perfect, but it is usually enough to get the job done.

Only go looking once you have outgrown a capability you already own. The reason is simple: once a tool is in your stack it is very hard to take out.

Start slow, prove the need, and adding the tool feels like progress rather than a mistake you have to live with.

Related reading

• Build vs Buy: What's best for your data team?
• Common Pricing Models of Data Tools
• The Hidden Cost of Open Source
• Avoiding Vendor Lock-in With Your Data Stack

Final takeaway

When I help a team select tooling, the question that changes the outcome most is not which tool is best. It is how many hours a month this will quietly take from the person who already has a full job, because that number is the one nobody put in the comparison.