#014: Manage & Scale Your Database with Automation

newsletter Oct 08, 2022

 After 8 years of working with databases, I love to be hands-on with managing them.

But this mentality leads to a false sense of control and can even turn into a bottleneck. 

Today, I’ll share how you can better manage and scale your database with automation by using:

  1. Infrastructure as Code (IaC)

  2. Identity Management

  3. Strategic database design

 

Infrastructure as Code gives more transparency and control.

Databases constantly require new objects, users and permissions.

While ad-hoc admin changes are quick, they’re extremely difficult to track.

Code-based approaches provide more organization and wont be dependent on a single user.

The result is more transparency, automation and fewer bottlenecks.

 

Example: Use Terraform to manage a Snowflake environment.

 

 

Identity management integrations consolidate your user base.

New employees or customers aren’t automatically added to your database server.

This means duplicate logins which is not only annoying, but hard to manage.

Fortunately, most platforms can be integrated with identity management (IAM) systems.

This can make on-boarding a breeze and allows for cool features like single sign on (SSO).

 

Example: Azure Active Directory + Snowflake

 

 

A strategic database design simplifies access control.

As new data becomes available, more database objects naturally get created.

But you might find yourself having to update permissions each time.

Try grouping tables into logical schemas, with permissions tied to roles (vs users).

This improves organization and reduces the amount of permissions to set.

 

Example: Create roles based on business or team function

 

 

A well-organized database is the foundation of any strong analytics team.

Once in place, you’ll more easily scale, have fewer bottlenecks and stay focused on developing.

Level-up your abilities as a Data Engineer, faster.

Subscribe to receive tips to improve your skillset as a data engineer every Saturday. Always readable in 2 minutes or less.

Other Recent Posts

#021: Why your data team needs version control

Nov 26, 2022

#020: What to Learn First as a Data Engineer

Nov 19, 2022

#019: 3 Time-Saving dbt Cloud Features

Nov 12, 2022

#018: Lessons Going from Snowflake to BigQuery

Nov 05, 2022