#038: How to Create a Virtual Machine on GCP

A virtual machine on Google Cloud is a computer you rent in a data center, and you create one through Compute Engine. The sequence is: enable the Compute Engine API, create an instance, give it a name, pick a region and zone, choose a machine type and a boot disk, then let it boot. Once it's running you connect over SSH from the browser, from the gcloud command line tool, or from Cloud Shell, and from there it behaves like any Linux box you'd use locally. Data teams use them to host open source tools, run Python scripts and keep services running outside anyone's laptop. The concepts carry over to AWS and Azure almost one for one, so learning it on one provider gets you most of the way on the others.

Create a new VM

Open the Google Cloud console and navigate to Compute Engine. There are a few routes to it, and the navigation menu is the obvious one.

The first thing you'll be asked to do is enable the Compute Engine API. Almost everything in Google Cloud is an API you turn on per project, and nothing works until this one is active. It can take a couple of minutes.

Then go to VM instances and start a new instance. The form is long, but only a handful of fields matter for a first machine.

Name, region and zone

Give the instance a name. Mine was my-demo-vm. Labels are optional and useful later for grouping and filtering costs across a project.

Region is where the machine physically sits. I used a central United States region, but you should pick one near your users or near the systems it will talk to.

Zone is a specific data center inside that region. The default is fine unless you have a reason to care.

Machine configuration

This is where the price is set. Google groups the options into families:

• General purpose, the sensible default for most workloads
• Compute optimized, for CPU heavy work
• Memory optimized, for large in memory datasets
• Shared core, the smallest and cheapest tier

I stay on general purpose and then pick a machine type inside it, which is what actually sets the CPU and memory. For a demo I use the smallest micro size.

Watch the two numbers the console shows you. The monthly estimate assumes the machine runs all month. The hourly rate is the one that matters if you only need it a few hours a day.

That reframes the decision. A bigger machine running two hours a day can cost less than a tiny one running constantly.

Containers and boot disk

You can build a VM from a container image stored in your container registry, with custom arguments. That's a clean way to spin up many identical machines from one image.

The boot disk is the storage the machine comes up with. By default that's a new persistent disk of around ten gigabytes running a Linux image.

• Image, the operating system the machine boots into
• Disk type, where SSD costs more than standard
• Size, which you can raise now or later
• Custom images and snapshots, for rebuilding from a previous state

I start with the basics here and adjust once I know what the machine is actually doing.

Access, scopes and firewall

Google Cloud runs on service accounts, which are identities for software rather than people. Compute Engine gives you a default service account, and you can swap in a custom one with tighter permissions.

Access scopes limit which APIs the machine can call with that identity. The defaults cover things like storage and service management, and narrowing them is another way to reduce blast radius.

Firewall settings control inbound traffic. How open you leave it depends entirely on what the machine is serving.

The startup script

Under the management section there's a field for a startup script, and it's my favorite option on this page.

It's a plain shell script that runs every time the machine boots. Put the commands that start your tool or service in there and they run on their own after every restart.

For hosting open source data tools, that's the difference between a machine that recovers by itself and one you have to SSH into after every reboot.

Create the instance and give it a few minutes to boot.

Three ways to work with a running VM

The instance page shows everything you configured: networking, the interface, the disk, the service account. Connecting to it is the next step.

Browser SSH

The simplest option is the SSH button on the instance row, which opens a terminal in the browser. It generates and manages the keys for you behind the scenes.

You land in a shell on the machine, logged in as your user, and from there it's a normal Linux box:

mkdir demo_dir
ls

Install packages, run scripts, build whatever you want. This is how I work with VMs most of the time, purely because there's nothing to set up.

The gcloud command line tool

gcloud is Google Cloud's CLI, and it works with VMs and nearly every other resource on the platform. Install it locally and the console hands you a ready made connect command:

gcloud compute ssh --zone "us-central1-a" "my-demo-vm" --project "my-project-id"

Run that from your own terminal and you're on the machine, same as the browser session.

Cloud Shell

Cloud Shell runs the equivalent of gcloud inside Google Cloud itself, so there's nothing to install locally. Paste the same SSH command, authorize it, and it updates the SSH metadata, meaning the keys and permissions, for you.

Run ls and the directory you made in the browser session is already there. Same machine, different door.

The serial port console

Under the connection options there's a list of serial ports, and serial port 1 gives you a live view of the machine's console output.

This is the first place I look when something isn't working. Scroll to the bottom, refresh, and you see line by line what the machine is doing.

It's especially useful with a startup script. You can watch how far the script got, or review what happened on a previous run, without installing any monitoring.

Stop, delete and schedule

At the top of the instance page you get start and resume, stop, suspend and delete. Suspend isn't available on every instance type.

The distinction that costs people real work is stop versus delete:

• Stop shuts the machine down temporarily. The disks, directories and anything you installed survive, and it comes back in the same state.
• Delete removes the instance and the boot disk with it. You start from scratch, as if you'd never created it.

Schedules

You can also create an instance schedule: start time, stop time, dates, and a repeat frequency like daily. Then assign the schedule to an instance.

That's the control that keeps costs honest. You get the machine you need during the hours you need it, and you're not paying for the other twenty.

What data engineers host on them

A VM is just computation in the cloud, so the list of things you can run on one is open ended. Four that come up constantly on data teams:

• An ingestion tool. Airbyte publishes recommended instance sizes and an install path for Compute Engine. You SSH in, run their install commands, and start the service.
• An orchestrator. Install Python and Prefect, or Airflow, then run a worker on the machine so your workflows trigger from there instead of your laptop.
• A messaging platform. Kafka installs and launches on a VM the same way, with the broker running in the cloud rather than locally.
• Plain scripts and services. Python jobs, an internal website, anything you'd otherwise run on a machine under your desk.

The pattern is the same every time: create the machine, SSH in, follow the tool's Linux install instructions, and make sure it restarts cleanly.

That last part is where the startup script earns its place.

Related reading

• Getting Started w/ Prefect (Task Orchestration)
• Getting Started w/ Airbyte (Open Source)
• The Hidden Cost of Open Source
• Data Automation (CI/CD) with a Real Life Example

Final takeaway

I've done this on two separate client projects that needed open source tools hosted somewhere other than a laptop, and both times the VM itself took fifteen minutes. The work that mattered was the startup script and the schedule, because those are what make the machine something the team can rely on rather than something you babysit.