Mastering Terminal Observability: A Practical Guide to Using the gcx CLI for You and Your AI Agents

Overview

The way we develop software is evolving rapidly. Engineers increasingly rely on command-line interfaces and AI-powered coding agents—like Cursor and Claude Code—to accelerate code generation. However, this shift introduces a critical blind spot: while these agents excel at manipulating source files, they lack visibility into the real-time behavior of production systems. A latency spike, a failing SLO, or a broken checkout flow goes unnoticed by the very tool writing your code. That’s where the gcx CLI (Grafana Cloud Command-Line Interface) steps in. It brings full observability into your terminal—and empowers your AI agents to act on production data, not just guesses.

Mastering Terminal Observability: A Practical Guide to Using the gcx CLI for You and Your AI Agents

This guide will walk you through everything you need to know to get started with gcx: from installation and initial setup to using it alongside agentic coding environments. By the end, you’ll be able to instrument a service, set up alerts and SLOs, and manage your entire observability stack as code—all without leaving the command line. Let’s dive in.

Prerequisites

Before you begin, ensure you have the following:

Grafana Cloud account – gcx requires access to a Grafana Cloud instance (free tier or higher). If you don’t have one, sign up at grafana.com.
API key or access token – You’ll need an API key with permissions to manage metrics, logs, traces, alerts, and SLOs. Generate one in your Grafana Cloud portal under “API Keys.”
A terminal – Any modern Unix-like shell (bash, zsh) on macOS or Linux. Windows users can use WSL or a similar environment.
Basic familiarity with the command line – You should know how to run commands and edit files in a text editor.
Optional but recommended – An AI coding agent (e.g., Cursor, Claude Code) configured to run shell commands and interact with your local environment.

Step-by-Step Instructions

1. Install the gcx CLI

Installation is straightforward. Use the following curl command to download and run the installer:

curl -fsSL https://grafana.com/install/gcx.sh | sh

This script works on macOS and Linux. After installation, verify it’s available:

gcx version

You should see the latest version number printed.

2. Configure Authentication

gcx needs to authenticate with your Grafana Cloud stack. Set the required environment variables or use an interactive login:

export GRAFANA_CLOUD_API_KEY="your_api_key_here"
export GRAFANA_CLOUD_STACK="your_stack_name"

Alternatively, run gcx login and follow the prompts. This will persist credentials in a config file (~/.gcx/config).

3. Instrument a Service with OpenTelemetry

One of gcx’s standout features is the ability to wire OpenTelemetry (OTel) into your codebase directly from the terminal. Imagine you have a Node.js service called checkout-service. Run:

gcx instrument add --language node --service checkout-service

This command:

Detects your project structure
Adds necessary OTel SDK packages
Configures exporters to send data to your Grafana Cloud backend
Validates that metrics, logs, and traces are flowing

You can confirm data is landing with:

gcx instrument validate --service checkout-service

If something’s missing, gcx provides hints to fix it.

4. Create Alerts and SLOs

Once your service emits signals, you need to know when things go wrong. gcx can generate alert rules from real data:

gcx alert create --service checkout-service --from-metric "http.server.duration" --condition "p99 > 500ms"

This creates an alert rule that fires when the P99 latency exceeds 500ms over a 5-minute window. To define an SLO for availability:

gcx slo create --service checkout-service --target 99.9 --window 30d --indicator availability

Push it live immediately—no need to touch a web UI.

5. Set Up Synthetic Monitoring

Catch issues before users do by creating a synthetic check:

gcx synthetic create --url https://api.example.com/checkout --frequency 5m --regions us-east-1,eu-west-1

This deploys probes that simulate user requests. Results flow into your Grafana Cloud dashboards automatically.

6. Manage Everything as Code

gcx lets you pull your entire observability configuration into local files:

gcx pull --all --dir ./observability

This generates YAML/JSON files for dashboards, alerts, SLOs, and checks. You can edit them with your agent or manually, then push changes:

gcx push --dir ./observability

This is perfect for version control and collaborative workflows.

7. Integrate with AI Coding Agents

The real power emerges when your agent can call gcx and read production state. For example, if your agent uses a shell command interface, you can instruct it:

"Check the current latency of checkout-service using gcx."

The agent runs:

gcx query --service checkout-service --metric "http.server.duration" --agg p99 --last 10m

It sees the actual P99 value and can adjust its code accordingly—e.g., adding caching or optimizing a database query. This bridges the gap between code generation and production reality.

Common Mistakes

Forgetting to export environment variables – gcx will fail silently if API keys aren’t set. Always verify gcx config show before proceeding.
Running instrument on the wrong directory – Make sure you’re in the root of your project. Otherwise, the OTel setup might be incomplete.
Using mismatched SLO indicators – For an availability SLO, use the --indicator availability flag. Using latency as an availability indicator leads to confusing results.
Not validating data flow – After instrumentation, run gcx instrument validate. It’s easy to assume data is flowing, but a typo in the service name can cause zero emissions.
Pushing before pulling – When managing as code, always pull first to get the latest state, make edits, then push. Overwriting without pulling can delete existing resources.

Summary

The gcx CLI transforms your terminal into a command center for observability. By bringing instrumentation, alerting, SLOs, synthetic monitoring, and as-code management into a single tool, it eliminates context switching and empowers AI agents to make decisions based on real production signals. With the step-by-step instructions above, you can go from a greenfield service to full observability in minutes—all from the command line. Start using gcx today and close the visibility gap between your code and your running system.

Tags: