A Disposable Production Clone on a Laptop

Most of my mistakes in GitOps-managed infrastructure aren’t logic errors. They’re mechanical: a values path that doesn’t exist in the chart, a template that renders differently than I pictured, a secret reference whose key is subtly wrong, a sync ordering I got backwards. All of them are cheap to catch and expensive to catch in a real cluster, where the feedback loop is a pull request, a review, a merge, and a reconcile.

So I built a copy of the control loop on my laptop. Not the workloads — the loop.

What’s in it

• A local Kubernetes cluster in a Linux VM, running a lightweight distribution, pinned to the same Kubernetes minor version as production. The pin is the point: version drift is exactly the class of surprise this is meant to catch.
• A local git server, holding the repository the GitOps controller watches.
• The GitOps controller itself, configured the same way it is in the real clusters, reaching the git server over in-cluster DNS.
• A cloud-API emulator for the AWS services we actually depend on — secrets manager, object storage, queues, notifications, parameter store, identity.
• The external-secrets operator, wired to that emulator, so secret injection follows the same path it does in production instead of being replaced by literal values.
• A workflow engine, installed through the same app-of-apps pattern as everything else.

Three scripts: bring it up, refresh the git contents, tear it down.

The bit that makes it usable

There’s a skeleton directory in the repo that represents what the controller sees. Edit files there, run the push script, and it force-pushes into the local git server. The controller reconciles within about thirty seconds.

That’s the whole inner loop: edit → push → watch reconcile, on a timescale where you can iterate. Same layout as the real repository, same app-of-apps structure, same ApplicationSets. When something works locally, the change is a copy-paste into the real repo rather than a translation.

The force-push is deliberate. Local history is worthless — I’m not collaborating with anyone here, I’m re-rendering. Treating the local git server as a mutable staging buffer instead of a repository removes all the ceremony that git otherwise imposes on experimentation.

What it catches, honestly

Real value:

• Chart values paths. The single most common thing I get wrong, and the one it catches fastest — you find out in thirty seconds that your carefully-nested override renders nothing.
• Template rendering: what a chart actually produces with your values, as manifests you can read.
• The secret-injection path end to end: the operator, the store definition, the reference keys, the resulting Secret’s shape.
• Sync ordering and health gating. Ordering deadlocks — where a resource in an earlier wave can never become healthy because its dependency is in a later one — reproduce perfectly here, and they are genuinely confusing to debug against a real cluster where each iteration is a merge.
• Workflow templates, which are otherwise tedious to iterate on remotely.

Not what it’s for:

• Performance or capacity. It’s a laptop.
• Anything involving real cloud IAM semantics. The emulator gives you API shapes, not the policy evaluation engine.
• Private-registry images, which simply won’t pull — the app shows as degraded while template rendering still validates fine. That distinction is worth knowing so you don’t chase a red status that means nothing locally.
• Charts hosted in a private OCI registry, which also won’t resolve. Git-sourced apps are the target.

I’ve stopped being bothered by that list. The failures I actually ship are structural, not performance-related, and the emulator is fine for “does this reference resolve” even when it’s useless for “is this policy correct.”

Gotchas worth writing down

The container runtime that builds your images isn’t the one running your pods. The VM’s Docker daemon and the Kubernetes distribution’s containerd are separate stores. Building an image locally does not make it available to the cluster, and the resulting ImagePullBackOff sends you looking for a registry problem that doesn’t exist. The bridge:

docker save <image>:<tag> | <vm> ssh -- sudo k3s ctr images import -

then set imagePullPolicy: IfNotPresent in the manifest, or the kubelet will try to fetch a tag that exists nowhere remote.

Give the VM enough memory. Six gigabytes minimum with this stack. Below that, things fail in ways that look like application problems — evictions, unexplained restarts, controllers timing out — and you’ll debug the wrong layer for an hour.

Keep a full reset in reach. Sometimes the cluster gets into a state that isn’t worth understanding, and the correct response is to destroy it. That’s the actual feature here: production teaches you to repair, because destroying isn’t an option. Locally, repair is usually the wrong instinct. A rebuild takes minutes and starts from a known state.

The general idea

The insight I’d extract, having built variations of this a few times: replicate the control loop, not the workloads.

I don’t run the applications locally. They’re irrelevant to the mistakes I make. What I need faithful is the chain — git to controller to cluster to secrets provider — because that’s where the mechanical errors live, and every link in it is something I can run on a laptop for free.

Which means the fidelity budget goes to specific places: the Kubernetes version, the controller’s configuration, the reconciliation path, the secret-injection mechanism. Everything else gets faked, stubbed, or skipped without apology.

The result isn’t a staging environment and shouldn’t be sold as one. It’s a rendering harness with a real reconciler attached — and it turns “push a PR and find out in twenty minutes” into “find out in thirty seconds,” for the entire category of error I’m most likely to make.

The setup is on GitHub: gceraso/local-gitops-sandbox.