Kubernetes Controllers: Kubebuilder & Go
Managing complex stateful software (like PostgreSQL databases, Redis clusters, or custom application deployments) using raw Kubernetes static YAML manifests (Deployment, Service) quickly hits operational boundaries. Static manifests cannot handle automated database failover, schema migrations, or dynamic backup routines when infrastructure conditions change.
The Kubernetes Operator Pattern extends the Kubernetes API by pairing Custom Resource Definitions (CRDs) with custom controller loops written in Go. Kubebuilder is the official CNCF framework for scaffolding production-ready Kubernetes Operators. By leveraging controller-runtime and client-go informers, a custom controller continuously drives cluster state toward the declared spec. This guide details CRD struct design, idempotent Reconcile() loops, event watches, and unit testing using envtest.
Mental Model: Static Imperative Manifests vs Declarative Kubernetes Reconciliation Loops
Imperative scripts execute kubectl apply once and exit. If a managed database primary node crashes 2 hours later, imperative scripts remain oblivious.
Declarative Reconciliation Controller Architecture runs an infinite event-driven control loop:
1. Observe: Informers watch etcd for changes to Custom Resources (DatabaseCluster) or child resources (Pods, Services).
2. Analyze: The controller compares the Observed State (actual cluster condition) against the Desired State (spec in the CRD).
3. Act: The Reconcile() method creates, updates, or deletes Kubernetes resources to eliminate the drift. For admission control and scaling, review building custom k8s admission webhooks open policy agent and optimizing kubernetes cluster autoscaling karpenter keda.
Quick reference
- Extends Kubernetes API natively using declarative Custom Resource Definitions (CRDs).
- Control loops run continuous level-triggered reconciliation to eliminate infrastructure drift.
- Informer caches watch etcd resources over HTTP/2 WebSockets without polling API servers.
- Automates complex operational knowledge (failover, backups, scaling) directly in Go code.
- Powers core cloud-native tools like Prometheus Operator, cert-manager, Karpenter, and CoreConcept.
Remember this
Build custom Kubernetes controllers with Kubebuilder to automate complex operational workflows.
Designing Custom Resource Definitions (CRDs) with Controller-Gen Annotations
In Kubebuilder, CRDs are defined as Go structs annotated with controller-gen markers:
1// +kubebuilder:object:root=true2// +kubebuilder:subresource:status3// +kubebuilder:printcolumn:name="Replicas",type="integer",JSONPath=".spec.replicas"4type DatabaseCluster struct {5 metav1.TypeMeta `json:",inline"` 6 metav1.ObjectMeta `json:"metadata,omitempty"` 7 8 Spec DatabaseClusterSpec `json:"spec,omitempty"` 9 Status DatabaseClusterStatus `json:"status,omitempty"` 10}11 12type DatabaseClusterSpec struct {13 // +kubebuilder:validation:Minimum=114 Replicas int32 `json:"replicas"` 15 Engine string `json:"engine"` // "postgres" or "mysql"16}Quick reference
- Go struct markers (+kubebuilder:validation) generate OpenAPI v3 validation schemas automatically.
- Subresource :status separates desired state (spec) from observed operational condition (status).
- printcolumn markers format custom columns for kubectl get databasecluster CLI outputs.
- OpenAPI schemas validate user input directly on the API server prior to etcd persistence.
- Guarantees type-safe resource definitions across polyglot Kubernetes development teams.
Remember this
Annotate Go CRD structs with controller-gen markers to generate OpenAPI schemas and CLI columns.
Writing the Reconcile() Loop: Idempotency, Status Updates, & Event Watched Caches
The core logic of a controller resides in the Reconcile() function:
1func (r *DatabaseClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {2 var dbCluster v1alpha1.DatabaseCluster3 if err := r.Get(ctx, req.NamespacedName, &dbCluster); err != nil {4 return ctrl.Result{}, client.IgnoreNotFound(err)5 }6 7 // Ensure StatefulSet child resource exists8 desiredSts := r.buildStatefulSet(&dbCluster)9 if err := r.CreateOrUpdate(ctx, desiredSts); err != nil {10 return ctrl.Result{RequeueAfter: 5 * time.Second}, err11 }12 13 // Update Status14 dbCluster.Status.ReadyReplicas = desiredSts.Status.ReadyReplicas15 r.Status().Update(ctx, &dbCluster)16 return ctrl.Result{}, nil17}Quick reference
- Reconcile() MUST be completely idempotent; running it 10 times in a row produces identical state.
- client.IgnoreNotFound handles resource deletion events cleanly without throwing errors.
- CreateOrUpdate utility pattern creates missing child resources or updates existing drift.
- Status().Update() mutates resource status subresources without triggering spec reconciliation loops.
- RequeueAfter schedules periodic background health checks for external non-k8s dependencies.
Remember this
Write idempotent Reconcile() functions that compare observed state against desired spec.
Testing Operators via EnvTest & Production Deployment Metrics
Kubebuilder includes envtest, a lightweight testing framework that spins up a real local etcd and kube-apiserver binary in memory:
1// Unit Testing Operator with EnvTest2func TestDatabaseClusterController(t *testing.T) {3 g := NewWithT(t)4 dbCluster := &v1alpha1.DatabaseCluster{5 ObjectMeta: metav1.ObjectMeta{Name: "test-db", Namespace: "default"},6 Spec: v1alpha1.DatabaseClusterSpec{Replicas: 3, Engine: "postgres"},7 }8 g.Expect(k8sClient.Create(ctx, dbCluster)).To(Succeed())9 10 // Verify StatefulSet was created by reconciler11 sts := &appsv1.StatefulSet{}12 g.Eventually(func() error {13 return k8sClient.Get(ctx, types.NamespacedName{Name: "test-db", Namespace: "default"}, sts)14 }, 5*time.Second, 100*time.Millisecond).Should(Succeed())15}Quick reference
- envtest runs real etcd and kube-apiserver binaries locally for sub-second controller integration tests.
- Avoids requiring slow Docker / Minikube cluster deployments during local TDD development cycles.
- Prometheus metrics registry exposes controller reconcile latency and queue depth metrics automatically.
- Leader election protocol (leaderelection) ensures active-standby HA across multi-replica controller pods.
- Delivers battle-tested production reliability for cloud-native infrastructure automation.
Remember this
Use envtest to write fast integration tests against local in-memory kube-apiserver instances.
Key takeaway
To test Kubebuilder locally, run kubebuilder init --domain company.com --repo company.com/operator and kubebuilder create api --group apps --version v1alpha1 --kind DatabaseCluster.
Related Articles
Explore this topic