Simulating failure on AWS EKS — what the observability pipeline caught
I built a fake Versace store, broke it with a feature flag, and traced the exact failure path in under two minutes.
I built a fake Versace store on AWS EKS and then broke it on purpose. Here's what the observability stack caught.
The store never sells anything real. But the infrastructure is completely real — microservices in Go, a React frontend, PostgreSQL on RDS, the whole thing running on EKS. I built it to answer one question: if something goes wrong in production, how fast can I actually find it?
So I broke it. Flipped a feature flag, watched the error rate climb to 7.04%, and traced the exact failure path through Jaeger in under two minutes. That's what this post is about.
$ What I Built
Four services, one database, one cluster:
| Service | Stack | Responsibility |
|---|---|---|
| Frontend | React + Vite | Luxury shopping UI |
| Product Service | Go | Product catalog API |
| Cart/Order Service | Go | Cart management & order processing |
| Payment Service | Go | Stripe-powered payments |
Backed by PostgreSQL on AWS RDS, orchestrated with Kubernetes on EKS, and fully observable via an OpenTelemetry-first stack.
$ Architecture
┌────────────────────────── AWS EKS Cluster ──────────────────────────────┐
│ │
│ React Frontend → Product Service │
│ → Cart/Order Service │
│ → Payment Service │
│ │
│ ↓ All services emit traces & metrics ↓ │
│ │
│ OTel Collector → Jaeger (traces) + Prometheus (metrics) │
│ ↓ │
│ Grafana (dashboards) │
│ │
│ flagd (OpenFeature) — feature flags across all services │
│ Locust — load testing │
│ Stripe Mock — payment simulation │
└──────────────────────────────────────────────────────────────────────────┘
↓
AWS RDS PostgreSQLEverything runs in two Kubernetes namespaces: apps for the application services and observability for the telemetry stack.
$ Why OpenTelemetry?
Before OpenTelemetry, observability was fragmented — each vendor had its own SDK, its own agent, and its own data format. Switching providers meant rewriting instrumentation code.
OpenTelemetry solves this with a vendor-neutral standard for traces, metrics, and logs. You instrument once, and you can export to Jaeger, Prometheus, Datadog, Grafana Cloud — whatever you choose.
In this project, each Go microservice is instrumented with the OpenTelemetry Go SDK:
import (
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
// Wrap the HTTP handler — zero manual span creation needed
router.Use(otelhttp.NewMiddleware("product-service"))
Every inbound HTTP request automatically creates a span with attributes like HTTP method, status code, route, and duration. Outbound calls to the database propagate trace context so Jaeger can stitch together the full request journey across services.
$ The Chaos Experiment — This Is the Part That Matters
I ran Locust against the cluster to simulate real shopping traffic: users browsing, adding to cart, checking out. Steady state. Then I flipped a single feature flag.
client := openfeature.NewClient("product-service")
failureEnabled, err := client.BooleanValue(
context.Background(),
"productCatalogFailure",
false,
openfeature.EvaluationContext{},
)
if err == nil && failureEnabled {
http.Error(w, "Simulated Product Catalog Failure", http.StatusInternalServerError)
return
}productCatalogFailure → true. No redeploy. No code change. One API call to flagd.
What happened next:
The Grafana error rate panel went from near-zero to 7.04% within one scrape interval. Not a gradual climb — a step function. Every /products request the frontend made was now returning a 500.
I opened Jaeger and filtered for error spans. Immediately visible: the GET /products trace showing a red error span on the product-service, with the status message Simulated Product Catalog Failure attached as a span attribute. The trace showed the frontend making the request, the OTel Collector receiving the span, and the exact line of execution where it failed — all linked by a single trace-id propagated through the HTTP headers.
The frontend showed degraded state gracefully. But without Jaeger, you'd be grepping logs across pods wondering why the product page was broken. With it, the failure path was a single click.
I flipped the flag back. Error rate dropped to zero inside 30 seconds. The Grafana panel returned to baseline. The whole exercise — break, observe, recover — took about four minutes.
That's the point of building this. Not the Versace branding. That four-minute loop.
$ Feature Flags with OpenFeature + flagd
The chaos experiment above used flagd as the feature flag backend and the OpenFeature SDK as the client interface. OpenFeature is vendor-neutral — swap flagd for LaunchDarkly or Unleash without changing application code.
Beyond chaos engineering, I used flags for:
- Gradual rollouts — enable features for a percentage of users before full release
- Instant kill switches — disable a broken feature without a redeploy
- Environment-specific behavior — different configs for staging vs. production without branching
$ Infrastructure as Code with Terraform
The entire AWS infrastructure is defined in Terraform:
- VPC with public and private subnets across availability zones
- EKS cluster with managed node groups
- RDS PostgreSQL in private subnets (no public IP — as it should be)
- ACM certificates for HTTPS
- ALB Ingress Controller via Helm
- SSM Parameter Store for secrets
One thing I got wrong early: publicly_accessible = true on the RDS instance. That's a misconfiguration when the instance sits in a private subnet with no public IP — the ENI can't be assigned one anyway, so it silently fails to be publicly accessible while also being a bad idea. Set it to false. Route all database traffic through the VPC.
$ The Observability Stack
Distributed Tracing — Jaeger
Every service exports traces to the OTel Collector, which forwards them to Jaeger. When a user places an order, you can pull up a single trace and see:
- Time spent in the Cart service validating the order
- The exact PostgreSQL query that ran and how long it took
- The outbound call to the Payment service
- Any errors, with full context
This turns "something is slow" into "the payment service's DB query is slow on orders with more than 5 line items" — in seconds.
Metrics — Prometheus + Grafana
Each service exposes a /metrics endpoint scraped by Prometheus. Grafana dashboards visualize:
- Request rates per service
- Error rates and status code distributions
- Database connection pool usage
- Custom business metrics (orders per minute, cart abandonment rate)
Load Testing — Locust
Locust generates realistic traffic patterns — browsing products, adding to cart, completing checkouts — so you can observe the system under load and catch performance regressions before users do. It's also what made the chaos experiment meaningful: without baseline load, a 7% error rate is just one failed request.
$ Kubernetes Deployment Highlights
OTel Operator for Auto-Instrumentation
The OpenTelemetry Operator can inject instrumentation as a sidecar for polyglot services via annotations:
annotations:
instrumentation.opentelemetry.io/inject-nodejs: "true"For the Go services here, I initialize the SDK explicitly at startup — Go's static linking means sidecar injection doesn't work the same way. But for anything else in the cluster, the operator handles it automatically.
Namespace Separation
apps/ → frontend, product-service, cart-order-service, payment-service, flagd
observability/ → otel-collector, jaeger, prometheus, grafana, locustSeparate namespaces means separate RBAC, separate resource quotas, and the ability to tear down the observability stack without touching the application. Do this from day one.
Health and Readiness Probes
Every deployment has properly configured livenessProbe and readinessProbe so Kubernetes knows exactly when a service is healthy and ready to receive traffic — preventing failed deployments from ever reaching users.
$ Lessons Learned
1. Instrument early, not as an afterthought.
Retrofitting OpenTelemetry into an existing codebase is painful. Starting with it from day one means every feature you add is observable by default.
2. Feature flags belong in production.
flagd + OpenFeature gave me the ability to simulate failures in production-like environments without code changes. This made chaos testing and gradual rollouts trivially easy.
3. Private subnets for databases — always.
Never expose your RDS instance publicly. Route traffic through the VPC. The extra configuration is minimal; the security benefit is significant.
4. Separate your namespaces.
Mixing application and observability workloads in a single namespace causes operational headaches. Keep them separate from the start.
5. The OTel Collector is your friend.
Don't export traces directly from each service to Jaeger. The Collector batches, filters, and routes telemetry — it's the single place to change your observability backend without touching application code.
$ What's Next
- Auto-scaling with KEDA based on custom metrics from Prometheus
- Service mesh (Istio or Linkerd) for mTLS and automatic traffic management
- Log aggregation with the OTel log pipeline feeding into Loki + Grafana
- SLO dashboards tracking error budgets per service
Built with Go, React, OpenTelemetry, Kubernetes, Terraform, and AWS EKS.
The full source is on [GitHub]:https://github.com/JobinAj/ecommerce-opentelemetry-demo.git.
-- null