ecom-observability-lab
Go microservices on AWS EKS built to break on purpose. Product, cart, and payment services over gRPC, fully instrumented with OpenTelemetry. Injected failures via flagd and validated the observability pipeline — Jaeger traces, Grafana dashboards, Locust load testing at 4000+ requests.
A production-grade e-commerce system designed to fail. Three Go
microservices — product, cart, and payment — communicating over
gRPC, deployed on AWS EKS, fully instrumented with OpenTelemetry.
$ The chaos experiment
Used flagd (OpenFeature) to inject a simulated failure into the
payment service mid-load. Locust was running 50 concurrent users
at ~30 RPS when the flag flipped.
$ What the observability stack caught
- Grafana: error rate spiked immediately to 7.04%
- Jaeger: distributed traces showed exactly where the gRPC call
failed in the payment service
- Prometheus: scraping metrics from all pods via otel-collector
$ Stack
- Backend: Go microservices over gRPC
- Infrastructure: AWS EKS, RDS Postgres, VPC, ALB
- Observability: OpenTelemetry → Jaeger + Grafana + Prometheus
- Payment: Stripe Mock
- Frontend: React
- Load testing: Locust (4000+ requests, 0% failure rate on happy path)
- Feature flags: flagd via OpenFeature
$ Result
Zero errors on the happy path.
7.04% error rate caught immediately when chaos was injected.
The pipeline worked exactly as designed.