Our team records a lot — meetings, interviews, lectures, demos. I needed transcriptions fast, accurate, and private. Sending audio to a cloud speech-to-text API meant every conversation touching a third-party server. That was a non-starter.

I already had a small on-premise Kubernetes cluster with a mix of NVIDIA GPUs sitting underutilized. The question was: could I build something production-grade on it?

The answer turned out to be yes — and it taught me a lot about running ML workloads on Kubernetes.

$ Architecture Overview

The platform has four moving parts:

Browser → FastAPI Backend → Redis Queue → GPU Worker Pods
                ↓                              ↓
           MongoDB               faster-whisper (CUDA)
                ↓
           Ollama (phi3:mini) — CPU-only LLM summarization

Everything runs in a media-sirpi namespace on Kubernetes. Deployments are managed by Flux GitOps — push to main, the cluster reconciles automatically.

$ The GPU Cluster

Four worker nodes, each with a different GPU:

| Node | GPU VRAM | Whisper Model | Use Case |
|---|---|---|---|
| node1 | 8 GB | whisper-medium | Standard jobs (5–30 min audio) |
| node2 | 16 GB | whisper-large-v3 | Long files (>30 min or >300 MB) |
| node3 | 4 GB | whisper-small | Quick/draft transcriptions |
| node4 | 8 GB | whisper-medium | Standard jobs (secondary) |

Jobs are routed to the right queue by file size and probed duration:

def route_to_queue(file_size_mb: float, duration_secs: int) -> str:
    if duration_secs > 1800 or file_size_mb > 300:
        return "transcription:large"
    return "transcription:standard"

Node affinity is enforced via node labels (gpu-vram: "8", "16", "4") and GPU taints (gpu=true:NoSchedule). Workers only land on nodes they can fit on.

$ The Stack

- faster-whisper — CTranslate2-optimized Whisper inference. Roughly 4× faster than OpenAI's original implementation at equivalent accuracy.
- FastAPI — async Python API, handles file uploads, auth, SSE streaming, and job dispatch.
- Redis — job queue (BRPOP-based worker loop) and pub/sub for real-time status events.
- MongoDB — persists transcription records, status, content, and summaries.
- Ollama + phi3:mini — local LLM for AI summarization. No data leaves the cluster.
- KEDA — scales GPU worker pods to zero when queues are empty, saving GPU memory for other workloads.
- Flux CD — GitOps reconciliation. The cluster tracks main and reconciles on every push.

$ File Upload Transcription

When a user uploads a file:

1. The FastAPI backend stores the file in Ceph object storage (S3-compatible).
2. A job is pushed to the appropriate Redis queue with a job_id and object_name.
3. The GPU worker pod picks up the job via BRPOP, downloads the file to a temp path, and runs faster-whisper.
4. Results (text, duration, language) are written to MongoDB and a status event is published via Redis pub/sub.
5. The frontend polls via Server-Sent Events (SSE) and shows the transcription as soon as it completes.

Users can upload multiple files at once — each gets its own job and tracks independently.

$ Live Transcription

For real-time transcription, I run a dedicated whisper-live service. The browser records audio in WebM/Opus chunks and posts them every ~2 seconds to a /live/transcribe endpoint.

The live pod:
- Starts at replicaCount: 0 — the backend scales it up on demand.
- Runs whisper-small on node3 (the 4 GB Quadro K1200).
- Returns transcribed text with a status field: "ok", "silent", or "unclear".

When status is "unclear", the UI shows: "Could you speak a little louder?" — giving users immediate feedback instead of silent failures.

$ The Hallucination Problem

Early on, both the live and batch workers were producing garbage output — random Korean, Hindi, or repetitive phrases like "Yeah. Yeah. Yeah." or "one one one one one one."

This is a known Whisper behavior called hallucination. The root cause: when condition_on_previous_text=True (the default), Whisper feeds its own previous output back as context for the next segment. If the audio is quiet or unclear, it loops its own output.

The fix was a set of parameters applied consistently across both workers:

segments, info = model.transcribe(
    tmp_path,
    beam_size=5,
    condition_on_previous_text=False,  # breaks the hallucination feedback loop
    no_speech_threshold=0.6,           # suppresses segments with low speech probability
    compression_ratio_threshold=2.4,   # rejects repetitive/looped output
    log_prob_threshold=-1.0,           # drops low-confidence segments
)

For live transcription, I added an additional VAD (Voice Activity Detection) guard:

if info.duration_after_vad < 0.5:
    return {"text": "", "status": "silent"}

This prevents running inference at all on chunks that are effectively silence after VAD filtering.

$ AI Summarization — Keeping It Private

After transcription, users can request an AI summary. The requirement was clear: no cloud APIs. Meeting content can be sensitive.

I chose Ollama running phi3:mini — Microsoft's 3.8B parameter model, quantized to Q4 (~2.3 GB), with a 128K context window. It runs on CPU so it doesn't compete for GPU VRAM with the whisper workers.

- Ollama pod runs on node1 or node4 (8 GB VRAM nodes, but using CPU only).
- The phi3:mini model is cached on a hostPath volume at /data/ollama — no re-download on pod restarts.
- OLLAMA_KEEP_ALIVE=5m unloads the model from RAM when idle, freeing memory for the OS.
- The FastAPI backend streams Ollama's token-by-token output directly to the browser via SSE.

The summary endpoint caches results in MongoDB so repeated requests don't re-run inference:

# Return cached summary instantly
if existing.get("summary"):
    yield f"data: {json.dumps({'token': existing['summary'], 'done': True})}\n\n"
    return
Stream from Ollama
async with httpx.AsyncClient() as client:
    async with client.stream("POST", f"{settings.ollama_url}/api/generate", ...) as response:
        async for line in response.aiter_lines():
            chunk = json.loads(line)
            yield f"data: {json.dumps({'token': chunk['response'], 'done': False})}\n\n"

On the frontend, since the EventSource API doesn't support custom headers (needed for JWT auth), I used fetch with ReadableStream:

const response = await fetch(/api/transcriptions/${id}/summary/stream, {
    headers: { Authorization: Bearer ${token} }
});
const reader = response.body!.getReader();
// ... decode chunks and stream tokens into the UI

$ GitOps with Flux

The entire cluster state lives in Git. Flux watches the main branch and reconciles on every commit.

Image automation handles rolling deployments: when CI builds a new Docker image and pushes it to Docker Hub, Flux's ImagePolicy and ImageUpdateAutomation detect the new tag, update whisper-release.yaml, and trigger a HelmRelease upgrade — zero manual kubectl needed.

One gotcha I hit: Flux caches Helm charts by version. If you add new templates to the chart but don't bump Chart.yaml's version field, Flux keeps rendering the old cached chart and your new templates never deploy.

My fix was to auto-bump the chart minor version in CI on every backend build:

- name: Bump chart version
  run: |
    VERSION=$(grep '^version:' backend/k8s/Chart.yaml | awk '{print $2}')
    MAJOR=$(echo $VERSION | cut -d. -f1)
    MINOR=$(echo $VERSION | cut -d. -f2)
    PATCH=$(echo $VERSION | cut -d. -f3)
    NEW="$MAJOR.$((MINOR + 1)).$PATCH"
    sed -i "s/^version: .*/version: $NEW/" backend/k8s/Chart.yaml
    git add backend/k8s/Chart.yaml
    git commit -m "chore: bump chart to $NEW [skip ci]"
    git push

Now every backend CI run increments the chart version, ensuring Flux always fetches and re-renders fresh templates.

$ KEDA Scale-to-Zero

GPU workers are idle most of the time. Leaving them running wastes VRAM and prevents other workloads from using the GPU.

KEDA's ScaledObject watches the Redis queue length. When the queue is empty for more than 2 minutes, it scales the worker Deployment to zero replicas. When a job arrives, it scales back up within 15 seconds.

workers:
  keda:
    pollingInterval: 15    # check Redis every 15s
    cooldownPeriod: 120    # wait 2 min before scaling to zero

The tradeoff: the first job after a scale-up has extra latency while the pod starts and the Whisper model loads into VRAM (~30–60 seconds for large-v3). Subsequent jobs in the same burst are fast. For my use case (batch transcription, not latency-critical), this is a good tradeoff.

$ Lessons Learned

1. Whisper hallucinations are real and need explicit mitigation.
The default settings are optimized for accuracy on clean audio. In production with noisy uploads, you need condition_on_previous_text=False and the threshold parameters, or you'll get looping gibberish.

2. Helm chart versioning matters for Flux.
If you treat the chart version as "bump only when chart logic changes," you'll eventually be confused by Flux silently not deploying your new templates. Auto-bump it in CI.

3. CPU inference for small LLMs is viable.
phi3:mini does adequate summarization on CPU in a few seconds for typical meeting transcripts. You don't need a GPU for everything.

4. VAD before inference saves a lot of noise.
Filtering silent audio chunks before sending them to Whisper dramatically reduces garbage output and saves GPU compute.

5. EventSource doesn't support headers.
If your SSE endpoint requires authentication, you need fetch + ReadableStream on the frontend, not EventSource.

$ What's Next

- Speaker diarization — identifying who said what, using pyannote.audio
- Frontend CI/CD pipeline — the React frontend currently has no automated build/deploy pipeline
- Transcription search — full-text search across the MongoDB transcription collection
- Webhook notifications — push to Slack/Teams when a long transcription completes

---

Full source on GitHub: We can discuss if you buy me a coffee (repo is confidential)

Questions or corrections → [[email protected]]