Software & DevOps Engineer

Mahad Amir

An engineer at heart. I like the problems that don't have a tidy answer yet: queues that can't drop a job, clusters that can't go down, pipelines nobody wants to babysit.

Backend systems, distributed architecture, and production Kubernetes. I take systems from a prototype that works on one machine to a platform that runs across accounts, regions, and thousands of deployments.

Open to offers5 case studies · 99%+ uptime

I build the parts of a product that nobody sees until they break. Job queues, worker pools, recovery logic, the Kubernetes underneath it — the systems that have to keep working while everything upstream and downstream misbehaves.

Most of what I've built started as someone's prototype. My job has usually been to take the version that works for one user on one machine and turn it into something that survives real load, real concurrency, and real failure — then own it in production. I'm comfortable being the person who gets paged.

Selected systems

Five systems I designed and ran in production.

Client details are generalised.

800+
YouTube channels automated
80,000+
Videos per day
99%+
Uptime
6–10
Person team replaced

The constraint

The manual version of this work took six to ten people and topped out at a few hundred videos a day. The target wasn't "faster" — it was a volume no team could reach at any headcount. That changes the design problem entirely: every step has to be automated end to end, every failure has to be recoverable without a human noticing, and the whole thing has to run across hundreds of independent accounts that must not interfere with each other.

The system also started life as a single-channel prototype. Everything below is what it took to get from that to a multi-tenant platform serving global agency clients.

Architecture

one job, five stages — each stage retries independentlySCHEDULERGENERATIONPROCESSINGPUBLISHINGCHANNELSjob schedulerper-accountconcurrencygeneration workersk8s podsscaled per workloadmedia processingFFmpegencode / muxpublisher workersper-accountquota aware800+ channelsYouTubeindependent accountsshared state — Redisqueues · limits · locksjob records — PostgreSQLstatus · history · audit

Decisions

Containerised per-workload execution on Kubernetes rather than a fixed worker fleet.

Video and audio processing is bursty and uneven — some jobs are 30 seconds, some are 20 minutes. Fixed workers either sit idle or become the bottleneck. Scaling deployments per workload meant the platform could go from a handful of containers to thousands of Kubernetes deployments without redesigning the execution model.

Account isolation enforced at the scheduler, not in application code.

With hundreds of accounts sharing one pipeline, one noisy tenant can starve every other one. Putting per-account concurrency limits in shared state — rather than trusting each service to behave — meant fairness held even when a single client queued thousands of jobs at once.

Publishing decoupled from generation.

Generation and upload fail for completely different reasons: model refusals and encoding errors on one side, API quotas and auth expiry on the other. Keeping them as separate stages with independent retry meant a publishing outage never wasted generated video, and a generation failure never blocked the publishing queue.

Failure modes

worker crash mid-job
Job stays claimed in the processing set and is recovered on the next worker start rather than silently disappearing
upstream model refusal
Detected as a distinct failure class from a transport error, then retried through a prompt-rewrite path instead of burning the retry budget
per-account quota exhaustion
Account is throttled at the scheduler; other accounts keep running at full rate
frequent platform-wide outages
Root cause was a single-point architecture; replaced with a distributed design, which is what took the platform to 99%+

Outcome

Replaced a six-to-ten person manual operation with a system running 80,000+ videos per day across 800+ channels. Eliminated the recurring outages that had been driving client escalations and held 99%+ uptime across long-running production periods. The platform scaled from a single-channel prototype to a multi-tenant system serving global agency clients, directly enabling new product launches.

Stack

  • Go
  • Node.js
  • Python
  • Kubernetes
  • GKE
  • Docker
  • Redis
  • PostgreSQL
  • FFmpeg
  • Vertex AI
  • GCP
  • Grafana

Experience

Techanzy Limited — Software Engineer

Owned end-to-end architecture, development, and production reliability for backend platforms and the Kubernetes infrastructure underneath them. Operated both self-managed clusters built with kubeadm and managed GKE clusters, running thousands of deployments with monitoring and automated recovery.

Acted as primary technical owner and stakeholder contact for global clients, taking multiple systems from MVP through to production platforms that the business now runs on. Worked alongside and supported junior engineers on delivery.

  • Designed and operated self-managed (kubeadm) and GKE Kubernetes clusters at thousands-of-deployments scale
  • Built distributed backend systems handling hundreds of concurrent workloads across video and audio processing, background job orchestration, and webhook-driven integrations
  • Led solution architecture from MVP to production as primary technical owner and client-facing contact
  • Rebuilt a failing single-point architecture into a distributed one, taking the platform from frequent outages to 99%+ uptime

Stack

What I work with, by layer.

Languages

  • Go
  • Node.js
  • Python
  • Bash

Backend

  • Gin
  • FastAPI
  • Distributed systems
  • Background job processing
  • Worker pools
  • Webhook architectures
  • Rate limiting
  • Recovery patterns
  • Circuit breaking

Infrastructure

  • Kubernetes
  • GKE
  • kubeadm (self-managed)
  • Docker
  • Helm
  • StatefulSets
  • Persistent volumes
  • Liveness & readiness probes
  • Pod anti-affinity
  • NGINX
  • Linux
  • GCP
  • Hetzner

Data

  • PostgreSQL
  • Table partitioning
  • JSONB
  • Index design
  • Redis
  • Redis Streams & consumer groups
  • Distributed locks
  • AOF & RDB persistence
  • Cloud SQL
  • MongoDB

AI models & media

  • Veo 3.1
  • Veo 3 Fast
  • Vertex AI
  • Seedance 1.5 Pro
  • Seedance 2.0
  • Nano Banana
  • Gemini (multimodal analysis, scoring, QA)
  • ElevenLabs
  • Sync.so
  • FFmpeg

Most of my time with these models has gone into their edges rather than their happy path: content-policy refusal handling and prompt-rewrite loops, clip extension ceilings and when to split a chain, long-running-operation polling versus webhook-driven APIs, words-per-second calibration for chunking a script, multimodal quality gating, and fallback chains for when a provider degrades. Knowing what a model will refuse and how it fails is usually more useful than knowing what it can do.

Observability

  • Structured JSON logging
  • Grafana
  • Health checks
  • Cloud Logging
  • Alerting

Web

  • React
  • Next.js
  • Airtable
  • Backblaze B2
  • Creatomate

Writing

Notes from inside the systems.

I'm starting to write up the parts of these systems that were hard to get right — recovery phases in a distributed queue, keeping per-account rate limits fair without starving small tenants, and what actually changes when you move from self-managed Kubernetes to GKE.

Nothing published yet. This is where it'll go.

Outside work

Running a homelab, mostly as an excuse to break things somewhere it doesn't matter. Currently working through Designing Data-Intensive Applications.

Contact

Open to backend and infrastructure roles.

Remote or Europe-based. The fastest way to reach me is email.