Building an Event Streaming Platform at Scale

Overview

In early 2024, our event ingestion pipeline was a single PostgreSQL write path behind an HTTP endpoint. At 50K events/sec, it was hitting connection pool exhaustion every few hours. At 100K, the database was on fire. The business was growing 30% month-over-month and we had roughly six months before the system would fall over completely.

This case study documents the architecture we designed, the decisions we made along the way, and what we'd do differently. The system now handles 2M events per second with p99 latency under 10ms at the ingest layer.

The Problem

The original system worked fine at launch. Events came in via HTTP POST, got written to a events table in PostgreSQL, and a cron job consumed them for downstream processing. Classic queue-over-database pattern.

Three problems emerged at scale:

Write amplification. Every event write hit the primary. PostgreSQL's WAL replication meant even reads on replicas caused write load. At 50K events/sec, we were generating ~800MB/s of WAL traffic.

Consumer coupling. Eleven different consumers were polling the same table with different filters. Query plans degraded. Autovacuum couldn't keep up. Table bloat became a weekly incident.

No backpressure. The ingest endpoint was synchronous — if the database was slow, the HTTP response was slow. Under load spikes, this caused cascading timeouts across upstream services.

Architecture

We moved to a three-tier architecture: an ingest layer that accepts and validates events, a streaming backbone via Kafka, and a set of consumer services that hydrate downstream systems.

The ingest layer is stateless Go services behind a load balancer. They validate, enrich, and produce to Kafka. No database writes on the hot path. Kafka provides durability and decoupling.

type EventProducer struct {
  writer *kafka.Writer
  schema *avro.Schema
}
 
func (p *EventProducer) Publish(ctx context.Context, e Event) error {
  payload, err := avro.Marshal(p.schema, e)
  if err != nil {
    return fmt.Errorf("marshal: %w", err)
  }
  return p.writer.WriteMessages(ctx, kafka.Message{
    Key:   []byte(e.UserID),
    Value: payload,
  })
}

Engineering Decisions

Challenges

Consumer rebalancing under load. When we deployed new consumer instances, Kafka's rebalance protocol caused processing to halt for 15–45 seconds across all consumers in the group. We migrated to cooperative-sticky rebalancing, reducing pause times to under 2 seconds.

Schema registry as a single point of failure. We discovered mid-incident that our producers would crash if the schema registry was unavailable, even when writing known schemas. We added a local schema cache with a 24-hour TTL as a fallback, which eliminated this failure mode.

Outcome

The migration took eight months, including a two-month shadow mode where both systems ran in parallel. We went live with zero downtime using a gradual traffic shift.

Peak throughput went from 100K events/sec (at the point of failure) to 2M events/sec with headroom. P99 ingest latency dropped from 850ms to 7ms. The engineering team went from weekly on-call escalations to one incident in six months post-launch.

More importantly, we decoupled the business's growth from the database's growth. Adding a new consumer is now a matter of writing a consumer service — no schema changes, no migrations, no coordination with the ingest team.