Engineering Notes by Kiran Sabne — deep dives into PostgreSQL performance tuning, database internals, AI & vector search workloads, CDC pipelines, and backend systems at scale. I'm a backend and database engineer with 8+ years working on production PostgreSQL, Aurora/RDS, Go, AI & Machine learning. This blog covers real production incidents, query optimization, indexing strategies, replication, and data pipeline design — the stuff you learn debugging systems at 3am. New posts weekly.
PROBLEM A Nested Loop join runs its inner side once per row on the outer side. That's the cheapest possible join when the outer side is small — and a real cost multiplier once it isn't. The operator i
"We have an index on that column, why is this still slow?" — because the index isn't broken. It's just fat. An index that used to fit in a handful of pages can silently grow to several times its neede
If a table's disk size keeps climbing even though the row count hasn't — and queries against it have gotten slower for no reason you can point to — you're looking at the same root cause almost every t
If you've ever run EXPLAIN ANALYZE on a slow PostgreSQL query and stared at a wall of text with no idea what to do next, this guide is for you. We'll go from "what is this tree" to "here's exactly how
Efficient pagination is absolutely critical when dealing with millions of rows in PostgreSQL — especially if you care about performance, index usage, and UX for APIs or UI pages. 1. OFFSET/LIMIT Pagin
When you’re running PostgreSQL with 100M+ rows, the difference between “it works” and “it flies” is all about tuning—but not the generic, copy-paste advice you get everywhere. Here’s the advanced, bat
Imagine this: Your high-traffic app is humming along smoothly. Most data requests are being served instantly from Redis or other, blazing-fast cache layer. But then — a cache key expires, and within milliseconds, 10,000 clients hit your backend at on...
In this post, I’ll walk you through how I built a semantic similarity search API using FastAPI, Sentence-BERT (SBERT), and PostgreSQL with the pgvector extension.This project began as a proof of concept (POC) to explore how seamlessly we can integrat...
If you’re indexing huge, append-only tables with naturally ordered data — BRIN can give you massive performance and space benefits compared to B-Tree. What is a BRIN Index? BRIN = Block Range Index A BRIN index doesn't index every row, unlike B-Tree....