1. OFFSET/LIMIT Pagination (a.k.a. “Skip-Limit”)

SELECT * FROM orders
ORDER BY created_at
OFFSET 10000
LIMIT 50;

Pros:

• Simple to implement

• Page numbers are intuitive for UIs

Cons:

• Slower as OFFSET increases — PostgreSQL must scan and discard the skipped rows

• Non-deterministic under concurrent updates (rows can shift across pages)

• Doesn’t scale beyond 100k rows

When to use:

• Small datasets (or <100 pages)

• Admin dashboards or static data

• Temporary/internal tooling

2. Keyset Pagination (a.k.a. Seek Method)

Description:

Use a cursor (like created_at, id) to fetch the next page.

-- First page
SELECT * FROM orders
ORDER BY created_at, id
LIMIT 50;

-- Next page (cursor = last row)
SELECT * FROM orders
WHERE (created_at, id) > ('2024-07-01 10:00:00', 'uuid-123')
ORDER BY created_at, id
LIMIT 50;

Pros:

• O(1) pagination — always uses index seek

• Safe under high concurrency

• Deterministic (no skipping/overlaps)

• Ideal for real-time apps

Cons:

• Can't jump to arbitrary page (e.g., page 10)

• Cursor management required

Index recommendation:

CREATE INDEX ON orders (created_at, id);

3. Hybrid Keyset + Offset (for Jumping to Specific Pages)

Description:

Use OFFSET for initial jump + keyset after that.

E.g., cache the (created_at, id) of the first row on page 10 → use that as keyset.

Pros:

• Allows page jumping

• Still uses index efficiently from that point onward

Cons:

• Adds complexity (requires cursor cache)

• Still slow on very high page numbers if not cached

Use case:

• UIs where both page numbers and infinite scroll are required

4. Cursor-based Pagination (Encoded Cursors)

This is commonly used in APIs (GraphQL, REST with cursor tokens).

Description:

Return a base64-encoded cursor like:

{
  "next_cursor": "created_at=2024-07-01T10:00:00&id=abc-123"
}

In the next query:

SELECT * FROM orders
WHERE (created_at, id) > (:created_at, :id)
ORDER BY created_at, id
LIMIT 50;

Pros:

• API-friendly, stateless pagination

• Easy to cache cursors

• Works with dynamic filters

How to Handle UPDATEs / Concurrent Changes

For Large Scale Dataset pagination

When to Use Each Method

The Hidden Checklist You Actually Need

1. Understand Your Workload First

Not a setting, but non-negotiable.
Ask yourself:

• OLTP or OLAP?

• Read-heavy or write-heavy?

• Bulk inserts? Lots of small transactions?

• Real-time queries or batch processing?

Every parameter depends on this.

2. shared_buffers – But Don’t Max It Out

Common advice: set shared_buffers = 25% of RAM
Better:
1. If your queries are heavy on joins and large scans: go higher (30–40%)
2. For many connections and small transactions: stay conservative (15–20%)
3. If using a connection pooler: you can afford a bit more here

Rule of Thumb: benchmark at 25%, then test up/down in 5% steps.

3. work_mem – The Secret Weapon for Big Joins

This controls sort/hash memory per operation, not per query.
Default is tiny. Too tiny.

Tune it dynamically:

SET work_mem = '256MB';  -- per session

But for config:

work_mem = 64MB   # OLAP
work_mem = 4-16MB # OLTP

Use EXPLAIN ANALYZE → look for “disk” in sort/hash ops. If you see that? Bump it.

4. maintenance_work_mem – Bulk Ops Love This

Used for VACUUM, CREATE INDEX, etc.
1. Default is too low (64MB).
2. Crank it up when running vacuums manually:

maintenance_work_mem = 1GB

If you're indexing 100M rows, go 2–4GB (if RAM allows).

5. effective_cache_size – Inform the Planner

Doesn't use memory, just tells the planner how much OS cache is available.
Set it to 50–75% of total RAM:

effective_cache_size = 24GB  # on a 32GB machine

Helps avoid bad nested loop plans on big tables.

6. Autovacuum Tuning – The Untold Bottleneck

Massive tables require aggressive VACUUM tuning, otherwise bloat kills you.

In postgresql.conf or via ALTER TABLE:

autovacuum_vacuum_threshold = 1000
autovacuum_vacuum_scale_factor = 0.01
autovacuum_analyze_scale_factor = 0.005
autovacuum_max_workers = 5
autovacuum_naptime = 10s
autovacuum_vacuum_cost_limit = 2000

For HOT update-heavy workloads, consider lower thresholds + more workers.

7. Parallelism Parameters – Scale Joins + Aggregates

Enable Postgres to use parallel query features.

max_parallel_workers = 8
max_parallel_workers_per_gather = 4
parallel_tuple_cost = 0.1
parallel_setup_cost = 1000

Lower parallel_tuple_cost and parallel_setup_cost to encourage parallelism.

8. random_page_cost & seq_page_cost – For SSD Optimization

If on SSD (you should be), reduce these to reflect reality.

random_page_cost = 1.1
seq_page_cost = 1.0

Default assumes spinning disks. SSDs have less cost difference between random vs sequential access.

9. Connection Limits – Don’t Overload

Too many active connections will kill performance.

Use a connection pooler like PgBouncer:

max_connections = 100  # keep it low

Let PgBouncer manage thousands of app connections.

10. Logging for Insightful Tuning

Turn on query logging to find bad queries:

log_min_duration_statement = 1000  # ms
log_checkpoints = on
log_autovacuum_min_duration = 0
log_temp_files = 0

Then mine the logs and use pg_stat_statements for real performance work.

Bonus: Table/Index-Level Tricks

• Use BRIN indexes for append-only, timestamped data

• Use partial indexes if only part of the data is queried often

• Consider UNLOGGED tables for transient data (faster inserts, no WAL)

• Use pg_repack to reclaim bloat without locking tables

• Implement Partitioning & Data Archival

• Smart Indexing & Monitoring index bloat regularly with pgstattuple

• Tablespaces, Compression & Storage

• Logical/Native Replication to scale out

• Aggressive Monitoring, Debugging & Benchmarking

TL;DR – The Advanced Tune Checklist

If you're managing 100M+ rows, this is not optional anymore. It’s engineering. And it works. Add your scaling experience in the comments below.

Your high-traffic app is humming along smoothly. Most data requests are being served instantly from Redis or other, blazing-fast cache layer. But then — a cache key expires, and within milliseconds, 10,000 clients hit your backend at once trying to fetch the same data from the database.

Boom — your production database is slammed, response times skyrocket, and the system becomes unresponsive.

You’ve just been trampled by a cache stampede.

What is a Cache Stampede?

A cache stampede (also called a cache miss storm) occurs when:

1. A popular cache key expires

2. Many clients attempt to read the same key

3. All of them get a cache miss

4. All of them bypass the cache simultaneously

5. They hit the backend/database at once

6. Overload happens — especially in high-concurrency environments

Even if your cache hit rate is 99%, a stampede on 1% of traffic can collapse the system.

Why Cache Stampedes Happen

Caches are typically built with a cache-aside pattern, where:

• You check the cache first

• If the key is missing, you recompute or fetch from the DB

• Then store it back into the cache

This works great for individual requests… but under heavy concurrency, when many requests miss the same key, all of them go through this pattern at the same time:

def get_data(key):
    data = redis.get(key)
    if not data:
        data = query_postgres()
        redis.set(key, data, ex=60)
    return data

Without protection, this causes 1000s of concurrent query_postgres() calls.

Real-world Impact

A cache stampede causes:

• Sudden spikes in DB traffic (often 10x to 100x)

• Connection pool exhaustions.

• Slow queries, timeouts, or even crashes

• Denial of service for downstream services

• Resource contention across your stack

When Does This Happen?

• After a cache TTL expires

• After a cache eviction (due to memory pressure)

• During cold starts or deployment rollouts

• When there’s only one shared cache key for a popular item (e.g., homepage data)

How to Mitigate Cache Stampedes

There are few ways to mitigate Cache Stampedes.

1. Request Coalescing / Single-flight Pattern

Let only one request rebuild the cache, while others wait for it to finish.

Concept:

• First request takes a lock and fetches the data from DB

• Others wait for that request to populate the cache

2. Add Jitter (Randomized Expiry)

Avoid simultaneous cache expiry by randomizing TTLs.

ttl = random.randint(50, 70)
redis.set(key, data, ex=ttl)

Prevents a "herd" of keys expiring together
Works best in high-concurrency apps with shared TTLs

3. Serve Stale Data While Rebuilding

Don’t block the user when the cache expires.

Instead:

• Return the stale value

• Trigger a background refresh

This is known as “stale-while-revalidate”.

Implementation Approach:

• Store TTL metadata separately

• If TTL is expired, serve stale data and refresh in background thread

if redis.ttl(key) <= 0:
    async_refresh(key)
return redis.get(key)

4. Proactive Cache Warming / Preload

Use background jobs to preload popular cache keys before they expire.

Example:

hot_keys = ['homepage', 'top_products', 'user_analytics']

for key in hot_keys:
    data = query_postgres()
    redis.set(key, data, ex=60)

Keeps your most important data hot
Ideal for dashboards, trending items, etc.

Use schedulers like cron, Celery beat, Sidekiq, etc.

5. Use Multi-Level Cache (L1 + L2)

Layered cache architecture:

• L1 cache: In-process or in-memory (e.g. Python LRUCache)

• L2 cache: Redis / Memcached

• L3: Database

Each layer reduces pressure on the next.

6. Batch Writes (if cache miss triggers DB writes)

If the cache miss causes multiple writes, use queues like Kafka or Redis Streams to batch and smooth load.

PostgreSQL Tips for Surviving Stampedes

If it still happens:

1. Use PgBouncer: Reduces connection overhead

2. Add read replicas: Distribute SELECT load

3. Materialized Views: Precompute expensive queries

4. Analyze slow queries: Use EXPLAIN ANALYZE + indexes

5. Use rate-limiting middleware: Prevent DoS

Conclusion

Cache stampedes are easy to miss in staging, but devastating in production. You don’t need a DDoS to bring down your system — just a single cache key expiring under high load.

The fix isn’t just “increase cache TTL” — it’s about smart architecture:

• Let only one request rebuild

• Serve stale when you can

• Add randomness to TTL

• Use background warming.

Why Semantic Search?

Traditional keyword search only matches exact terms.
Semantic search, on the other hand, understands context — for example, the phrases:

“How do I reset my password?”
and
“Forgot my login credentials.”

mean the same thing, even though they use completely different words.

That’s what Sentence-BERT (SBERT) enables — it transforms sentences into numerical embeddings that capture semantic meaning.
Once we have those embeddings, we can use vector similarity (like cosine similarity) to find text with similar meaning.

High Overview

Here’s the high-level flow of the system we built:

1. FastAPI serves REST endpoints for inserting and searching text.

2. Sentence-BERT generates a 384-dimensional vector for each text.

3. PostgreSQL with pgvector stores these embeddings and performs fast similarity queries using vector math.

Tech Stack

Model used:

all-MiniLM-L6-v2 — lightweight, accurate, and great for quick experimentation.

Here’s how the key pieces fit together.

Sentence-BERT Embeddings

Using Hugging Face’s sentence-transformers library, each text is transformed into a 384-dimensional embedding vector:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

text = "A forgotten attic filled with old memories"
embedding = model.encode(text)
print(len(embedding))  # 384

PostgreSQL + pgvector

To store and search these embeddings efficiently, we enable the pgvector extension.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE items (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    description TEXT NOT NULL,
    embedding VECTOR(384),
    created_at TIMESTAMP DEFAULT NOW()
);

We then create a vector index to speed up similarity queries:

CREATE INDEX items_embedding_idx
ON items
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

FastAPI Endpoints

/insert_item

Accepts multiple text items and inserts their embeddings into PostgreSQL.

@app.post('/insert_item')
def insert_items(request: ItemRequest):
    embeddings = [embedder.encode(i.description) for i in request.item_requests]
    # Store in DB as vector

/find_similar

Finds top-N most similar descriptions by comparing embeddings using pgvector’s cosine similarity operator <=>. There are other similarity operators as well for you to explore.

SELECT title, description, 1 - (embedding <=> %s::vector) AS similarity
FROM items
ORDER BY similarity DESC
LIMIT 5;

The result: a list of text items with semantic similarity scores.

Example in Action

Input Text

“The smell of aged paper and leather in a quiet bookstore.”

Output

{
  "similar_description": [
    {
      "title": "A vintage bookstore",
      "description_text": "The bookstore smelled of aged paper and leather...",
      "similarity": 0.86,
      "similarity_percent": 86.42
    },
    {
      "title": "A forgotten attic",
      "description_text": "The air in the attic hung heavy with the scent of forgotten things...",
      "similarity": 0.74,
      "similarity_percent": 74.10
    }
  ]
}

That’s semantic similarity in action. You can further improve/optimize it based on requirement and other pre-steps.

Alternatives to SBERT

Few other alternatives are

1. intfloat/e5-large-v2

2. nomic-ai/nomic-embed-text-v1

3. OpenAI Embeddings (text-embedding-3-large)

4. Cohere Embeddings (embed-multilingual-v3.0)

5. Sentence-T5 or Universal Sentence Encoder (USE)

Thoughts:

This little POC with proper architecture & implementation has worked wonder for one of use cases. With just a few hundred lines of Python and SQL, you can build a real semantic search engine — no external AI infrastructure required.

If you’re exploring NLP, information retrieval, or vector databases, this is one of the best starting points you can build on.

What is a BRIN Index?

BRIN = Block Range Index

A BRIN index doesn't index every row, unlike B-Tree. Instead, it stores summary data (min, max) about physical blocks of rows.

Think of BRIN as a metadata guide: it narrows down where to search, not what to return.

How BRIN Works Internally

When you create a BRIN index on a column, PostgreSQL organizes the index by summarizing values over physical block ranges in the table.

For each block range, the BRIN index stores summary metadata, including:

• Minimum value in the range

• Maximum value in the range

• And other summary data, depending on the BRIN operator class

During Index Creation

• PostgreSQL scans the table in fixed-size block ranges.

• For each range, it stores a tuple in the BRIN index with the min/max values (and possibly other stats).

During Query Execution

1. When a query has a search condition (e.g., WHERE column = 42):

2. PostgreSQL consults the BRIN index to check the min/max values for each range.

3. If the search value falls outside a block range’s min/max → the range is skipped.

4. If it falls inside the range → PostgreSQL performs a heap scan on that range (similar to a sequential scan but limited to that range).

BRIN does not point to individual rows—just ranges. It’s extremely space-efficient, but depends heavily on data correlation (e.g., time-ordered inserts).

BRIN vs. B-Tree: Feature Comparison

Benchmark: BRIN vs. B-Tree with Real Data

Dataset: 100 million records

CREATE TABLE sensor_data (
    id BIGSERIAL PRIMARY KEY,
    device_id INT,
    temperature NUMERIC,
    event_time TIMESTAMP
);

Populate with realistic data:

INSERT INTO sensor_data (device_id, temperature, event_time)
SELECT
  (random()*1000)::int,
  round(random()*50, 2),
  now() - (random() * interval '365 days')
FROM generate_series(1, 100000000);

Create Indexes

-- BRIN
CREATE INDEX brin_sensor_event_time ON sensor_data USING brin (event_time);

-- B-Tree
CREATE INDEX btree_sensor_event_time ON sensor_data USING btree (event_time);

Performance Test

Query:

SELECT * FROM sensor_data
WHERE event_time BETWEEN '2023-01-01' AND '2023-01-10';

BRIN is ~250x smaller, but slower for highly selective queries

When to Use BRIN (Best Use-Cases)

• Append-only or time-series data

• Large tables: 100M+ rows

• Columns with monotonic or naturally ordered data:

  • Timestamps

  • IDs

  • Dates

• Use with partitioned tables for cheap indexing

• Excellent for archival or rarely queried historical data

When Not to Use BRIN

• Highly randomized or non-sequential data

• High update/delete activity (BRIN won’t self-tune)

• You need fast, pinpoint lookups (use B-Tree instead)

Creating & Tuning BRIN

Basic Syntax

CREATE INDEX idx_brin_column
ON your_table USING brin(your_column);

Tuning pages_per_range

-- Lower = finer granularity (but larger index)
CREATE INDEX idx_brin_custom
ON your_table USING brin(your_column)
WITH (pages_per_range = 32);

Default = 128; use lower for sparse data, higher for dense data

Measuring BRIN Effectiveness

Check index size

SELECT pg_size_pretty(pg_relation_size('idx_brin_column'));

View BRIN summary info

SELECT * FROM pg_brin_bloom_summary;
-- Or use pg_visibility module for deeper inspection

REINDEX or VACUUM

If data distribution changed a lot:

REINDEX INDEX idx_brin_column;
VACUUM ANALYZE your_table;

Pairing BRIN with Other Indexes

Yes! Combine strategies:

• BRIN on event_time

• B-Tree on device_id

Allows PostgreSQL to combine filters effectively.

SELECT * FROM sensor_data
WHERE event_time > now() - interval '30 days'
AND device_id = 42;

Use multicolumn BRIN only if both columns are sequentially increasing.

Other BRIN Index Types

Available operator classes:

• brin_minmax (default)

• brin_bloom (Postgres 14+): bitset-based

• brin_inclusion (Postgres 15+): for ranges

• Custom extensions like brin_lov (local dictionary)

Pros and Cons

Pros

• Tiny disk footprint

• Fast to build

• Great for bulk time-based ingestion

• Easy maintenance

Cons

• Slower than B-Tree for pinpoint queries

• Sensitive to data distribution

• No enforcement of uniqueness

• Doesn't help much on UPDATE-heavy workloads

Final Thoughts

BRIN indexes are an absolute gem for large-scale, time-based, or append-only tables where:

• You care about index size and creation time

• Queries are range-based (e.g., logs, sensor data)

• You don’t need millisecond response times for every query

For more details refer the official doc at PostgreSQL: Documentation: 18: 65.5. BRIN Indexes

Key Highlights:

• Document-Level Locking: With the WiredTiger engine, MongoDB supports fine-grained locks at the document level.

• Lock Types: Internally uses Shared (S), Exclusive (X), Intent Shared (IS), and Intent Exclusive (IX) for coordination.

• Optimistic Locking: Uses a version field to prevent overwrites in high-read environments.

• Pessimistic Locking: Simulated via a lock field to prevent concurrent access during critical updates.

• Deadlock Prevention: Implement retry logic, consistent update order, and use $maxTimeMS to avoid transaction stalls.

• Monitoring Tools: Use serverStatus().locks and currentOp({ waitingForLock: true }) to track lock contention.

Use Cases Covered:

• Real-world optimistic & pessimistic lock patterns in MongoDB

• Deadlock detection and prevention strategies

• Lock optimization techniques and when to use which approach

More details are posted here at another post

If you’re preparing for data engineering, backend, or database-focused roles, this guide is for you.

Real SQL Questions I Was Asked in Interviews

Here are some real SQL questions I encountered across multiple interviews. These test your ability to work with complex queries and understand database structures:

• Recursive Hierarchy with Levels - Return employee ID, name, manager name, and depth in the org chart (Recursive CTE)

• Find Managers with 2 or more employees

• Departmental Top 3 Salaries

• Customers who bought all products

• Rolling sums - Calculate a 7-day rolling sum of sales for each product, ordered by date.

• Find continuous date ranges when there was sales activity every day without gaps.

The below are advanced topics came up in several interview rounds and often led to deep discussions.

1. PostgreSQL Indexing: Internals, Use Cases, and Performance

Had low level discussions on indexes along with their use cases.

• Covering Indexes: CREATE INDEX ON emp (deptid) INCLUDE (salary)

• Multicolumn Indexes: Consider column order (leading column matters!)

• Bloom Index: Used in specialized fuzzy match cases

• Index Usage: How planner choose Indexes, Analyze & Statistics effects on Index Usage.

• Index Maintenance: Need for vacuum &/ reindex, index Bloat, Fill factor & tuning.

• Other: Index Impact on Write Performance, OLTP & OLAP workloads, Finding unused index, Index usage metrics, Indexes and concurrent writes, etc.

2. Reading PostgreSQL Execution Plans

How to interpret & understand execution plans in turn understanding query performance. What are the things to look into the plans.

EXPLAIN (ANALYZE, BUFFERS, COSTS, VERBOSE)
SELECT * FROM employee WHERE salary > 100000;

Things to know:

• Seq Scan: No useful index found

• Index Scan: Ideal for high-selectivity queries

• Bitmap Heap Scan: Used for broader matches, combines index results

• Rows Removed by Filter: Wasted work — suggests filter isn’t pushed down

• Joins & Join Methods (Nested Join, Merge Join & Hash Join): Use case scenarios, Memory Impacts, how sorting effects the join, hash table spills etc.

• Other Topics: Parallel Query, CTEs & subplans, cost estimates, watch for loops > 1 in nested loops, monitor buffers read/hit to spot I/O inefficiencies, etc.

3. Partitioning

I was asked: “How would you partition a 5B-row logs table by region and time?”

Few of the topics discussed during interviews were,

• Partitioning Methods - Range Partition, List Partition, Hash Partition, Composite Partition

• Partitioning Work - Declarative partitioning vs table inheritance, Planner routes inserts/update, default partitions.

• Performance Implications - Partition Pruning, indexing on partitions vs global index (Postgres doesn’t support native global index)

• Impact on Query Plans & when pruning fails.

• Maintenance - Adding / Removing partitions, Detaching / Attaching partitions, Archiving, Table bloat, constraints behavior with partitioned tables etc.

4. Sharding PostgreSQL (Horizontal Scale)

Interviewer: “What happens when a single node isn’t enough?”

Sharding = horizontal partitioning → splitting data across multiple physical databases/servers, not just partitions in one DB.

• Each shard holds a subset of rows.

• Common shard keys: customer ID, tenant ID, geographic region, time.

• Goal: keep each shard small & fast to query independently.

Manual Sharding Approaches:

• Application-controlled sharding (based on user_id, region)

• Foreign Data Wrappers (postgres_fdw)

• Tools like Citus

Example Shard Strategy (Customer DB):

• Users A–M → shard_1

• Users N–Z → shard_2

• Metadata service maps user → shard

You must handle picking a good shard key, cross-shard joins, global transactions, Distributed transactions, Data rebalancing, Consistency & failover, replication lag etc.

5. Change Data Capture (CDC)

Change Data Capture means continuously capturing row-level data changes (INSERT, UPDATE, DELETE) from a source database and delivering them to downstream systems.

“How do you build a real-time pipeline to stream changes?”

✅ PostgreSQL CDC Techniques:

Logical Replication (Built-in)

• PostgreSQL 10+ supports logical replication.

• Publishes row-level changes as a stream.

• Use: CREATE PUBLICATION and CREATE SUBSCRIPTION.

• Works via WAL (Write-Ahead Log): changes are encoded as logical changes, not raw blocks.

• Good for replica clusters or feeding Kafka connectors.

Logical Decoding

• The foundation for logical replication.

• pgoutput is the default plugin.

• Or use plugins like wal2json or decoderbufs:

  • wal2json: output changes as JSON — easy for Kafka, Debezium.

  • decoderbufs: protobuf format.

• Tools read the WAL stream via a replication slot.

Triggers-Based CDC

• Implemented with AFTER INSERT/UPDATE/DELETE triggers.

• Simpler for small systems, but high overhead under heavy writes.

6. Other concepts:

Few other things which I think it’s important to go over are,

• When to use BRIN over B-tree indexes

• Index design tradeoffs in high-write OLTP systems

• Join types and when to apply each

• Timeseries partitioning and pruning validation

• Postgres internals: MVCC, WAL, autovacuum

• Locking, concurrency control, and deadlocks

• Designing a hybrid search (full-text + semantic)

• Data ingestion pipelines with Kafka and Postgres

• Handling SCDs, surrogate vs natural keys

• Real-time event guarantees: exactly-once vs at-least-once delivery

• What indexing strategies do you use for analytical vs transactional workloads?

If you’re preparing for roles that involve SQL, data engineering, or backend systems, mastering both query skills and PostgreSQL internals can really set you apart. The questions span from hands-on SQL to system-level architecture — and each was an opportunity to demonstrate practical depth.

Let me know in the comments if you'd like to add more questions or topics to this list!

Overview of the context Package

The context package helps you control the lifecycle of operations and control, from managing HTTP requests to orchestrating database calls:

• Request cancellation: Propagating cancellation signals to stop long-running tasks.

• Deadlines and timeouts: Automatically cancel operations if they exceed a given timeout.

• Passing request-scoped values: Sharing request-scoped data (e.g., user information, tracing IDs) across different program layers.

Basic Types in context

1. context.Context:

   • This is the main type used to carry deadlines, cancellations, and values across API boundaries. It is immutable and provides methods to access its state.

2. Context Creation Functions:

   • context.Background(): Returns an empty context. This is often used as a root context in long-running background tasks.

   • context.TODO(): Similar to Background(), but used when you’re unsure what context to use. It signals "I haven’t figured out what context to use here yet."

3. Context with Deadline, Timeout, or Cancellation:

   • context.WithCancel(parent): Creates a context that can be explicitly canceled.

   • context.WithDeadline(parent, deadline): Creates a context that will be canceled automatically at a specific time (deadline).

   • context.WithTimeout(parent, timeout): Creates a context that will be canceled after a specified timeout.

4. Value Context:

   • context.WithValue(parent, key, value): Attaches key-value pairs to a context. Useful for passing request-scoped data.

Why Use the context Package?

1. Cancellation Propagation

• Problem: In a concurrent system, if one part of the operation fails or is no longer needed, you want to propagate the cancellation to other goroutines to avoid wasted resources.

• Solution: context.WithCancel() allows you to cancel all child goroutines when the parent context is canceled.

2. Timeouts and Deadlines

• Problem: Long-running operations can consume system resources if they hang or fail to return within a reasonable time.

• Solution: context.WithTimeout() and context.WithDeadline() help ensure that operations do not run indefinitely by enforcing timeouts.

3. Passing Request-Scoped Values

• Problem: When processing a request, you may need to pass values (e.g., user info, tracing IDs, or other metadata) across function calls without modifying function signatures excessively.

• Solution: context.WithValue() allows you to pass values between layers of the program without cluttering function signatures.

Common Use Cases

1. HTTP Servers (Request Cancellation and Timeouts)

• In web servers, context is commonly used to manage request timeouts and propagate cancellations.

• For example, in an HTTP server, if a client disconnects or a request times out, the server should stop processing the request immediately. The context package allows the cancellation signal to propagate to all goroutines working on that request.

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    select {
    case <-time.After(5 * time.Second): // Simulating a long-running operation
        fmt.Fprintln(w, "Finished processing")
    case <-ctx.Done(): // If the request is cancelled or times out
        fmt.Fprintln(w, "Request cancelled")
    }
}

2. Database Operations (Timeouts and Cancellations)

• In database operations, contexts are used to set timeouts and ensure that if a query takes too long, it is canceled.

func queryDB(ctx context.Context, db *sql.DB) error {
    queryCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()

    rows, err := db.QueryContext(queryCtx, "SELECT * FROM users")
    if err != nil {
        return err // This will return if the query exceeds 2 seconds
    }
    defer rows.Close()

    // Process rows...
    return nil
}

3. Goroutines and Background Tasks (Graceful Shutdown)

• In systems with multiple goroutines, you often need a mechanism to stop all related goroutines when a parent operation is canceled (e.g., during a graceful shutdown).

func process(ctx context.Context) {
    for {
        select {
        case <-ctx.Done():
            fmt.Println("Context cancelled, shutting down")
            return
        default:
            // Do some work...
        }
    }
}

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    go process(ctx)
    time.Sleep(2 * time.Second)
    cancel() // Gracefully cancel the process
}

Advanced Use Cases

1. Distributed Tracing

• Context is heavily used in distributed systems to propagate tracing information (e.g., trace IDs) across services. Middleware can extract tracing data from incoming requests, store it in the context, and propagate it across calls.

func traceMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        traceID := r.Header.Get("X-Trace-ID")
        ctx := context.WithValue(r.Context(), "traceID", traceID)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

func handler(w http.ResponseWriter, r *http.Request) {
    traceID := r.Context().Value("traceID")
    fmt.Fprintf(w, "Trace ID: %v", traceID)
}

2. Propagating Deadline Across Microservices

• In microservices, a client making an external request can propagate the deadline using context.Context, so the downstream service knows when the request times out and can handle it accordingly.

func makeExternalCall(ctx context.Context) error {
    req, err := http.NewRequestWithContext(ctx, "GET", "http://example.com", nil)
    if err != nil {
        return err
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    return nil
}

3. Graceful Shutdown in Servers

• Using context with signal.NotifyContext allows for graceful shutdowns in servers. This ensures that the server stops accepting new requests while allowing ongoing requests to finish.

func main() {
    srv := &http.Server{Addr: ":8080", Handler: http.DefaultServeMux}

    // Listen for system interrupt signals (e.g., Ctrl+C)
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            fmt.Printf("Server failed: %v\n", err)
        }
    }()

    // Wait for interrupt signal
    <-ctx.Done()

    fmt.Println("Shutting down gracefully...")
    ctxShutDown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctxShutDown); err != nil {
        fmt.Printf("Server Shutdown Failed: %v\n", err)
    }
}

Industry Standards and Best Practices

1. Always Pass Context as the First Argument:

   • Functions that take a context.Context should accept it as the first argument. This is the convention in the Go standard library and ensures consistency across APIs.

    func DoSomething(ctx context.Context, arg int) {
        // Implementation...
    }

2. Avoid Storing Contexts in Structs:

   • Contexts are meant to be short-lived and should not be stored in global or long-lived variables. They are request-scoped or operation-scoped and should be passed explicitly through function calls.

3. Respect ctx.Done():

   • Always check ctx.Done() to ensure that long-running operations can gracefully terminate if the context is canceled or if a deadline is exceeded.

4. Use context.WithValue() Sparingly:

   • While context.WithValue() is useful for passing request-scoped values, avoid overusing it. If you need to pass many values, it's better to create a dedicated struct to hold those values and pass that struct around instead of storing them in the context.

Conclusion

The context package is a powerful tool for controlling the lifecycle of concurrent operations, managing deadlines, propagating cancellations, and passing request-scoped values across layers of a program. It is essential for writing clean, efficient, and manageable concurrent code, especially in large-scale systems or services. By following industry best practices, you can use the context package to build robust, production-grade applications and services.

Introduction & Definition

Table inheritance is a feature in PostgreSQL that allows you to create a hierarchy of tables based on a parent-child relationship. The child tables inherit the structure, constraints, and attributes of the parent table, while also having its definitions and additional attributes. This approach provides a convenient way to manage related data and simplify your database schema. To understand, let's see a simple and widely used scenario.

Suppose we have an application like HRMS or something wherein we have to manage employee data, and the organization has different types/categories of employees like full-time, part-time, contracts, special consultants etc. Each of these employees has some common attributes like first name, last name, email, joining date, date of birth etc but also specific attributes unique to those employee types. Below DDL script will help understand more,
Create the parent table called "employees" with common attributes:

-- Create the parent table called "employees" with common attributes:
CREATE TABLE employees (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(100),
    hire_date DATE
);
-- Create child tables for each type of employee, inheriting from the "employees" table:

CREATE TABLE full_time_employees () INHERITS (employees);
CREATE TABLE part_time_employees () INHERITS (employees);
CREATE TABLE contractors () INHERITS (employees);

-- Add specific attributes to each child table:

-- Adding columns to the full-time employees table
ALTER TABLE full_time_employees ADD COLUMN salary NUMERIC(10, 2);
ALTER TABLE full_time_employees ADD COLUMN vacation_days INTEGER;

-- Adding columns to the part-time employees table
ALTER TABLE part_time_employees ADD COLUMN hourly_rate NUMERIC(10, 2);
ALTER TABLE part_time_employees ADD COLUMN hours_worked INTEGER;

-- Adding columns to the contractors table
ALTER TABLE contractors ADD COLUMN contract_rate NUMERIC(10, 2);
ALTER TABLE contractors ADD COLUMN contract_duration INTEGER;

In this example, the parent table "employees" contains common attributes shared by all employees. The child tables, such as "full_time_employees", "part_time_employees", and "contractors", inherit these common attributes and allow for the addition of specific attributes related to each employee type.

With the use of table inheritance, we have done,

• Maintained a centralized employee table for common attributes and shared functionality.

• For unique attributes of each employee type, we created separate child tables for types with those attributes, ensuring data integrity and clarity.

• Retrieval and Filtering of data while performing queries specific to each employee type using the child tables.

This approach provides flexibility in managing different types of employees while maintaining a consistent structure and enabling specific attributes for each employee type.

Similarly, it can be implemented for various product categories in an e-commerce database. Each product category has its unique attributes other than the common attributes. The parent table will be a product table having common attributes like product name, price, etc and child tables having attributes specific to each product type. This design can also be used in designing content management systems, where types of content are different.

Pros:

• Table inheritance allows you to organize your data with the parent table containing common attributes shared by all child tables, while each child table can have its specific attributes. This logical organization makes it easier to manage and query data.

• By centralizing common attributes in the parent table, you avoid duplicating columns across multiple tables.

• Constraints defined on the parent table are automatically enforced on all child tables. This ensures data integrity and consistency throughout the inheritance hierarchy.

• With table inheritance, you can perform queries on specific child tables to retrieve data relevant to a particular entity type. This allows for efficient filtering and retrieval of data based on specific attributes.

Cons:

• The query execution planner will have to consider the structure and constraints defined for both parent and child tables, in turn adding complexity to query planning and can result in slightly longer planning time.

• PostgreSQL uses indexes defined on the specific table, if any, for querying that table. This means that you may need to create separate indexes for each child table to optimize query performance.

• Table inheritance is often used for dividing or partitioning large tables into more manageable chunks, which is also known as data segmentation. But it introduces additional complexity in making complex execution plans and may involve querying multiple child tables, causing a performance drop.

• When performing maintenance operations like VACUUM or ANALYZE on a parent table, PostgreSQL will also process the child tables. This can increase the time required for these operations, especially if the inherited tables contain a significant amount of data.

Summary:

Table inheritance in PostgreSQL offers a robust method for organizing related data, enhancing code reusability, and preserving data integrity, which can streamline your database schema, minimize redundancy, and boost query adaptability. When devising your database, assess the entities, their shared attributes, and unique traits to establish if table inheritance is an appropriate strategy. Keep in mind the importance of meticulously designing your database schema and taking into account the particular requirements and access patterns of your application when employing table inheritance.

PostgreSQL stops the execution of the block and the related transaction when a block contains an error. A block is a collection of statements that are contained within a BEGIN and END block structure in PL/pgSQL. In PostgreSQL, blocks are used to define unique procedures, triggers, and functions. The beginning and end of the block are indicated by the terms BEGIN and END, respectively.

Syntax

BEGIN
    -- Code goes here
EXCEPTION
    WHEN exception_type THEN
        -- Exception handling code goes here
END;

Besides Begin and End block, the EXCEPTION keyword indicates the start of the exception handling section, which is executed if an exception is thrown, inside which the WHEN clause specifies the type of exception that the exception handler will handle. The THEN keyword indicates the start of the code block that handles the exception.

PostgreSQL has a wide range of built-in exception types such as SQLSTATE, SQLERRM, NO_DATA_FOUND, and TOO_MANY_ROWS. Moreover, users can also create custom exceptions using the RAISE statement. For a complete list of condition names on the PostgreSQL website. To illustrate how PostgreSQL handles exceptions, let's take the basic example of a divide-by-zero exception. The PL/pgSQL block below demonstrates how to handle this exception:

do
$$
declare 
    result int;
begin
    SELECT 1/0 INTO result;

    -- exception example based on SQL ERROR CODE
    exception
        WHEN division_by_zero THEN
            result := NULL;
end;
$$
language plpgsql;

In the above block, if the SELECT statement attempts to divide by zero, a division_by_zero exception is raised. The exception handling code, specified in the WHEN clause, sets the result variable to NULL in this case.

When an error occurs within the BEGIN...EXCEPTION block in PL/pgSQL, the execution is stopped and the control is transferred to the exception list. Then, PL/pgSQL scans the exception list to find the first match for the error that occurred. If a match is found, the statements inside the corresponding EXCEPTION block execute, and the control passes to the statement after the END keyword. If no match is found, the error propagates outwards and can be caught by the EXCEPTION clause of the enclosing block. In case there is no enclosing block with the EXCEPTION clause, PL/pgSQL aborts processing.

Example Syntax Code block for Multiple Exceptions:

do
$$
declare
    rec record;
    emp_name varchar = 'abc';
begin
    select 
        <column_names>
    into strict rec
    from <table_name>
    where employee_name = emp_name; -- example where clause.

    -- exception example based on SQL ERROR CODE
    exception
        when sqlstate 'P0002' then
            raise exception 'employee with name % not found', emp_name;
        when sqlstate 'P0003' then
            raise exception 'employee with name % is already present', emp_name;

    -- exception example based on Expection Condition
    exception
        when too_many_rows then
            raise exception 'Search query returns more than one rows';
end;
$$
language plpgsql;

This exception-handling mechanism is crucial for handling errors gracefully and preventing application crashes or data corruption. It allows developers to provide better user experiences by responding appropriately to errors, rather than simply halting execution.

Below is the problem statement, I got on internet for Traveling Salesman Problem, try reading through the inputs and outputs required. And I also posted solutions I made in both T-SQL and PostgreSQL variants.

First, is the City table containing two columns, Id and Name. This time, we're traveling between cities in Germany and we've been given a second table called Road that has the columns CityFrom, CityTo, and Time which contain average trip durations on a given route from one city to another. Below is create statement and data to populate rows.

create schema trip
create table trip.city(
    id int identity(1,1), -- in postgresl use serial datatype instead
    name varchar(100)
);

insert into trip.city(name) values ('Berlin'), ('Hamburg'), ('Muinch'), ('Amsterdam');
select * from trip.city;

create table trip.road(
    city_from int,
    city_to int,
    time int
);
insert into trip.road(city_from, city_to, time) values 
(1, 2, 180), (1, 3, 365), (1, 4, 390), (2, 3, 490), (2, 4, 270), (3, 2, 420), (3, 4, 370), (4, 2, 245), (4, 3, 385);
select *  from trip.road;

The trip path should cover all cities, starting from Berlin. The main task is to return travel paths starting from Berlin and covering all cities in descending order by the total time taken.

The output should contain the following columns

1. path – the city names, separated by N' -> ',

2. last_city_id – the ID of the last city visited,

3. total_time – the total time spent driving,

4. places_count – the number of places visited; it should equal 4.

The below solution is T-SQL(SQL Server) based variant:

;with travel (path, last_city_id, total_time, places_count )
as
(
    select
        cast(name as nvarchar(max)),
        id,
        0,
        1
    from trip.city c
    where c.name = 'Berlin' -- since starting from berlin, then we initially return the anchor point

    union all 

    select
        t.path + N' -> ' + c.name, -- concate the city name with existing travel path name 
        c.id, 
        t.total_time + r.time,
        t.places_count + 1
    from travel t
    join trip.road r
        on t.last_city_id= r.city_from
    join trip.city c
        on c.id = r.city_to
    where charindex(c.name, t.path) = 0 -- where part was to prevent revisiting the same city twice in a path

)

select * from travel where places_count = 4 order by total_time desc;

And below is PostgreSQL varient

WITH RECURSIVE travel (path, last_city_id, total_time, places_count) AS (
  SELECT
    CAST(name AS text),
    id,
    0,
    1
  FROM trip.city c
  WHERE c.name = 'Berlin'

  UNION ALL

  SELECT
    t.path || ' -> ' || c.name,
    c.id,
    t.total_time + r.time,
    t.places_count + 1
  FROM travel t
  JOIN trip.road r
    ON t.last_city_id = r.city_from
  JOIN trip.city c
    ON c.id = r.city_to
  WHERE position(c.name IN t.path) = 0
)

SELECT *
FROM travel
WHERE places_count = 4
ORDER BY total_time DESC;

In the codes above, we used common table expression (CTE) to recursively build up a list of travel paths between cities. The travel CTE contains four columns: path (the travel path so far), last_city_id (the ID of the last city visited), total_time (the total travel time so far), and places_count (the number of places visited so far).

The first part of the CTE returns to the starting point in Berlin. Then, the union all clause is used to recursively build up the travel paths. In the second part of the CTE, you're joining the travel CTE with the road and city tables to find the next city to visit. The charindex function (in T-SQL) or position function (in PostgreSQL) is used to make sure you don't revisit any cities on the same path. You can also use the substring function in PostgreSQL as an alternative.

Finally, the SELECT statement is used to return all travel paths that visit all four cities and are ordered in descending order by total travel time.