Mohammad Abu Mattar | Code Snippets

Bash Script Locking: Prevent Concurrent Runs with a PID File

Mohammad Abu Mattar — Tue, 18 Aug 2026 13:48:21 GMT

Quick Tip

Wrap any script that must not run twice at once in a PID-file lock, and a second copy simply exits instead of corrupting your data.

The Problem

Cron does not care whether your last run finished. If a job is scheduled every five minutes but one run takes seven, cron cheerfully starts a second copy while the first is still going. Now two processes are reading the same input, writing the same output file, and racing each other to a half-written mess.

The same thing happens with a webhook that fires twice, a retry that overlaps the original, or an impatient human running the script by hand while the scheduled run is live.

Real-world impact

Overlapping runs are the kind of bug that only shows up under load, which is exactly when you can least afford it. You get truncated files, doubled database rows, and log output interleaved from two processes so you cannot even tell what happened. The flow below shows what a single guarded run should do instead.

The Solution

The Fix

Before doing any work, the script writes its own process ID into a lock file. If that file already exists, it checks whether the process it names is still alive. If it is, this run backs off and exits. If the process is gone, the lock is stale (a previous run crashed) so it clears it and carries on. A trap removes the lock file on the way out, whether the script finished cleanly or died.

TL;DR

One PID file per script decides who gets to run.
kill -0 $pid tells you if the previous owner is still alive without actually signalling it.
A trap ... EXIT guarantees the lock is released even on error.

When two runs overlap, the second one loses cleanly:

Script Implementation

Setup

Start strict, and decide where the lock lives. Naming it after the script keeps different jobs from stepping on each other's locks.

#!/usr/bin/env bash
set -euo pipefail

# One lock per script name, in a writable runtime dir.
LOCK_FILE="${TMPDIR:-/tmp}/$(basename "$0").pid"

Acquiring the lock

This is the core. If a lock file exists and names a live process, exit. If it exists but the process is gone, treat it as stale and reclaim it.

acquire_lock() {
  if [[ -e "$LOCK_FILE" ]]; then
    local old_pid
    old_pid=$(cat "$LOCK_FILE" 2>/dev/null || echo "")

    # kill -0 checks the process exists without sending a real signal.
    if [[ -n "$old_pid" ]] && kill -0 "$old_pid" 2>/dev/null; then
      echo "Already running as PID $old_pid, exiting." >&2
      exit 1
    fi

    # The owner is gone: stale lock from a crashed run. Reclaim it.
    echo "Clearing stale lock from PID ${old_pid:-unknown}." >&2
  fi

  echo $$ > "$LOCK_FILE"
}

Releasing on exit

Register the cleanup once, right after acquiring, so it runs on normal exit, on error (thanks to set -e), and on Ctrl-C.

release_lock() {
  rm -f "$LOCK_FILE"
}

acquire_lock
trap release_lock EXIT

Entry point

Everything after the trap is your real work. It runs knowing no other copy is active.

main() {
  echo "Running with lock held (PID $$)..."
  # ... the actual job goes here ...
  sleep 5
  echo "Done."
}

main "$@"

Usage and Benefits

Why This Helps

The whole guard is about fifteen lines and needs nothing outside of Bash itself. Drop it at the top of any script and overlapping runs stop being a problem: the first run works, the rest exit immediately, and a crash never leaves a lock that blocks every future run.

Running it

Prove it to yourself by starting one in the background and immediately launching a second:

./with_lock.sh &   # first run grabs the lock
./with_lock.sh     # second run prints "Already running..." and exits 1

The first run holds the lock for its full duration; the second sees a live PID and steps aside. Kill the first mid-run and the next invocation clears the stale lock instead of getting stuck.

Community Discussion

Your Turn

How do you keep your cron jobs from overlapping? PID files, lock directories, or something your scheduler does for you? I would like to hear what has held up in production.

Alternative approaches

If you are on Linux, flock is worth a look. It locks a file descriptor at the kernel level, so there is no stale-PID logic to write yourself:

exec 9>"${TMPDIR:-/tmp}/$(basename "$0").lock"
flock -n 9 || { echo "Already running, exiting." >&2; exit 1; }
# lock is released automatically when fd 9 closes on exit

The PID-file version is more portable (it works the same on macOS and older shells) and it records who holds the lock, which is handy for debugging. flock is simpler and race-free where you have it. Pick whichever matches where your scripts run.

AWS DynamoDB CRUD Operations in Node.js with the AWS SDK v3

Mohammad Abu Mattar — Wed, 05 Aug 2026 15:18:55 GMT

Quick Tip

Wrap the low-level DynamoDB client in a DynamoDBDocumentClient so you pass plain JavaScript objects in and get plain objects back, then guard your writes with ConditionExpression and read by key with QueryCommand.

The Problem

The Problem

The low-level DynamoDB API doesn't speak JavaScript. Every value has to be wrapped in a typed attribute map, so a simple { id: "42", count: 3 } turns into { id: { S: "42" }, count: { N: "3" } } on the way in, and you have to unwrap all of it on the way out. Write that marshalling by hand across a codebase and it becomes a steady drip of ValidationException errors and off-by-one type bugs.

Why raw attribute maps hurt

The attribute-map format leaks into every call site. You can't just hand DynamoDB the object your app already has, you first translate it, then translate the response back, and you repeat that boilerplate for put, get, update, and query. It's easy to send a number as a string or forget a nested map, and DynamoDB rejects the whole request rather than coercing anything for you.

The real-world impact

That friction shows up as slow feature work and brittle data access. New team members trip over the { S: ... } wrappers, updates accidentally overwrite fields they never meant to touch, and someone reaches for Scan because it "just works," quietly reading the entire table on every call. The result is code that's harder to review and a bill that grows faster than the data.

The Solution

The Fix

Use the AWS SDK v3 DynamoDBDocumentClient. It wraps the base DynamoDBClient and handles marshalling in both directions, so your CRUD code deals in plain objects. Pair it with a single table, address items by their partition and sort keys, use UpdateCommand to change only the fields you name, and add a ConditionExpression when a write must not clobber existing data.

TL;DR

Wrap DynamoDBClient in DynamoDBDocumentClient so you never touch raw attribute maps.
Use PutCommand, GetCommand, UpdateCommand, and DeleteCommand for CRUD, with ConditionExpression to keep writes safe.
Read with QueryCommand on the partition key instead of scanning the whole table.

Script Implementation

Setup and the DocumentClient

Create the base client once, wrap it in a DynamoDBDocumentClient, and turn on the marshalling options that smooth over DynamoDB's quirks. Reuse this single instance across your app so connections get pooled.

// dynamo.ts - CRUD helpers for a single DynamoDB table with the AWS SDK v3.
import {DynamoDBClient} from '@aws-sdk/client-dynamodb';
import {
  DynamoDBDocumentClient,
  PutCommand,
  GetCommand,
  UpdateCommand,
  DeleteCommand,
  QueryCommand,
} from '@aws-sdk/lib-dynamodb';

const TABLE_NAME = process.env.TABLE_NAME ?? 'AppTable';

// One low-level client for the process; region comes from the environment.
const baseClient = new DynamoDBClient({});

// The DocumentClient marshals plain JS objects to/from attribute maps.
export const ddb = DynamoDBDocumentClient.from(baseClient, {
  marshallOptions: {
    // Drop undefined fields instead of erroring on them.
    removeUndefinedValues: true,
    // Store empty strings and sets as-is rather than converting to null.
    convertEmptyValues: false,
  },
});

// A tiny item shape for the examples: single-table keys plus data.
export interface User {
  pk: string; // partition key, e.g. "USER#42"
  sk: string; // sort key, e.g. "PROFILE"
  name: string;
  email: string;
  loginCount?: number;
}

Create and read: put and get

PutCommand writes a whole item, overwriting any existing item with the same key. GetCommand fetches one item by its full primary key and returns a plain object, or undefined when nothing matches.

// Create or overwrite an item. Pass the object as-is, no attribute maps.
export async function putUser(user: User): Promise<void> {
  await ddb.send(
    new PutCommand({
      TableName: TABLE_NAME,
      Item: user,
    }),
  );
}

// Fetch a single item by its partition + sort key.
export async function getUser(
  pk: string,
  sk: string,
): Promise<User | undefined> {
  const {Item} = await ddb.send(
    new GetCommand({
      TableName: TABLE_NAME,
      Key: {pk, sk},
    }),
  );
  // Item is already a plain object, cast it to the known shape.
  return Item as User | undefined;
}

Update and delete safely

UpdateCommand changes only the fields you name in the UpdateExpression, so you never overwrite the rest of the item. The ConditionExpression makes the update fail loudly if the item doesn't already exist, which stops accidental "upserts." DeleteCommand removes an item by key.

// Update named fields only, and require that the item already exists.
export async function bumpLoginCount(pk: string, sk: string): Promise<User> {
  const {Attributes} = await ddb.send(
    new UpdateCommand({
      TableName: TABLE_NAME,
      Key: {pk, sk},
      // ADD creates loginCount at 1 if missing, otherwise increments it.
      UpdateExpression: 'ADD loginCount :one',
      ConditionExpression: 'attribute_exists(pk)',
      ExpressionAttributeValues: {':one': 1},
      ReturnValues: 'ALL_NEW',
    }),
  );
  return Attributes as User;
}

// Delete an item by key. Idempotent: deleting a missing key is a no-op.
export async function deleteUser(pk: string, sk: string): Promise<void> {
  await ddb.send(
    new DeleteCommand({
      TableName: TABLE_NAME,
      Key: {pk, sk},
    }),
  );
}

Query by key instead of scanning

QueryCommand reads every item that shares a partition key, optionally narrowed by the sort key. This is the read you want almost every time, it touches only the matching items instead of the whole table the way Scan does.

// Fetch every item under one partition key, e.g. all records for a user.
export async function queryByUser(pk: string): Promise<User[]> {
  const items: User[] = [];
  let ExclusiveStartKey: Record<string, unknown> | undefined;

  // Loop to follow pagination until DynamoDB stops returning a cursor.
  do {
    const page = await ddb.send(
      new QueryCommand({
        TableName: TABLE_NAME,
        KeyConditionExpression: 'pk = :pk',
        ExpressionAttributeValues: {':pk': pk},
        ExclusiveStartKey,
      }),
    );
    items.push(...((page.Items ?? []) as User[]));
    ExclusiveStartKey = page.LastEvaluatedKey;
  } while (ExclusiveStartKey);

  return items;
}

Entry point with error handling

Tie the helpers together in a small demo and catch the one error you actually expect: a failed ConditionExpression. DynamoDB reports it as a ConditionalCheckFailedException, which you handle rather than crash on.

import {ConditionalCheckFailedException} from '@aws-sdk/client-dynamodb';

async function main(): Promise<void> {
  const key = {pk: 'USER#42', sk: 'PROFILE'};

  await putUser({...key, name: 'Ada', email: 'ada@example.com'});
  console.log('created:', await getUser(key.pk, key.sk));

  try {
    const updated = await bumpLoginCount(key.pk, key.sk);
    console.log('login count is now', updated.loginCount);
  } catch (err) {
    if (err instanceof ConditionalCheckFailedException) {
      console.error('update skipped: that user does not exist yet');
    } else {
      throw err; // Anything else is unexpected, let it surface.
    }
  }

  console.log('all records:', await queryByUser(key.pk));
  await deleteUser(key.pk, key.sk);
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Usage and Benefits

Why This Helps

The DocumentClient deletes an entire category of bugs by letting you work in plain objects, and the CRUD helpers give every call site the same safe pattern: named updates, conditional writes, and key-based reads. You stop hand-writing attribute maps, stop accidentally overwriting fields, and stop reaching for Scan when a Query is right there.

Real invocations

# Install the SDK v3 packages.
npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

# Set the table and region, then run the compiled script.
export AWS_REGION=us-east-1
export TABLE_NAME=AppTable
node dynamo.js

# Or run the TypeScript directly during development.
npx tsx dynamo.ts

Tuning for single-table design

The pk/sk pair is the heart of single-table design: you overload one table with many item types by encoding the type into the key. A user profile might use pk = "USER#42", sk = "PROFILE", while their orders use the same pk with sk = "ORDER#1001". One QueryCommand on USER#42 then pulls the profile and every order in a single call.

// Same partition key, different sort keys, one query returns them all.
await putUser({
  pk: 'USER#42',
  sk: 'PROFILE',
  name: 'Ada',
  email: 'ada@example.com',
});
await putUser({pk: 'USER#42', sk: 'ORDER#1001', name: 'order', email: '-'});

// begins_with narrows a query to just the orders under this user.
const orders = await ddb.send(
  new QueryCommand({
    TableName: TABLE_NAME,
    KeyConditionExpression: 'pk = :pk AND begins_with(sk, :prefix)',
    ExpressionAttributeValues: {':pk': 'USER#42', ':prefix': 'ORDER#'},
  }),
);

Comparison

How the SDK v3 DocumentClient stacks up against the other common ways to talk to DynamoDB from Node.js.

Approach	Marshalling	API style	Maintained	Best for
SDK v3 `DynamoDBDocumentClient`	Automatic	Command objects	Yes	New Node.js apps on the SDK v3
SDK v3 low-level `DynamoDBClient`	Manual maps	Command objects	Yes	Fine-grained control, edge cases
SDK v2 `DynamoDB.DocumentClient`	Automatic	Method calls	Maintenance only	Legacy code still on v2
An ORM/ODM like Dynamoose	Automatic + schema	Model methods	Yes	Teams wanting model abstractions

The low-level client is worth dropping to only when you need something the DocumentClient doesn't expose. For everyday CRUD, the document client's automatic marshalling is the reason it exists.

Frequently Asked Questions

<Accordion client:load title="What's the difference between DynamoDBClient and DynamoDBDocumentClient?"

DynamoDBClient is the low-level client: it speaks DynamoDB's native format, so every value is a typed attribute map like { S: "text" } or { N: "3" }. DynamoDBDocumentClient wraps it and adds marshalling in both directions, so you pass and receive plain JavaScript objects. You still create the base client, then build the document client from it with DynamoDBDocumentClient.from(baseClient). Use the document client for CRUD and drop to the base client only for the rare feature it doesn't cover.

<Accordion client:load title="Why should I prefer Query over Scan?"

Query reads only the items that share a partition key, using the index directly, so it stays fast and cheap as the table grows. Scan reads every item in the table and then filters, which means its cost and latency scale with the whole dataset, not with the rows you want. Design your keys so the reads you need are Query calls. Keep Scan for genuine full-table jobs like exports or backfills, and even then paginate it.

<Accordion client:load title="How do I update a single field without overwriting the whole item?"

Use UpdateCommand with an UpdateExpression that names only the fields you're changing, such as SET #n = :name or ADD loginCount :one. That leaves every other attribute on the item untouched. PutCommand, by contrast, replaces the entire item, so any field you don't include is dropped. When a name collides with a DynamoDB reserved word, alias it through ExpressionAttributeNames (the #n above) so the expression still parses.

<Accordion client:load title="What does ConditionExpression protect against?"

A ConditionExpression makes a write happen only if the condition holds, and fail otherwise. attribute_exists(pk) blocks an update from silently creating a new item, attribute_not_exists(pk) blocks a put from overwriting an existing one, and a version check like version = :expected gives you optimistic locking. When the condition fails, DynamoDB throws ConditionalCheckFailedException, which you catch and handle instead of letting a bad write through.

<Accordion client:load title="How do I handle results that span multiple pages?"

DynamoDB returns at most 1 MB of data per Query or Scan. When there's more, the response includes a LastEvaluatedKey cursor. Pass it back as ExclusiveStartKey on the next call and loop until the response stops returning one, which is exactly what queryByUser does. Skip the loop and you silently read only the first page, a bug that hides until your data crosses the 1 MB boundary in production.

<Accordion client:load title="Do I still need the AWS SDK v2 DocumentClient?"

Only for code that already runs on it. The SDK v2 is in maintenance mode, so new work should use v3, which is modular (you install just @aws-sdk/client-dynamodb and @aws-sdk/lib-dynamodb), tree-shakeable, and actively developed. The v3 DynamoDBDocumentClient gives you the same automatic marshalling the v2 DocumentClient did, just with the command-object API. Migrate when you can, but there's no need to rush a rewrite that works.

References

Python Async HTTP Requests with aiohttp: Fetch Multiple URLs Concurrently

Mohammad Abu Mattar — Sun, 26 Jul 2026 14:40:06 GMT

Quick Tip

Reuse one aiohttp session, fan your requests out with asyncio.gather, and cap them with a semaphore to fetch hundreds of URLs in the time one loop would take.

The Problem

The Problem

A script that fetches a list of URLs with requests in a plain for loop spends almost all of its time doing nothing. Each call opens a connection, sends the request, and then blocks until the response comes back before the next one even starts. The network is slow and your CPU is fast, so the program just sits and waits, one round trip at a time.

Why sequential requests waste time

HTTP is I/O bound. The bottleneck isn't your code, it's the latency of each request. If a single call takes 200ms and you have 200 URLs, a sequential loop takes 40 seconds, and 39 of those seconds are pure waiting. Nothing about that work needs to happen in order, yet the loop forces it to.

The real-world impact

That waiting shows up as slow batch jobs, sluggish data-collection scripts, and API clients that time out because they can't keep up. Scale the list up and it gets worse linearly: a report that pulls from a few hundred endpoints turns a quick task into a coffee break, and a service that fans out to several upstreams makes every user wait for the slowest chain of calls.

The Solution

The Fix

Use aiohttp with asyncio so the requests overlap. While one call waits on the network, the event loop starts the next one. Reuse a single ClientSession so connections get pooled, launch everything with asyncio.gather, and put a Semaphore in front to keep the number of in-flight requests sane.

TL;DR

Create one aiohttp.ClientSession and reuse it for every request.
Fire all the fetches concurrently with asyncio.gather instead of a blocking loop.
Cap concurrency with asyncio.Semaphore so you don't overwhelm the server or get rate-limited.

Script Implementation

Setup and imports

Start with the async pieces you need: asyncio for the event loop and aiohttp for the HTTP calls. A small result type keeps the return values easy to read.

#!/usr/bin/env python3
"""fetch_urls.py - fetch many URLs concurrently with aiohttp."""
import asyncio
import time
from dataclasses import dataclass

import aiohttp

# Cap how many requests run at once. Tune to the server and your rate limits.
MAX_CONCURRENCY = 10
REQUEST_TIMEOUT = aiohttp.ClientTimeout(total=15)


@dataclass
class Result:
    url: str
    status: int | None  # HTTP status, or None if the request never completed
    body: str | None  # response text on success
    error: str | None  # error message on failure

Fetching one URL safely

Each fetch takes the shared session and the semaphore. The semaphore blocks once MAX_CONCURRENCY requests are already in flight, so the rest queue up instead of stampeding the server. The try/except keeps a single bad URL from crashing the whole batch.

async def fetch_one(
    session: aiohttp.ClientSession,
    sem: asyncio.Semaphore,
    url: str,
) -> Result:
    # The semaphore is the throttle: only MAX_CONCURRENCY of these run at once.
    async with sem:
        try:
            async with session.get(url) as response:
                # Read the body while the connection is still open.
                text = await response.text()
                return Result(url, response.status, text, None)
        except aiohttp.ClientError as exc:
            # Network, DNS, or connection errors land here.
            return Result(url, None, None, f"client error: {exc}")
        except asyncio.TimeoutError:
            return Result(url, None, None, "timed out")

Fetching the whole batch

Create one session and one semaphore, then build a task per URL and hand them all to asyncio.gather. Because every fetch_one already catches its own errors, gather returns a clean list of Result objects with no partial failures to unpack.

async def fetch_all(urls: list[str]) -> list[Result]:
    sem = asyncio.Semaphore(MAX_CONCURRENCY)
    # One session for the whole run so connections get pooled and reused.
    async with aiohttp.ClientSession(timeout=REQUEST_TIMEOUT) as session:
        tasks = [fetch_one(session, sem, url) for url in urls]
        # gather runs them concurrently and preserves input order.
        return await asyncio.gather(*tasks)

Entry point

Wrap the run in asyncio.run, then print a short summary. Timing the run makes the win obvious the first time you compare it to a sequential loop.

def main() -> None:
    urls = [f"https://httpbin.org/delay/1?id={i}" for i in range(20)]

    start = time.perf_counter()
    results = asyncio.run(fetch_all(urls))
    elapsed = time.perf_counter() - start

    ok = sum(1 for r in results if r.status == 200)
    failed = [r for r in results if r.error]
    print(f"fetched {ok}/{len(results)} in {elapsed:.2f}s")
    for r in failed:
        print(f"  failed: {r.url} ({r.error})")


if __name__ == "__main__":
    main()

Usage and Benefits

Why This Helps

One session, one semaphore, and one gather call turn a linear wait into an overlapping one. The requests still take the same time individually, but they no longer take that time one after another, so a batch that scaled with the number of URLs now scales with your concurrency limit instead.

Real invocations

# Install the dependency first.
pip install aiohttp

# Run the script. Twenty 1-second requests finish in about 2-3 seconds,
# not 20, because up to MAX_CONCURRENCY of them run at the same time.
python fetch_urls.py

Tuning concurrency for the target

There's no universal number for MAX_CONCURRENCY. A generous internal API might handle 50 in-flight requests; a rate-limited public one might block you above 5. Start low, watch for 429 responses, and raise it until you hit the server's comfort zone.

# Lower the limit for a strict public API to avoid 429s.
MAX_CONCURRENCY = 5

# Raise it for a fast internal service that can take the load.
MAX_CONCURRENCY = 50

Comparison

How the concurrent approach stacks up against the other common ways to fetch a list of URLs.

Approach	Concurrent	Connection reuse	Concurrency control	Best for
`aiohttp` + `asyncio.gather`	Yes	Yes, pooled	Yes, semaphore	Large async I/O-bound fetches
`requests` in a `for` loop	No	Per session only	N/A	A handful of simple calls
`requests` + `ThreadPool`	Yes	Yes	Pool size	Mixing with blocking code
`httpx.AsyncClient`	Yes	Yes, pooled	Yes, limits/semaphore	Async with a requests-like API

The threaded pool is a fine middle ground when the rest of your code is synchronous, but an event loop scales to far more concurrent connections without spinning up a thread per request.

Frequently Asked Questions

<Accordion client:load title="Why reuse one ClientSession instead of creating one per request?"

A ClientSession owns a connection pool. Reusing it lets aiohttp keep TCP and TLS connections alive across requests, which skips the handshake cost on every call. Creating a fresh session per request throws that away and leaks connections. Open one session for the batch, pass it to every fetch, and close it with async with when you're done.

<Accordion client:load title="What does the Semaphore actually protect against?"

Without a limit, asyncio.gather starts every request at once. A thousand URLs means a thousand simultaneous connections, which can exhaust file descriptors on your side and trip rate limits or overload the server on the other. The semaphore caps how many run concurrently: each async with sem acquires a slot and releases it when the request finishes, so the rest wait their turn.

<Accordion client:load title="How do I stop one failing URL from crashing the whole batch?"

Catch the errors inside each task, which is what fetch_one does with its try/except. Because every coroutine returns a Result instead of raising, asyncio.gather never sees an exception and returns a full list. If you'd rather let exceptions propagate, use asyncio.gather(*tasks, return_exceptions=True) and inspect the returned values, some of which will be exception objects.

<Accordion client:load title="Does asyncio.gather preserve the order of my URLs?"

Yes. gather returns results in the same order you passed the awaitables, regardless of which requests finish first. So results[i] always corresponds to urls[i], even though the fetches complete out of order. If you don't care about order and want to process results as they arrive, look at asyncio.as_completed instead.

<Accordion client:load title="When should I use threads or httpx instead of aiohttp?"

Reach for a thread pool with requests when the surrounding code is synchronous and you don't want to convert it to async just for a few calls. Reach for httpx.AsyncClient when you want an async client with an API closer to requests, or you need HTTP/2. aiohttp shines when you're already in an async application and need to fan out to many endpoints at high concurrency.

<Accordion client:load title="How do I add a timeout so slow requests don't hang forever?"

Pass an aiohttp.ClientTimeout to the session, as the snippet does with REQUEST_TIMEOUT. A total timeout bounds the whole request, including connect and read. When it's exceeded, aiohttp raises asyncio.TimeoutError, which the except in fetch_one turns into a clean failed Result instead of a hung coroutine.

References

Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff

Mohammad Abu Mattar — Mon, 20 Jul 2026 12:29:14 GMT

Quick Tip

Wrap any flaky command in one reusable retry function and stop re-running red pipelines by hand.

The Problem

The Problem

Some commands fail for reasons that have nothing to do with your code. A registry hiccups, an API rate-limits you for a second, a DNS lookup blips, an AWS endpoint returns a 503. Run the exact same command again and it works. That's a transient failure, and it's everywhere in scripts that touch the network.

Why blind retries make it worse

The naive fix is a quick loop that retries immediately, but that often makes things worse. If a service is already struggling, a tight retry loop hammers it harder. Worse, if a hundred CI jobs all fail at the same moment and all retry at the same moment, they come back in a synchronized wave. That's the thundering herd, and it can keep a recovering service down.

The real-world impact

The everyday cost is a pipeline that goes red for no real reason, so someone re-runs it by hand and loses ten minutes and their focus. The worse cost is a deploy script that gives up on the first blip and leaves a release half-applied. Both come from treating a temporary failure as a permanent one.

The Solution

The Fix

Retry the command a few times, but wait longer between each attempt, and add a little randomness so retries spread out instead of stacking up. That's exponential backoff with jitter, and it fits in one small Bash function you can wrap around anything.

TL;DR

Retry any command with a configurable number of attempts.
Back off exponentially (1s, 2s, 4s, 8s) so you stop hammering a struggling service.
Add jitter so a fleet of scripts doesn't retry in lockstep.

Script Implementation

Setup and defaults

Start with a safe shell, then read the knobs from the environment so the same function works in a laptop shell and a CI job without editing the code.

#!/usr/bin/env bash
# retry.sh - retry a command with exponential backoff and jitter.
set -euo pipefail

# Tunables (override via environment):
: "${RETRY_MAX_ATTEMPTS:=5}"   # how many tries before giving up
: "${RETRY_BASE_DELAY:=1}"     # first backoff, in seconds
: "${RETRY_MAX_DELAY:=60}"     # cap so backoff never runs away

The retry function

The function takes the command and its arguments as "$@", so it wraps anything without quoting gymnastics. It returns the command's own exit code on success, and the last exit code if it runs out of attempts.

retry() {
  local attempt=1
  while true; do
    # Run the command exactly as passed. On success, we're done.
    if "$@"; then
      return 0
    fi
    local exit_code=$?

    # Out of attempts: surface the failure clearly and pass the code up.
    if ((attempt >= RETRY_MAX_ATTEMPTS)); then
      echo "retry: '$*' failed after ${attempt} attempts (exit ${exit_code})" >&2
      return "$exit_code"
    fi

    # Exponential backoff: base * 2^(attempt-1), capped at max delay.
    local delay=$((RETRY_BASE_DELAY * 2 ** (attempt - 1)))
    ((delay > RETRY_MAX_DELAY)) && delay=$RETRY_MAX_DELAY

    # Full jitter: sleep a random amount between 0 and delay.
    local wait=$((RANDOM % (delay + 1)))
    echo "retry: attempt ${attempt}/${RETRY_MAX_ATTEMPTS} failed (exit ${exit_code}); waiting ${wait}s" >&2
    sleep "$wait"
    ((attempt++))
  done
}

Why the jitter matters

The backoff line grows the wait each round, and the cap stops it from turning a fifth attempt into a multi-minute stall. The jitter line is the part people skip, and it's the one that prevents the thundering herd. Instead of every caller waiting exactly 4 seconds and retrying at the same instant, each one waits a random slice of that window, so the load spreads out and the recovering service gets room to breathe.

Entry point

If you run the file directly it retries whatever you pass on the command line. If you source it, you just get the retry function.

# Run directly: retry.sh <command...>
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
  if (($# == 0)); then
    echo "usage: retry.sh <command> [args...]" >&2
    exit 64
  fi
  retry "$@"
fi

Usage and Benefits

Why This Helps

One function covers every flaky command in your scripts, so you stop copy-pasting sleep loops and you get consistent logging for free. Because it wraps "$@", it does not care whether the command is aws, curl, kubectl, or a script of your own.

Real invocations

# Wrap an AWS CLI upload that sometimes hits throttling.
retry aws s3 cp ./build.tar.gz s3://artifacts/build.tar.gz

# Wrap a health check that races a service starting up.
retry curl -fsS https://api.example.com/health

# Tune it inline for a slow, important step.
RETRY_MAX_ATTEMPTS=8 RETRY_BASE_DELAY=2 retry ./deploy.sh

Sourcing it in CI/CD

Keep retry.sh in a shared scripts library and source it at the top of your pipeline steps, so every job gets the same behavior.

source ./scripts/retry.sh
retry terraform apply -auto-approve

Comparison

How the function compares to the other common options.

Approach	Backoff	Jitter	Works for any command
`retry()` function	Yes, exponential	Yes	Yes
Tight `until` loop	No	No	Yes
`curl --retry N`	Yes (curl 7.66+)	Only with `--retry-all-errors` extras	No, curl only
AWS CLI built-in retries	Yes (adaptive mode)	Yes	No, AWS CLI only

Tool-specific retries are great when you're already in that tool, but the function is the one thing that wraps all of them the same way.

Frequently Asked Questions

<Accordion client:load title="Why use exponential backoff instead of a fixed delay?"

A fixed delay treats a one-second blip and a thirty-second outage the same way. Exponential backoff waits a little after the first failure and progressively longer after each one, so you recover fast from a quick hiccup but stop hammering a service that's genuinely down. The cap (RETRY_MAX_DELAY) keeps the later waits from becoming a stall.

<Accordion client:load title="What is jitter and why does it prevent a thundering herd?"

Jitter is deliberate randomness added to the wait time. Without it, every script that failed at the same moment retries at the same moment, hitting the recovering service in a synchronized wave. With full jitter, each caller sleeps a random amount between zero and the backoff window, so the retries spread out and the load smooths.

<Accordion client:load title="How do I retry only on certain failures, not every non-zero exit?"

Wrap the command in a small function that maps the failures you care about to a non-zero exit and everything else to zero. For example, treat a curl exit of 22 (HTTP error) or 28 (timeout) as retryable and return success for the rest, then pass that wrapper to retry. That keeps the retry loop generic while you decide what "failure" means.

<Accordion client:load title="Does this work in POSIX sh or only Bash?"

It uses Bash features: RANDOM, local, ((...)) arithmetic with **, and BASH_SOURCE. In a strict POSIX sh you'd swap RANDOM for something like awk or /dev/urandom, replace local, and compute the power manually. If you can run #!/usr/bin/env bash, keep it as is; it's simpler and clearer.

<Accordion client:load title="How is this different from curl --retry or the AWS CLI's own retries?"

Those are excellent inside their own tool. curl --retry retries HTTP calls; the AWS CLI has adaptive retry modes for API throttling. The Bash function is the layer above: it retries anything, including your own scripts and combinations of commands, with one consistent policy and one consistent log format.

<Accordion client:load title="How do I cap the total time spent retrying?"

Either lower RETRY_MAX_ATTEMPTS and RETRY_MAX_DELAY so the worst case is bounded, or wrap the call in GNU timeout, for example timeout 120 retry ./deploy.sh. The timeout approach gives you a hard ceiling regardless of how the backoff math works out.

References

Node.js Environment Variable Validation with Zod at Startup

Mohammad Abu Mattar — Wed, 27 May 2026 20:02:59 GMT

Most Node.js apps treat process.env like a trusted friend. You reach into it whenever you need a value, assume the key is there, assume it's spelled right, and assume the string is actually the type you want. Since process.env hands you everything as a plain string, you end up parsing ports into integers and turning "false" into a real boolean all over the codebase.

The fix is small and boring in the best way: validate your environment once, at startup, with a schema. If the config is wrong, the app refuses to boot and tells you exactly what's broken before a single request is served. Here's how to do it with Zod and a little bit of TypeScript.

The Problem

Why does reading straight from process.env keep biting you?

Silent configuration drift

The Problem In most Node.js apps it is normal to grab values straight from process.env whenever and wherever you need them. You probably load these into your environment from a local file, but that setup never actually checks if your data makes sense. Your code just assumes every setting is present, spelled correctly, and formatted right. And because process.env treats everything as a string, you find yourself parsing variables by hand all over the place. That's where subtle bugs creep in: maybe you forgot to parse a port into an integer, or a string that says "false" gets evaluated as truthy.

The real-world impact

When these settings break, your application usually doesn't crash right away. Instead, you get painful runtime errors much later. Miss a database URL or an API key, and your server might start up perfectly fine. It's only hours later, when a user triggers a specific request, that the app suddenly blows up. These delayed failures make debugging a nightmare because the crash happens far from the actual mistake.

Hunting for a one-character typo in your environment config during a live production outage is exactly the kind of 2 AM stress nobody signed up for. The earlier you catch it, the cheaper it is.

The Solution

How do you catch bad config before the app even starts?

The startup schema assertion

To put an end to this, set up a validation schema that runs the second your application boots. With Zod, you write a clear, strict schema describing exactly what your environment should look like. This runs before any of your server logic starts, so an invalid configuration won't let the app launch at all. Zod checks the values and converts strings into real numbers, booleans, or objects.

It helps to picture it as a mapping. Your raw environment is just string keys paired with string values pulled from :

Zod's parser maps that loose, unstructured data into a strictly typed configuration space :

where is the valid typed domain for each key, such as for ports, a format-compliant URI for the database connection, or a value from a fixed enum.

**The Fix:** one schema, verified the moment your app turns on. The rest of your system then works with safe, fully typed values instead of messy raw strings.

Quick takeaways

TL;DR

Crash early and safely: the app won't boot if any required setting is missing or invalid.
Automatic type conversion: raw strings become real numbers, booleans, and validated URLs, fully typed for TypeScript.
Single source of truth: all configuration checks live in one place, so validation logic isn't scattered everywhere.

Where this fits in your project

Before the code, here's where each piece lives.

your-app/
- src/
  - env.ts the schema and the validated env export
  - index.ts imports env at the very top
  - server.ts uses the typed config
- .env local values, gitignored
- .env.example documented keys, committed
- package.json

The whole trick is that env.ts runs its validation on import, so importing it from your entry file is what makes a broken config stop the boot.

Building the validator

Four small pieces, wired together once.

First, install Zod (and dotenv so you can load a local .env while developing):

npm install zod dotenv

pnpm add zod dotenv

yarn add zod dotenv

bun add zod dotenv

Now build it up step by step.

Load the local environment. Pull .env into process.env before anything else runs, and bring in Zod.

import dotenv from 'dotenv';
import {z} from 'zod';

// Load our local .env file before we validate anything
dotenv.config();

Declare a strict schema. Spell out every variable you need and the type you expect. Zod handles strings, coerced numbers, booleans, and custom enums for you, and z.infer gives you a matching TypeScript type for free.

// Define the validation rules and expected types for our variables
export const envSchema = z.object({
  NODE_ENV: z
    .enum(['development', 'production', 'staging', 'test'])
    .default('development'),
  PORT: z.coerce.number().min(1).max(65535).default(3000),
  DATABASE_URL: z
    .string()
    .url({message: 'DATABASE_URL must be a valid connection URL'}),
  JWT_SECRET: z
    .string()
    .min(32, {message: 'JWT_SECRET must be at least 32 characters long'}),
  ENABLE_LOGS: z.preprocess((val) => val === 'true', z.boolean()).default(true),
});

// Infer the strict TypeScript type directly from our schema
export type EnvConfig = z.infer<typeof envSchema>;

Parse once and fail loudly. Run the schema against process.env. If anything is wrong, print a clean list of issues and exit before the server starts.

// Parse the environment and handle any validation failures gracefully
const validateEnv = (): EnvConfig => {
  const result = envSchema.safeParse(process.env);

  if (!result.success) {
    console.error('Environment validation failed during boot:');

    // Print each issue clearly without leaking any sensitive data
    result.error.issues.forEach((issue) => {
      const propertyPath = issue.path.join('.');
      console.error(`  - [${propertyPath}]: ${issue.message}`);
    });

    // Terminate the process immediately
    process.exit(1);
  }

  return result.data;
};

// Export our validated configuration so the rest of the app can use it
export const env = validateEnv();

Import it at the very top of your entry file. Because validation runs on import, a broken config stops the process before app.listen is ever reached.

import {env} from './env';
import express from 'express';

const app = express();

// The port is now guaranteed to be a valid, parsed number
app.listen(env.PORT, () => {
  console.log(`Server is running in ${env.NODE_ENV} mode on port ${env.PORT}`);
});

Usage and Benefits

What do you actually get out of this?

Architectural advantages

Why This Helps This pattern saves you from writing repetitive null checks and manual parsing logic throughout your codebase. You don't have to wonder whether a variable is undefined or whether a port was correctly parsed into a number. Some developers prefer to extend the global ProcessEnv type in TypeScript for editor autocomplete, but that only helps at compile time. It won't stop a runtime error when a variable is missing or a boolean string is parsed incorrectly. By validating at startup, you get both editor autocomplete and guaranteed runtime safety.

Diagnostic console output

If you miss a variable or format one incorrectly, the app halts immediately and prints a neat, readable error message. This feedback makes it easy to see exactly what went wrong so you can fix it right away.

$ npm run dev

Environment validation failed during boot:
  - [DATABASE_URL]: DATABASE_URL must be a valid connection URL
  - [PORT]: Number must be less than or equal to 65535
  - [JWT_SECRET]: JWT_SECRET must be at least 32 characters long

Notice the output shows the field path and the message, never the value. A bad `JWT_SECRET` or `DATABASE_URL` won't end up in your production logs.

Comparing validation approaches

Is Zod really the best fit, or just the popular one?

Here's a quick look at how Zod stacks up against other common ways developers handle configuration checks.

Validation Pattern	Type Safety (TypeScript)	Automatic Coercion	Boilerplate Required	Error Output Detail
Zod Schema	Native type inference	High, built-in	Low definition overhead	Detailed, structured issues
Joi Library	Requires manual types	Moderate	Medium validation block	Detailed schema errors
Custom Script	Manual type assertions	Manual conversion code	High custom coding	Whatever you write yourself

Manual scripts and older libraries get the job done, but Zod gives you full TypeScript inference, automatic coercion, and clean error reporting with almost no extra work.

Frequently Asked Questions

<Accordion client:load title="Should I validate environment variables in every file or just once?"

Just once, at startup. Validate in a single env.ts module, export the typed env object, and import that everywhere else instead of reading process.env directly. Because the validation runs on import, the check happens exactly one time as the app boots, and every other file consumes already-safe values.

<Accordion client:load title="How do I handle optional variables or sensible defaults?"

Use Zod's .optional() for values that may be absent and .default(...) for ones that should fall back to a known value. For example, PORT: z.coerce.number().default(3000) means a missing PORT quietly becomes 3000, while DATABASE_URL: z.string().url() stays required and fails the boot if it's missing or malformed.

<Accordion client:load title="Does this give me autocomplete on my config?"

Yes, that's one of the best parts. z.infer<typeof envSchema> produces a TypeScript type that matches your schema exactly, so the exported env object is fully typed: env.PORT is a number, env.NODE_ENV is the enum, and so on. You get autocomplete and compile-time checks without manually maintaining a separate type.

<Accordion client:load title="Won't printing validation errors leak my secrets?"

No. The error handler only logs each issue's path and message (for example [JWT_SECRET]: must be at least 32 characters long), never the actual value. Your secrets stay out of the logs even when validation fails, which keeps the output safe to surface in CI and production.

<Accordion client:load title="Can I use this in a frontend or browser bundle?"

This pattern is built for server-side Node.js. In the browser, secrets should never ship in the bundle at all, so rely on your framework's public-env mechanism instead (like NEXT_PUBLIC_ variables or import.meta.env). Keep this Zod schema on the server, and only expose the handful of values that are genuinely safe for the client.

<Accordion client:load title="How is this different from envalid or Joi?"

The core idea is the same: assert your config against a schema at startup. Zod's edge is native TypeScript inference (no separate type definitions), built-in coercion, and the fact that many projects already use Zod for request and form validation. Reusing one library for both means less to learn and one consistent way to describe data.

References

AWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances

Mohammad Abu Mattar — Tue, 14 Apr 2026 20:02:59 GMT

If you've ever spent 20 minutes clicking through the AWS Console just to stop a handful of dev instances, you already know the pain. It's tedious, it doesn't scale, and one wrong click can ruin your afternoon.

That's where Boto3 comes in. It's the official AWS SDK for Python, and it lets you talk to AWS services, including EC2, directly from your code. Instead of pointing and clicking, you write a script once and run it whenever you need it. Your code doesn't get tired, doesn't misclick, and works the same at 3 AM as it does at 3 PM.

This guide covers everything you need to get comfortable with EC2 automation: setting up authentication, starting and stopping instances, waiting for state changes, filtering by tags, running bulk operations, and handling errors.

Why automate EC2 management with Boto3?

Why bother writing code when the AWS Console already does the job?

Fair question. The console is fine when you've got two or three instances. But once you're managing a real environment, manual work starts costing you more than time.

Every action you take through the console depends on a human doing it right, every single time. With Boto3, you write the logic once, and it runs consistently regardless of who's on call or how tired they are.

Here's where automation pays off most:

Scheduled shutdowns: Stop dev and staging instances overnight or on weekends. There's no reason to pay for servers that nobody's using at 2 AM.
Consistent tagging: Every instance your script touches gets tagged correctly. No more missing Environment or Owner tags because someone forgot.
Incident response: When something goes wrong, your runbook can stop a compromised instance or spin up a replacement without waiting for a human to log in.
Audit trails: Log every action with timestamps, so you've got a clear record of what happened, when, and why.
Fewer mistakes: Repetitive manual work breeds errors. Scripts don't forget steps or click the wrong button.

Once your automation is in place, managing 200 instances takes roughly the same effort as managing two.

Setting up Boto3 and authenticating with AWS

How do you connect Boto3 to your AWS account securely?

Start with the install:

pip install boto3

Now, authentication. This is where a lot of people make mistakes, so it's worth doing right from the start. The method you use should depend on where your code runs.

Option 1: IAM roles (best for EC2 and Lambda)

If your script runs on an EC2 instance or a Lambda function, attach an IAM role to that resource. Boto3 picks up the credentials automatically from the instance metadata service. You don't need to touch a key file or set any environment variables.

import boto3

# No keys needed here Boto3 reads the IAM role attached to this instance
ec2_client = boto3.client('ec2', region_name='us-east-1')

# Quick check to confirm the connection works
response = ec2_client.describe_instances()
print('Connected via IAM role good to go.')

This is the cleanest approach for production. There are no credentials to rotate, leak, or accidentally commit to Git.

Option 2: named profiles (best for local development)

When you're working locally, use a named AWS CLI profile. It keeps credentials in ~/.aws/credentials where they belong, not in your source code.

# Run this once in your terminal to set up a profile
aws configure --profile myproject

import boto3

# Load credentials from the 'myproject' profile in ~/.aws/credentials
session = boto3.Session(profile_name='myproject')
ec2_client = session.client('ec2', region_name='us-east-1')

print(f'Using profile: {session.profile_name}')

You can have multiple profiles for different accounts or environments, which makes switching between dev and prod much easier.

A word on hardcoded credentials: Don't do it. Not in scripts, not in config files checked into source control, not anywhere. Use IAM roles in production and named profiles locally. If you're using access keys, rotate them regularly and give them only the permissions they actually need.

Starting and stopping EC2 instances

How do you start and stop instances programmatically with Boto3?

This is probably why you're here. Boto3 uses start_instances and stop_instances for these operations. Both are simple calls, but there are a few details worth knowing.

Stopping an instance

import boto3
from botocore.exceptions import ClientError

def stop_instance(instance_id: str, region: str = 'us-east-1') -> dict:
    """Stop a running EC2 instance and return the state transition."""
    ec2 = boto3.client('ec2', region_name=region)

    try:
        response = ec2.stop_instances(InstanceIds=[instance_id])

        # AWS tells us both the old state and the new state
        state_info = response['StoppingInstances'][0]
        previous = state_info['PreviousState']['Name']
        current = state_info['CurrentState']['Name']

        print(f'{instance_id}: {previous} -> {current}')
        return response

    except ClientError as e:
        print(f'Could not stop instance: {e.response["Error"]["Code"]} - {e}')
        raise

# Usage
stop_instance('i-0abcd1234ef567890')

Starting an instance

import boto3
from botocore.exceptions import ClientError

def start_instance(instance_id: str, region: str = 'us-east-1') -> dict:
    """Start a stopped EC2 instance and return the state transition."""
    ec2 = boto3.client('ec2', region_name=region)

    try:
        response = ec2.start_instances(InstanceIds=[instance_id])

        state_info = response['StartingInstances'][0]
        previous = state_info['PreviousState']['Name']
        current = state_info['CurrentState']['Name']

        print(f'{instance_id}: {previous} -> {current}')
        return response

    except ClientError as e:
        code = e.response['Error']['Code']

        # Don't crash if the instance is already running
        if code == 'IncorrectInstanceState':
            print(f'{instance_id} is already in the desired state.')
        else:
            raise

# Usage
start_instance('i-0abcd1234ef567890')

One thing worth noting: both methods accept a list of instance IDs, so you can act on multiple instances in a single API call. If you're managing more than one instance, that's a lot cleaner than looping and calling separately.

# Stop multiple instances at once no loop needed
ec2 = boto3.client('ec2', region_name='us-east-1')
ec2.stop_instances(InstanceIds=[
    'i-0abcd1234ef567890',
    'i-0efgh5678ij901234',
    'i-0klmn9012op345678',
])

The response tells you the previous and current state for each one, which makes logging and auditing easy.

Using Boto3 waiters to block until a state is reached

How do you know when an instance has actually finished starting or stopping?

Here's a common mistake: you call start_instances, assume it's done, and immediately try to SSH in. But the instance is still booting. Your script fails, and now you're debugging something that wasn't actually broken.

The fix is waiters. A Boto3 waiter is a built-in polling loop that keeps checking the AWS API until your instance reaches the state you want. It handles the timing for you and raises an error if something goes wrong or takes too long.

import boto3
from botocore.exceptions import WaiterError

def start_and_wait(instance_id: str, region: str = 'us-east-1'):
    """Start an instance and block until it's fully running."""
    ec2 = boto3.client('ec2', region_name=region)

    ec2.start_instances(InstanceIds=[instance_id])
    print(f'Starting {instance_id}...')

    waiter = ec2.get_waiter('instance_running')

    try:
        waiter.wait(
            InstanceIds=[instance_id],
            WaiterConfig={
                'Delay': 15,       # Check every 15 seconds
                'MaxAttempts': 40  # Give up after ~10 minutes
            }
        )
        print(f'{instance_id} is running.')

    except WaiterError as e:
        print(f'Waiter timed out: {e}')
        raise


def stop_and_wait(instance_id: str, region: str = 'us-east-1'):
    """Stop an instance and block until it's fully stopped."""
    ec2 = boto3.client('ec2', region_name=region)

    ec2.stop_instances(InstanceIds=[instance_id])
    print(f'Stopping {instance_id}...')

    waiter = ec2.get_waiter('instance_stopped')

    try:
        waiter.wait(
            InstanceIds=[instance_id],
            WaiterConfig={'Delay': 15, 'MaxAttempts': 40}
        )
        print(f'{instance_id} is stopped.')

    except WaiterError as e:
        print(f'Waiter failed: {e}')
        raise

The four most useful EC2 waiters are instance_running, instance_stopped, instance_terminated, and instance_exists. Each one polls describe_instances under the hood and checks the state automatically, so you don't have to.

If you want to wait on multiple instances at once, just pass all the IDs together:

# Wait for several instances to stop one waiter call handles all of them
waiter = ec2.get_waiter('instance_stopped')
waiter.wait(
    InstanceIds=['i-0abc123', 'i-0def456', 'i-0ghi789'],
    WaiterConfig={'Delay': 15, 'MaxAttempts': 40}
)
print('All instances are stopped.')

Tip: Tune Delay and MaxAttempts based on your actual instances. A small instance with a simple AMI might be running in under a minute. A larger one running a heavy bootstrap script could take five or ten. If your waiter times out too often, bump up MaxAttempts before adding more complex logic.

Filtering and querying instances

How do you find the right instances without knowing their IDs ahead of time?

Hardcoding instance IDs is a trap. Instances get replaced, IDs change, and suddenly your script is managing the wrong thing, or nothing at all. A better approach is to query by attributes you control, like tags or state.

Querying by state

import boto3

def get_instances_by_state(state: str, region: str = 'us-east-1') -> list:
    """
    Return all instances in a given state.
    Valid states: 'running', 'stopped', 'pending', 'stopping', 'terminated'
    """
    ec2 = boto3.client('ec2', region_name=region)

    response = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': [state]}]
    )

    # describe_instances groups results into Reservations, so we flatten them
    instances = []
    for reservation in response['Reservations']:
        instances.extend(reservation['Instances'])

    return instances


# Find everything that's currently running
running = get_instances_by_state('running')
print(f'{len(running)} running instance(s) found.')

for inst in running:
    print(f"  {inst['InstanceId']}  {inst['InstanceType']}  {inst['Placement']['AvailabilityZone']}")

Querying by tags

import boto3

def get_instances_by_tag(tag_key: str, tag_value: str, region: str = 'us-east-1') -> list:
    """Find instances that have a specific tag key/value pair."""
    ec2 = boto3.client('ec2', region_name=region)

    # AWS filter syntax for tags is 'tag:<key>'
    response = ec2.describe_instances(
        Filters=[{'Name': f'tag:{tag_key}', 'Values': [tag_value]}]
    )

    instances = []
    for reservation in response['Reservations']:
        instances.extend(reservation['Instances'])

    return instances


# Find all production instances
prod = get_instances_by_tag('Environment', 'production')

for inst in prod:
    # Pull the Name tag if it exists, otherwise label it 'unnamed'
    name = next(
        (t['Value'] for t in inst.get('Tags', []) if t['Key'] == 'Name'),
        'unnamed'
    )
    print(f"  {inst['InstanceId']}  {name}  {inst['State']['Name']}")

You can stack filters together, too. AWS applies them with AND logic, so you'll only get instances that match all of them:

# Only running instances in the production environment both filters must match
response = ec2.describe_instances(
    Filters=[
        {'Name': 'tag:Environment', 'Values': ['production']},
        {'Name': 'instance-state-name', 'Values': ['running']}
    ]
)

That single call replaces what would otherwise be a manual filter in the console or a clunky post-processing loop in code.

Bulk operations: stopping instances by tag or VPC

How do you stop an entire group of instances at once without listing each ID manually?

This is where scripting really earns its keep. Instead of hunting down individual instance IDs, you describe what you want, collect the IDs, and stop them all in one shot.

Stop all instances in a VPC

import boto3

def stop_instances_in_vpc(vpc_id: str, region: str = 'us-east-1') -> list:
    """Stop all running instances in a given VPC."""
    ec2 = boto3.client('ec2', region_name=region)

    # Find every running instance in this VPC
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'vpc-id', 'Values': [vpc_id]},
            {'Name': 'instance-state-name', 'Values': ['running']}
        ]
    )

    instance_ids = [
        inst['InstanceId']
        for r in response['Reservations']
        for inst in r['Instances']
    ]

    if not instance_ids:
        print(f'Nothing running in {vpc_id}.')
        return []

    print(f'Stopping {len(instance_ids)} instance(s) in {vpc_id}...')
    ec2.stop_instances(InstanceIds=instance_ids)

    return instance_ids


stopped = stop_instances_in_vpc('vpc-0abc12345def67890')
print(f'Stopped: {stopped}')

Stop all instances with a specific tag

import boto3

def stop_instances_by_tag(
    tag_key: str,
    tag_value: str,
    region: str = 'us-east-1',
    dry_run: bool = True
) -> list:
    """
    Stop all running instances that match a tag.
    Set dry_run=True to preview what would be stopped without actually doing it.
    """
    ec2 = boto3.client('ec2', region_name=region)

    response = ec2.describe_instances(
        Filters=[
            {'Name': f'tag:{tag_key}', 'Values': [tag_value]},
            {'Name': 'instance-state-name', 'Values': ['running']}
        ]
    )

    instance_ids = [
        inst['InstanceId']
        for r in response['Reservations']
        for inst in r['Instances']
    ]

    if not instance_ids:
        print(f'No running instances tagged {tag_key}={tag_value}.')
        return []

    if dry_run:
        print(f'[DRY RUN] Would stop {len(instance_ids)} instance(s):')
        for iid in instance_ids:
            print(f'  - {iid}')
        return instance_ids

    ec2.stop_instances(InstanceIds=instance_ids)
    print(f'Stopped {len(instance_ids)} instance(s) tagged {tag_key}={tag_value}.')

    return instance_ids


# Always preview before committing
stop_instances_by_tag('Environment', 'dev', dry_run=True)

# When you're confident, flip the flag
# stop_instances_by_tag('Environment', 'dev', dry_run=False)

Always build a dry_run mode into your bulk scripts. It takes two minutes to add and can save you from a very bad day. Preview first, execute second.

Error handling: throttling, permissions, and invalid states

What happens when your Boto3 script hits an error, and how do you handle it cleanly?

AWS APIs fail. Not often, but enough that your production scripts need to be ready for it. The three most common issues you'll run into are rate limiting, missing permissions, and trying to do something that doesn't make sense for the instance's current state.

Here's a function that handles all of them gracefully:

import boto3
import time
from botocore.exceptions import ClientError

def robust_stop_instance(
    instance_id: str,
    region: str = 'us-east-1',
    max_retries: int = 3
) -> bool:
    """
    Stop an instance with sensible error handling and retry logic.
    Returns True on success, False if the operation can't be completed.
    """
    ec2 = boto3.client('ec2', region_name=region)

    for attempt in range(1, max_retries + 1):
        try:
            response = ec2.stop_instances(InstanceIds=[instance_id])
            state = response['StoppingInstances'][0]['CurrentState']['Name']
            print(f'Stopped {instance_id}. Current state: {state}')
            return True

        except ClientError as e:
            code = e.response['Error']['Code']

            if code in ('RequestLimitExceeded', 'Throttling'):
                # Back off exponentially and try again
                wait_time = 2 ** attempt
                print(f'Rate limited. Waiting {wait_time}s before retry {attempt}/{max_retries}...')
                time.sleep(wait_time)

            elif code == 'UnauthorizedOperation':
                # No point retrying this is a permissions issue
                print(f'Permission denied: {e.response["Error"]["Message"]}')
                print('Check the IAM policy on your role or profile.')
                return False

            elif code == 'IncorrectInstanceState':
                # Instance might already be stopped or terminated
                print(f'{instance_id} is in an unexpected state. Skipping.')
                return False

            elif code == 'InvalidInstanceID.NotFound':
                # Wrong region, or the instance no longer exists
                print(f'{instance_id} not found in {region}.')
                return False

            else:
                # Something unexpected surface it clearly
                print(f'Unexpected error [{code}]: {e}')
                raise

    print(f'Gave up after {max_retries} attempts.')
    return False

The key error codes to know:

RequestLimitExceeded / Throttling: You're hitting the AWS API rate limit. Use exponential backoff and retry.
UnauthorizedOperation: The IAM role or user doesn't have the right permission. No amount of retrying will fix this. You need to update the policy.
IncorrectInstanceState: You're trying to do something the instance's current state doesn't allow, like stopping an instance that's already stopped.
InvalidInstanceID.NotFound: The instance ID doesn't exist in that region. Double-check both the ID and the region.

Frequently asked questions

<Accordion client:load title="Can Boto3 manage EC2 instances across multiple regions in one script?"

Yes, and it's pretty easy to do. Just create a separate client for each region you need. You can loop over a list of region names and run the same operations in each one. If you need it to go fast, concurrent.futures lets you run those calls in parallel.

import boto3
from concurrent.futures import ThreadPoolExecutor

regions = ['us-east-1', 'eu-west-1', 'ap-southeast-1']

def count_running(region):
    ec2 = boto3.client('ec2', region_name=region)
    resp = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    )
    count = sum(len(r['Instances']) for r in resp['Reservations'])
    print(f'{region}: {count} running instance(s)')

# Query all regions at the same time instead of one by one
with ThreadPoolExecutor() as pool:
    pool.map(count_running, regions)

<Accordion client:load title="What IAM permissions does my role need to start and stop instances?"

At a minimum you'll need ec2:StartInstances, ec2:StopInstances, ec2:DescribeInstances, and ec2:DescribeInstanceStatus. If you're filtering by tags, add ec2:DescribeTags too. Scope permissions as tightly as you can using IAM condition keys, especially for production environments.

<Accordion client:load title="How do Boto3 waiters differ from just sleeping for a fixed amount of time?"

A fixed sleep is a guess. A waiter actually checks. Waiters poll the AWS API and return the moment the state changes, which means they're faster when things go smoothly and more informative when they don't. If the timeout is exceeded, you get a WaiterError you can handle, rather than a script that silently moves on assuming everything is fine.

<Accordion client:load title="Is there a difference between using a Boto3 client and a resource for EC2?"

Yes. boto3.client('ec2') is the low-level interface that maps directly to API calls and returns raw dictionaries. boto3.resource('ec2') gives you a higher-level, object-oriented interface with things like ec2.Instance('i-0abc123'). Both work, but the client is more flexible for filtering and bulk operations, which is why most automation scripts use it.

<Accordion client:load title="Can I use Boto3 to manage spot instances the same way as on-demand instances?"

Mostly yes. start_instances, stop_instances, and describe_instances all work with spot instances. The main difference is that spot instances can be interrupted by AWS when capacity is reclaimed, so your automation should be ready for unexpected state changes. Use describe_spot_instance_requests to check spot-specific status and watch for interruption notices.

<Accordion client:load title="How do I handle pagination when there are many instances to query?"

describe_instances only returns up to 1,000 results per call. For bigger environments, use a paginator instead of calling directly:

import boto3

ec2 = boto3.client('ec2', region_name='us-east-1')
paginator = ec2.get_paginator('describe_instances')

instances = []
# paginate() handles NextToken automatically no manual looping needed
for page in paginator.paginate(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]):
    for reservation in page['Reservations']:
        instances.extend(reservation['Instances'])

print(f'Total running instances: {len(instances)}')

The paginator handles NextToken automatically, so you'll always get everything without extra logic.

<Accordion client:load title="What's the safest way to test bulk stop scripts before running them in production?"

Use the dry_run pattern shown in the bulk operations section. Run with dry_run=True first to see exactly which instances would be affected before you commit. Beyond that, test in a staging account or a non-production VPC. Some Boto3 calls also support a DryRun=True parameter at the API level, which validates your permissions without making any changes.

References

Redis Caching Patterns: Cache-Aside, Write-Through & Cache Invalidation

Mohammad Abu Mattar — Tue, 07 Apr 2026 20:02:59 GMT

Need to scale your backend without throwing money at servers? Start with Redis caching patterns.

Most databases can handle hundreds of queries per second, but thousands? Your app slows to a crawl. Adding another database server just moves the problem. Serving data from memory 99% of the time with Redis fixes it instead.

The Problem

Database load under scale

100 requests/sec for "get product details":
• Database: 200ms per query
• 100 × 200ms = serious bottleneck
• Add 1000 concurrent users = complete collapse
• Throwing more servers doesn't help (they all queue at the database)

The Solution

Redis caching patterns

Redis stores data in memory, so results come back in microseconds instead of milliseconds. The caching pattern you choose decides whether the cache helps or hurts.

TL;DR

Cache-Aside: Check cache first, miss triggers database load + populate cache (most flexible)
Write-Through: Update cache and database together (most consistent)
TTL Strategy: Keys expire automatically, so stale data clears without manual invalidation
Cache Invalidation: Delete the cache entry when data updates so the next request refreshes it
Stampede Prevention: Stop the thundering herd that follows a popular key expiring

Pattern 1: Cache-Aside (Lazy Loading)

How it works

1. Request comes in
2. Check Redis cache
3. Cache HIT → return immediately (microseconds)
4. Cache MISS → query database (milliseconds)
5. Populate cache with TTL
6. Next request hits cache

Python implementation

import redis
import json
from functools import wraps
import time

redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)

def cache_aside(ttl=3600):
    """Decorator implementing cache-aside pattern"""
    def decorator(func):
        @wraps(func)
        def wrapper(key, *args, **kwargs):
            # Check cache first
            cached = redis_client.get(key)
            if cached:
                # Cache hit
                return json.loads(cached)

            # Cache miss: call function (hits database)
            result = func(*args, **kwargs)

            # Populate cache with TTL
            redis_client.setex(key, ttl, json.dumps(result))

            return result
        return wrapper
    return decorator

# Database query (expensive)
def get_user_from_db(user_id):
    # Simulated database query
    time.sleep(0.2)  # 200ms
    return {"id": user_id, "name": f"User {user_id}", "email": f"user{user_id}@example.com"}

# Decorated function
@cache_aside(ttl=3600)
def get_user(user_id):
    return get_user_from_db(user_id)

# Usage
print(get_user(f"user:1"))  # First call: 200ms (cache miss)
print(get_user(f"user:1"))  # Second call: <1ms (cache hit)

Node.js implementation

import {createClient} from 'redis';
import {promisify} from 'util';

const redis = createClient();
redis.connect();

const getAsync = promisify(redis.get).bind(redis);
const setexAsync = promisify(redis.setex).bind(redis);

// Database query (expensive)
async function getUserFromDB(userId: string) {
  // Simulated database query
  await new Promise((resolve) => setTimeout(resolve, 200));
  return {
    id: userId,
    name: `User ${userId}`,
    email: `user${userId}@example.com`,
  };
}

// Cache-aside implementation
async function getUser(userId: string, ttl = 3600) {
  const cacheKey = `user:${userId}`;

  // Check cache
  const cached = await getAsync(cacheKey);
  if (cached) {
    return JSON.parse(cached);
  }

  // Cache miss: hit database
  const user = await getUserFromDB(userId);

  // Populate cache
  await setexAsync(cacheKey, ttl, JSON.stringify(user));

  return user;
}

// Usage
(async () => {
  console.time('first');
  await getUser('user:1'); // 200ms (miss)
  console.timeEnd('first');

  console.time('second');
  await getUser('user:1'); // <1ms (hit)
  console.timeEnd('second');
})();

Go implementation

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"github.com/redis/go-redis/v9"
	"time"
)

type User struct {
	ID    int    `json:"id"`
	Name  string `json:"name"`
	Email string `json:"email"`
}

var rdb = redis.NewClient(&redis.Options{
	Addr: "localhost:6379",
})

func getUserFromDB(ctx context.Context, userID int) (User, error) {
	// Simulated database query
	time.Sleep(200 * time.Millisecond)
	return User{
		ID:    userID,
		Name:  fmt.Sprintf("User %d", userID),
		Email: fmt.Sprintf("user%d@example.com", userID),
	}, nil
}

func getUser(ctx context.Context, userID int) (User, error) {
	cacheKey := fmt.Sprintf("user:%d", userID)

	// Check cache
	val, err := rdb.Get(ctx, cacheKey).Result()
	if err == nil {
		var user User
		json.Unmarshal([]byte(val), &user)
		return user, nil
	}

	// Cache miss: hit database
	user, err := getUserFromDB(ctx, userID)
	if err != nil {
		return user, err
	}

	// Populate cache (1 hour TTL)
	data, _ := json.Marshal(user)
	rdb.SetEx(ctx, cacheKey, string(data), time.Hour)

	return user, nil
}

Pattern 2: Write-Through

How it works

1. Data update arrives
2. Update cache AND database together
3. Both succeed or both fail (atomic)
4. Next read hits cache (always consistent)

When to use it

Critical data (payments, user accounts)
Data that changes frequently
When consistency is more important than speed

Python implementation

import redis
import json

redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)

def update_user_write_through(user_id: int, user_data: dict):
    """Update cache AND database together"""
    cache_key = f"user:{user_id}"

    try:
        # Update cache first (faster)
        redis_client.setex(cache_key, 3600, json.dumps(user_data))

        # Then update database
        # Assuming `db.update_user(user_id, user_data)` exists
        db.update_user(user_id, user_data)

        return True
    except Exception as e:
        # If database fails, invalidate cache
        redis_client.delete(cache_key)
        raise e

# Usage
update_user_write_through(42, {
    "name": "Alice",
    "email": "alice@example.com"
})

# Next read hits cache
user = redis_client.get("user:42")

Node.js implementation

async function updateUserWriteThrough(userId: string, userData: any) {
  const cacheKey = `user:${userId}`;

  try {
    // Update cache first
    await redis.setex(cacheKey, 3600, JSON.stringify(userData));

    // Then update database
    await db.updateUser(userId, userData);

    return true;
  } catch (error) {
    // Rollback: invalidate cache on database failure
    await redis.del(cacheKey);
    throw error;
  }
}

Pattern 3: TTL Strategy

Automatic expiration

Redis automatically deletes keys after TTL expires. No manual invalidation needed.

import redis
from datetime import datetime, timedelta

redis_client = redis.Redis()

# Different TTL for different data types
def cache_with_ttl(key: str, value: str, data_type: str):
    ttl_map = {
        "user_profile": 3600,           # 1 hour (changes infrequently)
        "product": 86400,                # 24 hours (stable data)
        "product_price": 300,            # 5 minutes (changes frequently)
        "session": 1800,                 # 30 minutes (security)
        "statistics": 60,                # 1 minute (real-time)
    }

    ttl = ttl_map.get(data_type, 3600)
    redis_client.setex(key, ttl, value)

# Usage
cache_with_ttl("product:123", "iPhone 15", "product")        # 24 hours
cache_with_ttl("price:apple", "$999", "product_price")       # 5 minutes
cache_with_ttl("stats:daily", "1000 users", "statistics")    # 1 minute

Sliding Window TTL

Reset TTL on each access (useful for sessions):

def get_with_sliding_ttl(key: str, ttl: int):
    """Get value and reset TTL"""
    value = redis_client.get(key)
    if value:
        # Reset TTL on access
        redis_client.expire(key, ttl)
    return value

# Usage: Session stays alive as long as user is active
session = get_with_sliding_ttl("session:abc123", 1800)  # 30 min

Pattern 4: Cache Invalidation

Event-driven invalidation

Delete cache when data changes:

def update_product(product_id: int, new_data: dict):
    # Update database
    db.update_product(product_id, new_data)

    # Invalidate cache immediately
    redis_client.delete(f"product:{product_id}")

    # Also invalidate related caches (cascade invalidation)
    redis_client.delete(f"category:{new_data['category']}")
    redis_client.delete("products:all")

# When product price changes
update_product(123, {"price": "$899", "category": "electronics"})
# → Deletes "product:123", "category:electronics", "products:all"

Tag-based invalidation

Invalidate multiple keys with one operation:

import json

def cache_with_tags(key: str, value: str, tags: list):
    """Cache value and tag it for bulk invalidation"""
    # Store value
    redis_client.setex(key, 3600, value)

    # Store tag reference (set of keys with this tag)
    for tag in tags:
        redis_client.sadd(f"tag:{tag}", key)

def invalidate_by_tag(tag: str):
    """Invalidate all keys with a tag"""
    # Get all keys with this tag
    keys = redis_client.smembers(f"tag:{tag}")

    # Delete all tagged keys
    if keys:
        redis_client.delete(*keys)

    # Clean up tag
    redis_client.delete(f"tag:{tag}")

# Usage
cache_with_tags("product:1", "iPhone", ["electronics", "apple"])
cache_with_tags("product:2", "MacBook", ["electronics", "apple"])
cache_with_tags("product:3", "Apple Watch", ["wearables", "apple"])

# Invalidate all Apple products at once
invalidate_by_tag("apple")

Preventing cache stampedes

The Problem

Thousands of requests for a popular key
↓
Key expires
↓
ALL requests hit database simultaneously (thundering herd)
↓
Database overload, service degradation

Solution: probabilistic regeneration

import redis
import random
import time

redis_client = redis.Redis()

def get_with_low_ttl_regen(key: str, expensive_calculation, ttl=3600):
    """
    Regenerate cache probabilistically near expiration
    Avoids thundering herd when TTL reaches 0
    """
    value = redis_client.get(key)

    if value:
        # Check TTL
        remaining_ttl = redis_client.ttl(key)

        # Near end of life? Regenerate probabilistically
        if remaining_ttl < 0:
            # Key expired
            pass  # Recalculate below
        elif remaining_ttl < ttl * 0.2:  # Last 20% of life
            # Probability of regeneration increases as TTL decreases
            prob = 1 - (remaining_ttl / (ttl * 0.2))
            if random.random() < prob:
                # First request regenerates, others get stale data temporarily
                new_value = expensive_calculation()
                redis_client.setex(key, ttl, new_value)
                return new_value

        return value

    # Cache miss: calculate fresh data
    result = expensive_calculation()
    redis_client.setex(key, ttl, result)
    return result

Distributed lock pattern

Prevent parallel cache recalculations:

import time
import uuid

def get_with_lock(key: str, expensive_calculation, lock_timeout=10):
    """Use lock to prevent multiple calculations"""
    lock_key = f"lock:{key}"
    cache_key = key

    # Check cache
    cached = redis_client.get(cache_key)
    if cached:
        return cached

    # Try to acquire lock
    lock_id = str(uuid.uuid4())
    acquired = redis_client.set(lock_key, lock_id, nx=True, ex=lock_timeout)

    if acquired:
        # We got the lock, calculate
        try:
            result = expensive_calculation()
            redis_client.setex(cache_key, 3600, result)
            return result
        finally:
            # Release lock
            if redis_client.get(lock_key) == lock_id:
                redis_client.delete(lock_key)
    else:
        # Someone else got the lock, wait for result
        for _ in range(10):
            time.sleep(0.1)
            cached = redis_client.get(cache_key)
            if cached:
                return cached

        # Timeout: calculate ourselves
        return expensive_calculation()

Serialization strategies

JSON vs MessagePack vs Protocol Buffers

import json
import msgpack

def store_json(key, obj):
    """Simple but larger"""
    redis_client.set(key, json.dumps(obj))

def store_msgpack(key, obj):
    """Smaller, faster"""
    redis_client.set(key, msgpack.packb(obj))

# Benchmark: 1000 products
products = [{"id": i, "name": f"Product {i}", "price": 99.99} for i in range(1000)]

json_size = len(json.dumps(products))          # ~50KB
msgpack_size = len(msgpack.packb(products))    # ~30KB (40% smaller)

# Choice depends on:
# - JSON: Human readable, broad ecosystem
# - MsgPack: Smaller, faster
# - Protobuf: Strongly typed, enterprise

Monitoring cache performance

Track hit rate

import redis

redis_client = redis.Redis()

def get_cache_stats():
    """Get cache performance metrics"""
    info = redis_client.info('stats')

    total_commands = info.get('total_commands_processed', 0)
    hits = info.get('keyspace_hits', 0)
    misses = info.get('keyspace_misses', 0)

    hit_rate = (hits / (hits + misses)) * 100 if (hits + misses) > 0 else 0

    return {
        "hit_rate": hit_rate,
        "total_hits": hits,
        "total_misses": misses,
        "evictions": info.get('evicted_keys', 0),
        "used_memory": info.get('used_memory_human', 'N/A'),
    }

# Target: >80% hit rate
# <50%: Cache is ineffective, reconsider keys/TTLs
# >95%: Good fit for caching strategy

Connection pooling

Reuse connections instead of creating new ones:

import redis
from redis.connection import ConnectionPool

# Without pooling (inefficient)
conn = redis.Redis()

# With pooling (efficient)
pool = ConnectionPool.from_url(
    'redis://localhost:6379',
    max_connections=50,
    decode_responses=True
)
redis_client = redis.Redis(connection_pool=pool)

# Reuses connections automatically
for i in range(1000):
    redis_client.get(f"key:{i}")

Best practices

1. Consistent key naming

# Good: Hierarchical, searchable
"user:123"
"user:123:settings"
"product:456"
"product:456:reviews"
"order:789:items"

# Bad: Ambiguous
"u123"
"prod_456"
"item-789"

2. Set Reasonable TTLs

TTL_STRATEGY = {
    "user_profile": 3600,           # 1 hour
    "product_catalog": 86400,       # 24 hours
    "session": 1800,                # 30 minutes
    "api_response": 300,            # 5 minutes
    "real_time_data": 60,           # 1 minute
}

3. Plan for failures

def safe_cache_get(key: str, fallback_fn):
    """Gracefully handle Redis failures"""
    try:
        value = redis_client.get(key)
        if value:
            return value
    except redis.ConnectionError:
        # Redis down? Use fallback
        pass

    # No cache: call function
    return fallback_fn()

Conclusion

Redis caching is the difference between a responsive application and a slow one.

Choose the right pattern (cache-aside for flexibility, write-through for consistency), set appropriate TTLs, prevent stampedes, and monitor hit rates. Combined with database optimization, caching is what keeps a high-traffic system fast.

Resources

PostgreSQL Query Optimization: Indexes, EXPLAIN ANALYZE & Execution Plans

Mohammad Abu Mattar — Sun, 29 Mar 2026 20:02:59 GMT

Need to optimize slow PostgreSQL queries? Use EXPLAIN ANALYZE and targeted indexing.

Slow database queries kill application performance. Most developers don't know where the actual bottleneck is, so they guess at fixes. EXPLAIN ANALYZE shows what is expensive and where to optimize.

The Problem

Why queries get slow

In development with small datasets, missing indexes don't matter. Switch to production with millions of rows, and sequential table scans start to cost real time. Queries that ran in 50ms take 5+ seconds. You need to know why.

Common performance issues

Missing indexes: Forcing full table scans on every query
N+1 queries: Fetching data in loops instead of bulk operations
Bad query plans: Using inefficient joins or sorts
Undersized indices: Index on wrong columns
Connection pool exhaustion: Running out of available connections

The Solution

Reading EXPLAIN ANALYZE

EXPLAIN ANALYZE shows exactly how PostgreSQL executes your query, including:

Which operations are most expensive (by cost)
Actual vs. estimated row counts
Time spent in each step
Seq Scans vs. Index Scans
Sort and Hash operations

TL;DR

Run EXPLAIN ANALYZE to identify expensive operations
Add indexes where sequential scans happen on large tables
Use pg_stat_statements to automatically find slow queries
Fix N+1 query patterns before rewriting complex queries
Set up connection pooling so connections get reused instead of reopened

Understanding EXPLAIN ANALYZE

Basic query analysis

-- Compare these two queries
EXPLAIN ANALYZE
SELECT * FROM users WHERE created_at > '2025-01-01';

-- Output example:
-- Seq Scan on users  (cost=0.00..35.50 rows=1000 width=100)
--   Filter: (created_at > '2025-01-01')
--   Planning Time: 0.123 ms
--   Execution Time: 45.234 ms

What this tells you:

Seq Scan: Reading entire table (slow for large tables)
cost=0.00..35.50: PostgreSQL's estimated cost
rows=1000: Expected to return 1000 rows
Execution Time: 45.234 ms: Actual time taken

Reading the cost

EXPLAIN ANALYZE
SELECT u.id, u.name, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.status = 'active';

-- Output:
-- Hash Left Join  (cost=2534.50..5892.33 rows=5000 width=50)
--   Hash Cond: (o.user_id = u.id)
--   ->  Seq Scan on orders o  (cost=0.00..1234.50 rows=50000 width=8)
--   ->  Hash  (cost=2500.00..2500.00 rows=5000 width=42)
--         ->  Index Scan using idx_users_status on users u  (cost=10.00..2500.00 rows=5000 width=42)

Cost interpretation:

Lower cost = faster execution
cost=A..B: A = startup cost, B = total cost
Focus on the highest-cost operations first

Creating effective indexes

Basic index creation

-- Simple B-tree index (most common)
CREATE INDEX idx_users_email ON users(email);

-- Multi-column index
CREATE INDEX idx_orders_user_date ON orders(user_id, created_at);

-- Partial index (only index active users)
CREATE INDEX idx_users_active ON users(id) WHERE status = 'active';

-- Unique index (constraint + performance)
CREATE UNIQUE INDEX idx_users_email_unique ON users(email);

Index types and when to use each

-- B-tree (default, best for most queries)
CREATE INDEX idx_standard ON table_name(column);

-- Hash (equality only, rarely faster than B-tree)
CREATE INDEX idx_hash ON table_name USING hash(column);

-- GiST (geometric, full-text search)
CREATE INDEX idx_fulltext ON articles USING gist(search_vector);

-- GIN (array, JSON, full-text - faster for large result sets)
CREATE INDEX idx_json ON logs USING gin(metadata);

Partial indexes for common cases

-- Don't index inactive records
CREATE INDEX idx_active_orders ON orders(id) WHERE status != 'cancelled';

-- Only index recent data
CREATE INDEX idx_recent_events ON events(id) WHERE created_at > NOW() - INTERVAL '90 days';

-- Saves space and speeds up queries on active data

Multi-column index strategy

-- For WHERE + ORDER BY combinations
-- Query: WHERE user_id = ? ORDER BY created_at DESC
CREATE INDEX idx_user_date ON orders(user_id, created_at DESC);

-- Column order matters: Put filtered columns first
-- Good:   WHERE status = ? AND type = ?
CREATE INDEX idx_good ON events(status, type);

-- Bad:    WHERE status = ? AND type = ?
CREATE INDEX idx_bad ON events(type, status);  -- Wrong order

Detecting and fixing N+1 queries

The N+1 problem

# BAD: N+1 queries (1 + N queries)
users = db.query(User).all()  # Query 1
for user in users:
    orders = db.query(Order).filter(Order.user_id == user.id).all()  # Queries 2 to N+1
    print(f"{user.name}: {len(orders)} orders")

# Result with 1000 users = 1001 queries

The fix: eager loading

# GOOD: Eager load with join
from sqlalchemy import joinedload

users = db.query(User).options(joinedload(User.orders)).all()  # Single query with join

for user in users:
    print(f"{user.name}: {len(user.orders)} orders")

SQL equivalent

-- BAD (N+1):
SELECT * FROM users;  -- 1000 queries...
SELECT * FROM orders WHERE user_id = ?;

-- GOOD (Single query):
SELECT u.id, u.name, o.id, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

Finding slow queries automatically

Enable query logging

-- Enable logging of slow queries (500ms+)
ALTER SYSTEM SET log_min_duration_statement = 500;
SELECT pg_reload_conf();

-- View current setting
SHOW log_min_duration_statement;

Use pg_stat_statements

-- Install extension
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Find slowest queries
SELECT query, calls, mean_time, max_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

-- Find queries that were called most
SELECT query, calls, mean_time
FROM pg_stat_statements
ORDER BY calls DESC
LIMIT 10;

-- Clear stats to get fresh baseline
SELECT pg_stat_statements_reset();

Query optimization patterns

Pattern 1: missing index on a WHERE clause

-- Before: Seq Scan (slow)
EXPLAIN ANALYZE
SELECT * FROM orders WHERE customer_id = 42;

-- Fix: Add index
CREATE INDEX idx_orders_customer ON orders(customer_id);

-- After: Index Scan (fast)
EXPLAIN ANALYZE
SELECT * FROM orders WHERE customer_id = 42;

Pattern 2: inefficient subqueries

-- SLOW: Subquery evaluated for each row
SELECT * FROM orders
WHERE customer_id IN (
  SELECT id FROM customers WHERE status = 'premium'
);

-- FAST: Use JOIN instead
SELECT DISTINCT o.*
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE c.status = 'premium';

Pattern 3: functions in WHERE clauses

-- SLOW: Function applied to indexed column
SELECT * FROM users WHERE LOWER(email) = 'test@example.com';

-- FAST: Use expression index
CREATE INDEX idx_users_email_lower ON users(LOWER(email));
SELECT * FROM users WHERE LOWER(email) = 'test@example.com';

-- OR: Normalize incoming data instead
SELECT * FROM users WHERE email = 'test@example.com';

Pattern 4: missing ORDER BY index

-- SLOW: Sort step required
EXPLAIN ANALYZE
SELECT * FROM events ORDER BY created_at DESC LIMIT 10;

-- Fix: Add index for sort
CREATE INDEX idx_events_date_desc ON events(created_at DESC);

-- Now uses Index Scan + Limit, no Sort step

Connection pooling

Why connection pooling matters

-- Without pooling: Each request opens/closes connection (expensive)
-- With pooling: Reuse existing connections (cheap)

PgBouncer configuration

[databases]
myapp = host=localhost port=5432 dbname=myapp_prod

[pgbouncer]
listen_port = 6432
listen_addr = 127.0.0.1
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

# Connection pooling settings
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 25
min_pool_size = 10
reserve_pool_size = 5
reserve_pool_timeout = 3

Application connection

# Before pooling: Direct to PostgreSQL
import psycopg2
conn = psycopg2.connect("dbname=myapp user=postgres host=localhost")

# After pooling: Connect to PgBouncer on port 6432
import psycopg2
conn = psycopg2.connect("dbname=myapp user=postgres host=localhost port=6432")

Real-world optimization workflow

Step 1: identify the slow query

# From application logs or pg_stat_statements
# Example: SELECT query is taking 8000ms

Step 2: run EXPLAIN ANALYZE

EXPLAIN ANALYZE
SELECT o.id, o.total, c.name
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.created_at > '2025-01-01'
ORDER BY o.total DESC;

Step 3: look for expensive operations

Sort (cost=9234.50..9244.50)  ← Look here
  ->  Hash Join (cost=2500.00..9234.00)  ← And here
        ->  Seq Scan on orders o (cost=0.00..5000.00)  ← Sequential scan on large table
        ->  Hash (cost=100.00..100.00 rows=1000)
              ->  Seq Scan on customers c (cost=0.00..100.00)

Step 4: add indexes

-- Index for WHERE clause
CREATE INDEX idx_orders_date ON orders(created_at);

-- Index for JOIN condition
CREATE INDEX idx_orders_customer_id ON orders(customer_id);

-- Index for ORDER BY
CREATE INDEX idx_orders_total ON orders(total DESC);

Step 5: rerun EXPLAIN ANALYZE

-- Should now use Index Scans instead of Seq Scans
-- No Sort step if you have the right index

Best practices

1. Index naming convention

-- Consistent naming helps find related indexes
CREATE INDEX idx_table_column_type ON table_name(column);

-- Examples:
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_orders_customer_date ON orders(customer_id, created_at);
CREATE INDEX idx_products_active ON products(id) WHERE active = true;

2. Monitor index usage

-- Find unused indexes (bloat)
SELECT schemaname, tablename, indexname
FROM pg_indexes
WHERE schemaname NOT IN ('pg_catalog', 'information_schema');

-- Check if index is being used
SELECT
  indexrelname,
  idx_scan,
  idx_tup_read,
  idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;

3. Regular maintenance

-- Analyze table stats (for query planner)
ANALYZE users;

-- Vacuum to clean up dead rows
VACUUM FULL users;

-- Reindex if fragmented (heavy write workloads)
REINDEX INDEX idx_users_email;

Resources

Multi-Environment Secret Management with HashiCorp Vault

Mohammad Abu Mattar — Tue, 03 Mar 2026 00:00:00 GMT

Managing secrets safely across multiple environments with HashiCorp Vault

Storing secrets in .env files, hardcoding them, or even using separate secret managers per environment creates security risks and operational chaos. Vault provides a unified, audited solution for secret generation, rotation, and access control across all your environments.

The Problem

Multi-environment secret chaos

When managing dev, staging, and production environments, secrets management becomes a headache:

Different secrets per environment with no central tracking
Manual rotation processes prone to human error
No audit trail for who accessed which secrets and when
Secrets leaking through Git history or logs
Difficulty revoking compromised credentials instantly

The Solution

HashiCorp Vault overview

Vault is a secrets management platform that provides:

Dynamic secrets: Generate credentials on-demand that auto-expire
Secret rotation: Automatically rotate credentials without downtime
Audit logging: Complete trail of all secret access and operations
Environment isolation: Separate secrets per environment with shared policies
Multi-auth methods: Support for AppRole, Kubernetes, IAM, JWT, and more

TL;DR

Use HashiCorp Vault as a centralized secret backend
Generate dynamic credentials that auto-expire
Implement automatic rotation for long-lived secrets
Audit all secret access across environments

Installation and setup

Start Vault with Docker

services:
  vault:
    image: vault:latest
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
    command: server -dev

volumes:
  vault-data:

Start the Vault server:

docker-compose up -d
export VAULT_ADDR='http://localhost:8200'
export VAULT_TOKEN='root'

CLI examples

Basic secret storage

# Store a static secret
vault kv put secret/dev/database \
  username="dbuser" \
  password="supersecret" \
  host="db.dev.internal"

# Retrieve secret
vault kv get secret/dev/database

# Get specific field
vault kv get -field=password secret/dev/database

Organize secrets by environment

# Development secrets
vault kv put secret/dev/app-api \
  api_key="dev-key-12345" \
  api_secret="dev-secret-67890"

# Staging secrets
vault kv put secret/staging/app-api \
  api_key="staging-key-abcde" \
  api_secret="staging-secret-fghij"

# Production secrets
vault kv put secret/prod/app-api \
  api_key="prod-key-xyz123" \
  api_secret="prod-secret-abc456"

Programming examples

Python: fetch secrets

import hvac
import os
from typing import Dict, Any

class VaultClient:
    def __init__(self, vault_addr: str = "http://localhost:8200", token: str = None):
        self.client = hvac.Client(url=vault_addr, token=token or os.getenv('VAULT_TOKEN'))

    def get_secret(self, path: str, field: str = None) -> Dict[str, Any]:
        """Fetch a secret from Vault"""
        response = self.client.secrets.kv.v2.read_secret_version(path=path)
        data = response['data']['data']

        if field:
            return data.get(field)
        return data

    def get_database_creds(self, environment: str) -> Dict[str, str]:
        """Get database credentials for specific environment"""
        path = f"secret/{environment}/database"
        return self.get_secret(path)

    def get_api_keys(self, environment: str) -> Dict[str, str]:
        """Get API keys for specific environment"""
        path = f"secret/{environment}/app-api"
        return self.get_secret(path)

# Usage
if __name__ == "__main__":
    vault = VaultClient()

    # Get dev database credentials
    db_creds = vault.get_database_creds("dev")
    print(f"Database: {db_creds['host']}")
    print(f"User: {db_creds['username']}")

    # Get specific field
    api_key = vault.get_secret("secret/prod/app-api", field="api_key")
    print(f"API Key: {api_key}")

Node.js: fetch secrets

import * as VaultClient from 'node-vault';

class SecretManager {
  private vault: ReturnType<typeof VaultClient>;

  constructor(address: string = 'http://localhost:8200', token: string) {
    this.vault = VaultClient({
      apiVersion: 'v1',
      endpoint: address,
      token: token || process.env.VAULT_TOKEN,
    });
  }

  async getSecret(path: string, field?: string): Promise<any> {
    try {
      const response = await this.vault.read(path);
      const data = response.data.data;

      if (field) {
        return data[field];
      }
      return data;
    } catch (error) {
      console.error(`Failed to retrieve secret from ${path}:`, error);
      throw error;
    }
  }

  async getDatabaseConfig(environment: string): Promise<any> {
    return this.getSecret(`secret/${environment}/database`);
  }

  async getApiCredentials(environment: string): Promise<any> {
    return this.getSecret(`secret/${environment}/app-api`);
  }
}

// Usage
(async () => {
  const manager = new SecretManager('http://localhost:8200', 'root');

  const dbConfig = await manager.getDatabaseConfig('dev');
  console.log(`Connecting to: ${dbConfig.host}`);

  const apiKey = await manager.getApiCredentials('prod');
  console.log(`API Key: ${apiKey.api_key}`);
})();

Dynamic database credentials

Configure the database secret engine

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://admin:password@db.prod.internal:5432/appdb" \
  username="vault_admin" \
  password="vault_admin_password"

# Create readonly role
vault write database/roles/readonly \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Create readwrite role
vault write database/roles/readwrite \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="6h" \
  max_ttl="24h"

Retrieve dynamic credentials

# Get temporary readonly credentials
vault read database/creds/readonly
# Output:
# Key                Value
# ---                -----
# lease_duration     1h
# lease_id           database/creds/readonly/...
# password           a-temporary-password
# username           v-token-readonly-xxxxxxxx

# Get temporary readwrite credentials
vault read database/creds/readwrite

Python: use dynamic credentials

import hvac
import psycopg2
from typing import Generator
import contextlib

class DynamicDatabaseAccess:
    def __init__(self, vault_addr: str, vault_token: str):
        self.vault = hvac.Client(url=vault_addr, token=vault_token)
        self.db_host = os.getenv('DB_HOST', 'db.prod.internal')
        self.db_port = os.getenv('DB_PORT', '5432')
        self.db_name = os.getenv('DB_NAME', 'appdb')

    def get_dynamic_credentials(self, role: str) -> dict:
        """Fetch temporary database credentials from Vault"""
        response = self.vault.secrets.database.read_dynamic_credentials(role)
        return response['data']

    @contextlib.contextmanager
    def get_connection(self, role: str = 'readonly') -> Generator:
        """Get a database connection with dynamic credentials"""
        creds = self.get_dynamic_credentials(role)

        conn = psycopg2.connect(
            host=self.db_host,
            port=self.db_port,
            database=self.db_name,
            user=creds['username'],
            password=creds['password']
        )

        try:
            yield conn
        finally:
            conn.close()

# Usage
db_access = DynamicDatabaseAccess('http://localhost:8200', 'root')

with db_access.get_connection(role='readonly') as conn:
    cursor = conn.cursor()
    cursor.execute("SELECT COUNT(*) FROM users")
    print(f"Total users: {cursor.fetchone()[0]}")

Policy-based access control

Create policies for different roles

# Read-only access to dev/staging secrets
path "secret/data/dev/*" {
  capabilities = ["read", "list"]
}

path "secret/data/staging/*" {
  capabilities = ["read", "list"]
}

# Deny access to production
path "secret/data/prod/*" {
  capabilities = ["deny"]
}

# Allow token self-renewal
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Full access to production secrets
path "secret/data/prod/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Read-only access to staging
path "secret/data/staging/*" {
  capabilities = ["read", "list"]
}

# Database credential access
path "database/creds/prod/*" {
  capabilities = ["read"]
}

# Audit logging
path "sys/audit" {
  capabilities = ["read"]
}

Apply policies

# Create policies
vault policy write dev-team policies/dev-team.hcl
vault policy write prod-team policies/prod-team.hcl

# Assign to users/apps
vault write auth/userpass/users/jane policies="prod-team"
vault write auth/userpass/users/alice policies="dev-team"

Kubernetes integration

Enable Kubernetes auth

# Enable Kubernetes authentication
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create role for pods
vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app \
  bound_service_account_namespaces=default \
  policies=default,app-secrets \
  ttl=24h

Pod configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      serviceAccountName: app
      containers:
        - name: myapp
          image: myapp:latest
          env:
            - name: VAULT_ADDR
              value: 'http://vault.vault.svc.cluster.local:8200'
            - name: VAULT_SKIP_VERIFY
              value: 'false'
          volumeMounts:
            - name: vault-token
              mountPath: /vault/secrets
      volumes:
        - name: vault-token
          projected:
            sources:
              - serviceAccountToken:
                  audience: vault
                  expirationSeconds: 3600
                  path: token

Audit logging

Enable audit logging

# Enable file audit backend
vault audit enable file file_path=/vault/logs/audit.log

# Enable syslog audit backend
vault audit enable syslog tag="vault"

# View audit logs
tail -f /vault/logs/audit.log | jq '.'

Query audit logs

# Show all secret reads
cat /vault/logs/audit.log | jq 'select(.type == "response" and .auth.policy_results.granted_policies >= 0)'

# Show secret modifications
cat /vault/logs/audit.log | jq 'select(.type == "request" and .request.operation == "write")'

# Show failed auth attempts
cat /vault/logs/audit.log | jq 'select(.response.auth == null)'

Best practices

1. Use AppRole for applications

# Create AppRole
vault auth enable approle

# Generate application role
vault write auth/approle/role/my-app \
  token_ttl=1h \
  token_max_ttl=4h \
  policies="app-secrets"

# Get role ID
vault read auth/approle/role/my-app/role-id

# Generate secret ID
vault write -f auth/approle/role/my-app/secret-id

2. Rotate secrets regularly

# Configure auto-rotation every 30 days
vault write database/config/postgresql \
  connection_url="postgresql://admin:password@db.prod.internal:5432/appdb" \
  rotation_statements="ALTER ROLE \"{{name}}\" WITH PASSWORD '{{password}}';" \
  rotation_period=720h

3. Implement least privilege

# Only grant necessary capabilities
path "secret/data/myapp/*" {
  capabilities = ["read"]  # Not "update" or "delete"
}

Resources

Top 7 Open Source OCR Models for Document Processing

Mohammad Abu Mattar — Fri, 09 Jan 2026 00:00:00 GMT

AI Tool

Turn your documents into accurate digital copies with these open source OCR models. Instead of fighting messy text extraction, you get clean markdown from PDFs, images, and scanned documents.

What modern OCR models do

Beyond basic text extraction

These models do more than read text. They recognize document structure, tables, diagrams, math equations, and multiple languages, and turn all of it into well-formatted markdown.

Local processing

Run these models on your own machine without sending sensitive documents to cloud services. You keep control over your data and get accuracy close to enterprise services.

olmOCR-2-7B-1025

High-performance document OCR

Allen Institute's flagship OCR model. It is fine-tuned from Qwen2.5-VL-7B-Instruct using GRPO reinforcement learning and scores 82.4 on the olmOCR-bench evaluation.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the model
model_id = "allenai/olmOCR-2-7B-1025"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

# Process document
messages = [
    {"role": "user", "content": "Convert this document to markdown"},
    {"role": "user", "content": f"[IMAGE_PLACEHOLDER]"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=4096)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)

Key features

Mathematical equations: Handles complex math expressions
Table recognition: Keeps table structure and formatting intact
Layout understanding: Manages multi-column documents and complex layouts
Large-scale processing: Built for processing millions of documents
Automated retries: Includes error handling and automatic rotation correction

PaddleOCR VL

Efficient multilingual OCR

An ultra-compact vision-language model. It integrates the NaViT visual encoder with the ERNIE language model and supports 109 languages with minimal resource usage.

import paddle
from paddlenlp import Taskflow

# Initialize OCR pipeline
ocr = Taskflow("document_parsing")

# Process document
result = ocr({"doc": "path/to/document.pdf"})
print(result)

Key features

109 languages: Chinese, English, Japanese, Arabic, Hindi, Thai
Complex elements: Tables, formulas, charts recognition
High accuracy: Strong performance on OmniDocBench
Fast inference: Optimized for real-world deployment
Compact size: Efficient use of memory and compute

OCRFlux-3B

Multimodal document conversion

A preview release from ChatDOC, fine-tuned from Qwen2.5-VL-3B-Instruct for clean markdown output from PDFs and images.

from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "ChatDOC/OCRFlux-3B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("ChatDOC/OCRFlux-3B")

# Process image
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path/to/image.jpg"},
            {"type": "text", "text": "Convert to markdown"}
        ]
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, return_tensors="pt").to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=4096)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

Key features

Clean markdown: Structured output with proper formatting
Cross-page tables: Merges table data across multiple pages
Consumer hardware: Runs on GTX 3090 and similar GPUs
Scalable deployment: vLLM inference support
High accuracy: Strong parsing quality

MiniCPM-V 4.5

Mobile-optimized OCR

The latest model in the MiniCPM-V series. Built on Qwen3-8B and SigLIP2-400M, it handles text recognition in images, documents, and video.

from transformers import AutoModel, AutoTokenizer
import torch

# Load model
model = AutoModel.from_pretrained('openbmb/MiniCPM-V-4.5', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-4.5', trust_remote_code=True)
model.eval().cuda()

# Process image
image = Image.open('path/to/image.jpg').convert('RGB')
question = 'Convert this document to text'

msgs = [{'role': 'user', 'content': [image, question]}]
result = model.chat(msgs=msgs, tokenizer=tokenizer, sampling=True, temperature=0.7)
print(result)

Key features

Mobile deployment: Optimized for edge devices
Multi-image processing: Handles multiple images simultaneously
Video OCR: Text recognition in video content
Benchmark results: Leading performance across evaluations
Practical efficiency: Everyday application ready

InternVL2.5-4B

Compact multimodal understanding

An efficient vision-language model. It combines the InternViT vision encoder with the Qwen2.5 language model for OCR and document understanding.

import torch
from transformers import AutoTokenizer, AutoModel

# Load model
model = AutoModel.from_pretrained(
    'OpenGVLab/InternVL2_5-4B',
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(
    'OpenGVLab/InternVL2_5-4B',
    trust_remote_code=True
)

# Process image
image = Image.open('path/to/image.jpg').convert('RGB')
question = 'Extract all text from this image'

response = model.chat(tokenizer, image, question)
print(response)

Key features

Dynamic resolution: 448x448 pixel tile processing
Resource efficient: Suitable for constrained environments
Text recognition: Strong OCR performance
Reasoning tasks: Advanced multimodal understanding
Compact architecture: 4 billion parameters total

Granite Vision 3.3 2b

Document understanding specialist

IBM's vision-language model, built on Granite 3.1-2b-instruct with a SigLIP2 vision encoder for automated content extraction.

from transformers import AutoProcessor, AutoModelForVision2Seq
import torch

# Load model
processor = AutoProcessor.from_pretrained("ibm-granite/granite-vision-3.3-2b")
model = AutoModelForVision2Seq.from_pretrained("ibm-granite/granite-vision-3.3-2b", device_map="auto")

# Process image
image = Image.open("path/to/document.png").convert("RGB")
text_prompt = "<|start_of_text|>"

inputs = processor(text=text_prompt, images=image, return_tensors="pt").to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

Key features

Table extraction: Automated table content extraction
Chart recognition: Infographics and plot understanding
Multi-page support: Handles multi-page documents
Image segmentation: Advanced visual processing
Safety: Improved security features

TrOCR Large (SROIE Fine-tuned)

Specialized text recognition

A transformer-based OCR model. Its encoder-decoder architecture combines the BEiT image transformer with the RoBERTa text transformer.

from transformers import TrOCRProcessor, VisionEncoderDecoderModel
import torch

# Load model
processor = TrOCRProcessor.from_pretrained('microsoft/trocr-large-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-large-printed')

# Process image
image = Image.open('path/to/image.jpg').convert('RGB')
pixel_values = processor(images=image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)

generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_text)

Key features

Single-line text: Optimized for printed text recognition
High accuracy: Strong performance on benchmarks
Transformer architecture: Modern deep learning approach
Pre-trained models: Uses large-scale training data
Sequence processing: Handles text as sequential tokens

Choosing the right model

Use case considerations

High accuracy: olmOCR-2-7B-1025, OCRFlux-3B
Efficiency: PaddleOCR VL, InternVL2.5-4B
Multilingual: PaddleOCR VL, MiniCPM-V 4.5
Mobile/Edge: MiniCPM-V 4.5, InternVL2.5-4B
Specialized: TrOCR (printed text), Granite Vision (documents)

Performance optimization

Hardware is usually the limiting factor.

GPU memory: Match model size to available VRAM
Batch processing: Use models supporting multiple images
Quantization: Consider quantized versions for efficiency
Local deployment: All models support local inference

Implementation best practices

Preprocessing

Resize oversized images before inference so they fit the model's input budget.

# Image preprocessing
def preprocess_image(image_path):
    image = Image.open(image_path).convert('RGB')
    # Resize if needed
    if max(image.size) > 2240:
        image = image.resize((2240, 2240), Image.Resampling.LANCZOS)
    return image

Error handling

Catch failures per document so one bad file does not stop a batch.

def safe_ocr_processing(model, image_path):
    try:
        image = preprocess_image(image_path)
        result = model.process(image)
        return result
    except Exception as e:
        logging.error(f"OCR processing failed: {e}")
        return None

Output formatting

Wrap the extracted text with metadata so downstream code knows which model produced it.

def format_ocr_output(raw_text, confidence_scores=None):
    """Format OCR output with metadata"""
    return {
        "text": raw_text,
        "confidence": confidence_scores,
        "timestamp": datetime.now().isoformat(),
        "model_version": "olmOCR-2-7B-1025"
    }

These open source OCR models cover most document processing needs today. Choose based on what your workload demands: accuracy, speed, language coverage, or the hardware you have available.

Why printf Beats echo in Linux Scripts

Mohammad Abu Mattar — Fri, 02 Jan 2026 00:00:00 GMT

Scripting Tip

A script that works on your machine can produce different output on another system. The output command is often the reason. printf behaves the same way across shells, and echo does not.

The problem with echo

Unpredictable behavior

The echo command doesn't behave the same way everywhere. Options like -n (to suppress the newline) or escape sequences like \n and \t might work fine in one shell but act completely different in another.

Real-world script failures

A script that runs fine locally starts misbehaving once you deploy it or hand it to a colleague. Those inconsistencies produce bugs that are hard to track down.

Why printf is more reliable

POSIX standard compliance

printf follows the POSIX standard, so it works exactly the same across all shells and systems. You don't have to worry about whether your escape sequences will actually work or not.

Advanced formatting

printf gives you real control over how your output looks. You can align text, set numeric precision, and print several values from one command.

# Simple alignment
printf "%-10s %s\n" "User:" "$USER"

# Clean numeric formatting
printf "Usage: %.2f%%\n" 85.6789

# Multiple values at once
printf "%s logged in at %s\n" "$USER" "$(date)"

Replacing echo with printf

Basic text output

Instead of echo "Hello, world", write printf "Hello, world\n". The newline is spelled out in the command, so you always know what you are getting.

Suppressing newlines

echo -n "Processing..." behaves differently from shell to shell. With printf you leave the newline out of the format string instead: printf "Processing...".

Variable output safety

echo $VARIABLE breaks on spaces and special characters. Use printf "%s\n" "$VARIABLE" instead. It is safe and predictable.

# Safe variable printing
name="John Doe"
printf "User: %s\n" "$name"

# Multiple variables work great
printf "User: %s | UID: %d\n" "$USER" "$UID"

Escape sequences

printf always handles escape sequences correctly. echo may print them as literal text, depending on your shell.

# Reliable line breaks
printf "Line 1\nLine 2\n"

# Clean tabbed output
printf "Name:\t%s\nAge:\t%d\n" "$name" "$age"

Structured output with printf

Table formatting

You can build aligned tables and structured output that other programs can read.

# CPU usage table
printf "%-8s %s\n" "CPU" "Usage"
printf "%-8s %d%%\n" "core0" 42
printf "%-8s %d%%\n" "core1" 37

CSV generation

Generate CSV files that format the same way no matter where your script runs.

printf "%s,%s,%s\n" "Name" "Age" "City"
printf "%s,%s,%s\n" "$name" "$age" "$city"

When to use which command

echo for quick tasks

echo is still fine for quick checks in the terminal:

Testing something quickly
Simple debugging output
Interactive shell sessions
One-off commands

printf for scripts

Use printf when it matters:

Production scripts
Automated tools
Log file generation
Any output that other programs will read
Scripts that need to work across different systems

Locale considerations

Numeric formatting

printf respects your system's locale settings. In some locales, decimal points become commas, which can break scripts that parse the output.

# Force standard decimal format
LC_NUMERIC=C printf "%.2f\n" 3.14159

Safe numeric handling

For scripts that need to work everywhere, set LC_NUMERIC=C or handle numbers explicitly so the locale cannot change the output.

Making the switch

Gradual migration

You can replace your echo statements one by one:

echo "text" becomes printf "text\n"
echo -n "text" becomes printf "text"
echo $var becomes printf "%s\n" "$var"

Testing output

Always test your scripts on different systems and shells to confirm the output is the same everywhere.

The printf advantage

printf gives you the reliability and control that scripts need. echo works for quick tasks, but printf behaves predictably no matter where it runs. Make printf the default for output in production code.

Essential Bash Variables for Every Script

Mohammad Abu Mattar — Fri, 26 Dec 2025 00:00:00 GMT

Overview

Quick Tip

You know what's worse than writing scripts? Writing scripts that break every time you move them to a different machine. Built-in Bash variables fix that.

The problem with hard-coded paths

We've all been there. You hard-code /home/john/.config in your script, and it works great on your laptop. Then you run it on the server, and everything breaks because the server uses a different username or directory structure. When you've got dozens (or hundreds) of scripts, tracking down all these hard-coded values becomes a real headache.

The solution: built-in Bash variables

Bash already knows about your system. It's got built-in variables for script paths, user IDs, home directories, and more. Instead of guessing or hard-coding, just ask Bash. Your scripts will work everywhere without modification.

Key Bash variables summary

TL;DR

Use $BASH_SOURCE (or $0) to get the script path reliably
Check exit codes with $? to handle errors properly
Access arguments with $1, $2, $@, and $*
Get user IDs with $UID and $EUID for permission checks
Use XDG variables for standard user directories instead of hard-coding paths

Special parameters

Get the script path

Sometimes you need to know where your script lives. Maybe you're building a help menu and want to show the script name, or you need to reference files relative to the script's location. Here's how you grab that info.

#!/usr/bin/env bash

# $BASH_SOURCE gives you the full path to this script
SCRIPT_PATH="$BASH_SOURCE"
echo "Full path: $SCRIPT_PATH"
# Output: /home/user/scripts/my_script.sh

# Use basename to strip the directory and keep just the filename
SCRIPT_NAME="$(basename "$BASH_SOURCE")"
echo "Script name: $SCRIPT_NAME"
# Output: my_script.sh

# Here's a practical use case: building a help menu
cat <<EOF
Usage: $SCRIPT_NAME [OPTIONS]

Options:
  -h, --help     Show this help message
  -v, --version  Show version information
EOF

There's also $0, which works similarly but has a quirk. If someone sources your script (like source my_script.sh), $0 returns "bash" instead of the script name. $BASH_SOURCE doesn't have this issue, so use that if you're writing Bash-specific code.

Check exit status

When a command finishes, it leaves behind an exit code, a kind of status report. If everything went fine, you get 0. If something went wrong, you get a number between 1 and 255. The variable $? holds this exit code, and you can use it to figure out what happened and respond accordingly.

#!/usr/bin/env bash

# Method 1: Explicit check using $?
ls /nonexistent_directory
if [[ $? -ne 0 ]]; then
  echo "Error: Directory does not exist"
  exit 1
fi

# Method 2: Cleaner approach test the command directly
# If ls succeeds, run the 'then' block. If it fails, run 'else'
if ls /home; then
  echo "Directory exists"
else
  echo "Directory does not exist"
fi

# Method 3: One-liner with logical operators
# && means "run this if the previous command succeeded"
# || means "run this if the previous command failed"
ls /home && echo "Success!" || echo "Failed!"

# Method 4: Handle specific error codes
# Some commands return different numbers for different errors
grep "pattern" file.txt
case $? in
  0) echo "Pattern found";;
  1) echo "Pattern not found";;
  2) echo "File error or syntax problem";;
  *) echo "Unknown error";;
esac

Here's the catch. $? only remembers the last command. If you run another command, the old exit code is gone. So if you need it later, save it right away:

some_command
exit_code=$?  # Save it now!
echo "Did some other stuff"
# Now we can still check the original exit code
if [[ $exit_code -ne 0 ]]; then
  echo "Original command failed"
fi

Access script arguments

You'll want your scripts to accept arguments: filenames, options, or configuration values. Bash makes this pretty straightforward, though the syntax looks a bit weird at first.

#!/usr/bin/env bash

# Individual arguments are numbered: $1, $2, $3, etc.
echo "First argument: $1"
echo "Second argument: $2"
echo "Third argument: $3"
# Run like: ./script.sh hello world test
# Output: hello, world, test

# Count how many arguments were passed
echo "Total arguments: ${#@}"

# Loop through all arguments
# $@ treats each argument separately, which is usually what you want
echo -e "\nProcessing all arguments:"
for arg in "$@"; do
  echo "  - $arg"
done

# The difference between $@ and $*
# This matters when your arguments contain spaces
function demo_at {
  # Each argument stays separate
  printf '@: [%s]\n' "$@"
}

function demo_star {
  # All arguments merge into one string
  printf '*: [%s]\n' "$*"
}

echo -e "\nDifference between \$@ and \$*:"
demo_at "arg one" "arg two"    # Prints: @: [arg one] and @: [arg two]
demo_star "arg one" "arg two"  # Prints: *: [arg one arg two] (all together)

# Real-world example: A function that processes files
function process_files() {
  # First, make sure we got at least one file
  if [[ ${#@} -eq 0 ]]; then
    echo "Error: No files specified"
    echo "Usage: process_files file1 file2 ..."
    return 1
  fi

  # Process each file
  for file in "$@"; do
    if [[ -f "$file" ]]; then
      echo "Processing: $file"
      # Your actual file processing logic goes here
      # wc -l "$file"  # Example: count lines
    else
      echo "Warning: File not found: $file"
    fi
  done
}

# You'd call it like this:
# process_files report.txt data.csv notes.md

Always use quotes around "$@". If you don't, filenames with spaces will break. For example, without quotes, "my file.txt" becomes two separate arguments: "my" and "file.txt". With quotes, it stays as one argument.

Environment variables

Get user ID

Sometimes you need to know who's running your script. Maybe you're creating user-specific temp files, or you need to check if someone's running as root. That's where $UID and $EUID come in.

#!/usr/bin/env bash

# Root check: A common pattern for admin scripts
# Root user always has UID 0
if [[ $EUID -eq 0 ]]; then
  echo "Running with root privileges"
  # Safe to do system-level stuff here
else
  echo "Running as regular user (UID: $UID)"
  echo "Some operations may require sudo"
  # Maybe prompt for sudo, or just warn and continue
fi

# Build paths specific to the current user
# This is useful for multi-user systems
USER_CACHE_DIR="/run/user/$UID/cache"
echo "User cache directory: $USER_CACHE_DIR"
# Example: /run/user/1000/cache

# Practical example: Create a temp directory that's unique per user
# This prevents conflicts when multiple users run your script
function create_user_temp() {
  local temp_dir="/tmp/myapp-$UID"

  if [[ ! -d "$temp_dir" ]]; then
    mkdir -p "$temp_dir"
    echo "Created temp directory: $temp_dir"
  fi

  echo "$temp_dir"
}

# Now each user gets their own isolated temp space
TEMP_DIR=$(create_user_temp)
echo "Using temp directory: $TEMP_DIR"
# User 1000: /tmp/myapp-1000
# User 1001: /tmp/myapp-1001

You might be wondering about the difference between $UID and $EUID. For most scripts, they're the same. They only differ in weird edge cases involving sudo or special setuid programs. When in doubt, use $EUID for permission checks. That is the one that matters.

Use the XDG directory specification

Where should your script save config files? Or cache data? Or store downloaded files? You might think "just use ~/.config" but what if a user wants to organize things differently? That's where XDG variables come in. They're the standard way Linux systems handle user directories.

#!/usr/bin/env bash

# Set XDG variables if they're not already set
# The syntax ${VAR:=default} means "use VAR if it exists, otherwise use default"
export "${XDG_CONFIG_HOME:=$HOME/.config}"        # Config files
export "${XDG_CACHE_HOME:=$HOME/.cache}"          # Temporary cache data
export "${XDG_DATA_HOME:=$HOME/.local/share}"    # Application data
export "${XDG_STATE_HOME:=$HOME/.local/state}"   # State files (logs, history, etc.)

# Now build your app's specific directories
APP_NAME="myapp"
APP_CONFIG_DIR="$XDG_CONFIG_HOME/$APP_NAME"  # ~/.config/myapp
APP_CACHE_DIR="$XDG_CACHE_HOME/$APP_NAME"    # ~/.cache/myapp
APP_DATA_DIR="$XDG_DATA_HOME/$APP_NAME"      # ~/.local/share/myapp

# Create these directories if they don't exist yet
function initialize_app_directories() {
  mkdir -p "$APP_CONFIG_DIR"
  mkdir -p "$APP_CACHE_DIR"
  mkdir -p "$APP_DATA_DIR"

  echo "Initialized application directories:"
  echo "  Config: $APP_CONFIG_DIR"
  echo "  Cache:  $APP_CACHE_DIR"
  echo "  Data:   $APP_DATA_DIR"
}

initialize_app_directories

# Example: Save settings to the config directory
function save_config() {
  local config_file="$APP_CONFIG_DIR/settings.conf"

  cat > "$config_file" <<EOF
# Application settings
debug_mode=false
log_level=info
max_connections=100
EOF

  echo "Saved config to: $config_file"
}

# Example: Load settings from the config directory
function load_config() {
  local config_file="$APP_CONFIG_DIR/settings.conf"

  if [[ -f "$config_file" ]]; then
    # Source the config file to load variables
    source "$config_file"
    echo "Loaded config from: $config_file"
    echo "Debug mode: $debug_mode"
    echo "Log level: $log_level"
  else
    echo "Config file not found: $config_file"
    return 1
  fi
}

save_config
load_config

This respects how users organize their systems, not just the spec. Some people put config files on a separate partition, or use custom paths for backups. By using XDG variables, your script just works in all these scenarios.

Complete example: production-ready script

Here's what a script looks like when you use all these variables properly. It's got error handling, logging, argument processing, and follows XDG standards. Use it as a template for your own scripts.

#!/usr/bin/env bash

# Strict mode: exit on errors, undefined variables, and pipe failures
set -euo pipefail

# Set up XDG directories with sensible defaults
export "${XDG_CONFIG_HOME:=$HOME/.config}"
export "${XDG_CACHE_HOME:=$HOME/.cache}"
export "${XDG_DATA_HOME:=$HOME/.local/share}"

# Grab script info using the variables we learned about
SCRIPT_NAME="$(basename "$BASH_SOURCE")"
SCRIPT_DIR="$(cd "$(dirname "$BASH_SOURCE")" && pwd)"
APP_NAME="myapp"

# Build our application's directory structure
APP_CONFIG_DIR="$XDG_CONFIG_HOME/$APP_NAME"
APP_CACHE_DIR="$XDG_CACHE_HOME/$APP_NAME"
APP_LOG_FILE="$APP_CACHE_DIR/app.log"

# ANSI color codes for pretty output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'  # Reset to no color

# Simple logging function with timestamps
# Usage: log "INFO" "Something happened"
function log() {
  local level="$1"
  shift
  local message="$*"
  local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

  # Print to screen AND save to log file
  echo "[$timestamp] [$level] $message" | tee -a "$APP_LOG_FILE"
}

# Centralized error handling
# This logs the error and exits with a non-zero status
function error_exit() {
  log "ERROR" "$*" >&2
  exit 1
}

# Check if we have everything we need to run
function check_prerequisites() {
  # Example: Some scripts need root privileges
  if [[ $EUID -ne 0 ]]; then
    error_exit "This script must be run as root (current UID: $UID)"
  fi

  # Verify required commands are installed
  local required_commands=("curl" "jq" "aws")
  for cmd in "${required_commands[@]}"; do
    if ! command -v "$cmd" &> /dev/null; then
      error_exit "Required command not found: $cmd. Please install it first."
    fi
  done

  log "INFO" "All prerequisites satisfied"
}

# Set up directories and initialize the app
function initialize() {
  # Create necessary directories
  mkdir -p "$APP_CONFIG_DIR" "$APP_CACHE_DIR"
  touch "$APP_LOG_FILE"

  # Log some useful info for debugging
  log "INFO" "Initialized $APP_NAME"
  log "INFO" "Script: $SCRIPT_NAME (located at $SCRIPT_DIR)"
  log "INFO" "User: $USER (UID: $UID)"
}

# Parse command-line arguments
function process_arguments() {
  # If no arguments, show help and exit
  if [[ ${#@} -eq 0 ]]; then
    cat <<EOF
Usage: $SCRIPT_NAME [OPTIONS] [FILES...]

Options:
  -h, --help     Show this help message
  -v, --verbose  Enable verbose logging
  -d, --debug    Enable debug mode

Examples:
  $SCRIPT_NAME file1.txt file2.txt
  $SCRIPT_NAME --verbose *.log
EOF
    exit 0
  fi

  # Set up some flags
  local verbose=false
  local debug=false
  local files=()

  # Loop through all arguments
  while [[ $# -gt 0 ]]; do
    case "$1" in
      -h|--help)
        process_arguments  # Recursively call to show help
        ;;
      -v|--verbose)
        verbose=true
        log "INFO" "Verbose mode enabled"
        shift
        ;;
      -d|--debug)
        debug=true
        set -x  # This makes Bash print every command before running it
        shift
        ;;
      -*)
        error_exit "Unknown option: $1"
        ;;
      *)
        # Anything that doesn't start with - is treated as a file
        files+=("$1")
        shift
        ;;
    esac
  done

  log "INFO" "Processing ${#files[@]} file(s)"

  # Process each file
  for file in "${files[@]}"; do
    if [[ -f "$file" ]]; then
      log "INFO" "Processing file: $file"
      # Your actual file processing logic would go here
      # Example: wc -l "$file"
    else
      log "WARN" "File not found: $file"
    fi
  done
}

# Main entry point
function main() {
  initialize
  check_prerequisites || exit $?
  process_arguments "$@" || exit $?

  log "INFO" "Script completed successfully"
}

# This is where everything starts
# We pass all command-line arguments to main using "$@"
main "$@"

Quick reference

Variable	Description	Example
`$0` or `$BASH_SOURCE`	Script path	`/home/user/script.sh`
`$(basename $0)`	Script name only	`script.sh`
`$?`	Exit status of last command	`0` (success) or `1-255` (error)
`$1`, `$2`, `$3`...	Positional parameters	First, second, third argument
`$@`	All arguments as array	Each argument separate
`$*`	All arguments as string	All arguments in one string
`${#@}`	Number of arguments	`3`
`$UID`	Real user ID	`1000`
`$EUID`	Effective user ID	`0` (when using sudo)
`$USER`	Username	`john`
`$HOME`	Home directory	`/home/john`
`$XDG_CONFIG_HOME`	Config directory	`~/.config`
`$XDG_CACHE_HOME`	Cache directory	`~/.cache`
`$XDG_DATA_HOME`	Data directory	`~/.local/share`

Best practices

Do these things

Always quote your variables: Use "$var" instead of $var. This prevents weird issues when variables contain spaces or special characters.
Set defaults for XDG variables: Use ${XDG_CONFIG_HOME:=$HOME/.config} so your script works even if the variable isn't set.
Prefer $BASH_SOURCE over $0: It's more reliable, especially when your script might be sourced instead of executed.
Check exit codes for important operations: Don't just assume things worked. Check $? or use if statements.
Quote "$@" when passing arguments: This preserves spaces in filenames and arguments.

Avoid these mistakes

Don't hard-code paths: Never write /home/john/.config directly. Use variables so your script works for everyone.
Don't use $* by default: It merges all arguments into one string, which usually isn't what you want. Stick with "$@".
Don't assume $? sticks around: It changes with every command. Save it immediately if you need it later.
Don't forget to quote path variables: Unquoted variables break when paths have spaces.
Don't use $UID for permission checks: Use $EUID instead. That is what actually matters for access control.

References

Per-App Shell History for Bash

Mohammad Abu Mattar — Thu, 18 Dec 2025 00:00:00 GMT

Organize your Bash history per terminal app.

Ever jumped between iTerm2, Ghostty, and VS Code's terminal only to have your command history get all mixed up? This Bash snippet keeps things clean by creating separate history files for each terminal app you use.

The Problem

Mixed command history

When you use multiple terminal emulators (iTerm, Alacritty, Ghostty, VS Code, etc.), they all share the same shell history file by default. This means commands you ran in one app show up when you press the up arrow in another. It gets messy fast, especially when you use different terminals for different projects or contexts.

Context switching issues

Different terminals used for different purposes (work, personal, testing) end up with mixed command histories, which makes context hard to keep.

The Solution

Per-app history files

This Bash function detects which terminal application you're using and automatically assigns a dedicated history file for it. Apple Terminal and Ghostty on Linux get the default history, while everything else gets its own isolated file. It all happens on shell startup.

Automatic organization

TL;DR

Automatically creates separate history files for each terminal app.
Keeps Apple Terminal and Ghostty (Linux) on the default history.
Works with iTerm2, Alacritty, Warp, Kitty, and more.
Just add it to your ~/.bashrc or ~/.bash_profile and forget about it.

Implementation

Bash function setup

function set_app_history() {

  # Get terminal app name (fallback: Apple_Terminal)
  local app="${TERM_PROGRAM:-Apple_Terminal}"

  # Default history file
  HISTFILE="$HOME/.bash/.bash_history"

  # Ghostty on Linux / WSL → use default history
  if [[ "$app" == "Ghostty" && "$(uname -s)" == "Linux" ]]; then
    export HISTFILE
    return
  fi

  # Apple Terminal → use default history
  if [[ "$app" == "Apple_Terminal" ]]; then
    export HISTFILE
    return
  fi

  # Everything else → per-app history
  app="${app//[^A-Za-z0-9]/_}"
  HISTFILE="$HOME/.bash/.bash_history_${app}"
  export HISTFILE
}

set_app_history

Benefits and usage

Organization advantages

Why This Helps

This approach keeps your shell history organized without any manual work. You can switch between terminals without worrying about command pollution or losing context. It helps most if you use different terminals for work, personal projects, or testing, since each gets its own clean slate.

Managing history files

Bonus Tip

Want to see which history files you have? Run this:

ls -lh ~/.bash/.bash_history*

You'll see all your per-app history files. Each one is tied to a specific terminal emulator, so you can even back them up or clean them individually.

Community Discussion

Your history management

Your Turn

How do you organize your shell history? Do you use per-app separation or something else? Drop your setup below!

Alternative approaches

Share your favorite methods for keeping shell history organized across different terminal applications.

Per-App Shell History for Zsh

Mohammad Abu Mattar — Thu, 18 Dec 2025 00:00:00 GMT

Organize your shell history per terminal app.

Ever jumped between iTerm2, Ghostty, and VS Code's terminal only to have your command history get all mixed up? This Zsh snippet keeps things clean by creating separate history files for each terminal app you use.

The Problem

Mixed command history

Context switching issues

Different terminals used for different purposes (work, personal, testing) end up with mixed command histories, which makes context hard to keep.

The Solution

Per-app history files

This Zsh function detects which terminal application you're using and automatically assigns a dedicated history file for it. Apple Terminal and Ghostty on Linux get the default history, while everything else gets its own isolated file. It all happens on shell startup.

Automatic organization

TL;DR

Automatically creates separate history files for each terminal app.
Keeps Apple Terminal and Ghostty (Linux) on the default history.
Works with iTerm2, Alacritty, Warp, Kitty, and more.
Just add it to your ~/.zshrc and forget about it.

Implementation

Zsh function setup

function set_app_history() {

  # Get terminal app name (fallback: Apple_Terminal)
  local app=${TERM_PROGRAM:-Apple_Terminal}

  # Default history file
  HISTFILE="$HOME/.zsh/.zhistory"

  # Ghostty on Linux / WSL → use default history
  if [[ "$app" == "Ghostty" && "$(uname -s)" == "Linux" ]]; then
    export HISTFILE
    return
  fi

  # Apple Terminal → use default history
  if [[ "$app" == "Apple_Terminal" ]]; then
    export HISTFILE
    return
  fi

  # Everything else → per-app history
  app=${app//[^A-Za-z0-9]/_}
  HISTFILE="$HOME/.zsh/.zhistory_${app}"
  export HISTFILE
}

set_app_history

Benefits and usage

Organization advantages

Why This Helps

Managing history files

Bonus Tip

Want to see which history files you have? Run this:

ls -lh ~/.zsh/.zhistory*

You'll see all your per-app history files. Each one is tied to a specific terminal emulator, so you can even back them up or clean them individually.

Community Discussion

Your history management

Your Turn

How do you organize your shell history? Do you use per-app separation or something else? Drop your setup below!

Alternative approaches

Share your favorite methods for keeping shell history organized across different terminal applications.

Optimizing your python code with slots?

Mohammad Abu Mattar — Wed, 16 Jul 2025 00:00:00 GMT

Memory optimization with `slots`

Understanding the problem

Optimizing data models in big data workflows with `slots`

In big data and MLOps workflows, you often work with massive datasets where you create millions of objects to represent data points, features, or model predictions. Traditional Python classes spend extra memory on the hidden __dict__ attribute. That overhead turns into memory bottlenecks and slower processing, especially in large-scale data pipelines and machine learning models.

How `slots` works

__slots__ reduces the memory footprint of your objects. By defining __slots__, you tell Python to allocate a fixed memory space for a class's attributes instead of a dynamic __dict__. Objects get smaller and attribute access gets faster, which matters for high-performance computing tasks in MLOps and big data.

Real-world use case: MLOps data pipeline

Customer churn prediction example

A data pipeline for MLOps

Imagine you're building a data pipeline for a machine learning model that predicts customer churn. Your pipeline processes millions of customer records, and each record is represented by a Python object.

Using a regular class, each customer object would have a memory overhead from its __dict__. When you process millions of these objects, the cumulative memory usage grows large enough to crash your application or push you onto more expensive, higher-memory machines.

By using a class with __slots__, you can create a memory-efficient data model. This approach cuts memory consumption, so you process more data with the same resources and the pipeline runs faster. In MLOps that matters, because resource use drives both your cloud bill and how far the pipeline can scale.

Big data processing benefits

An MLOps data pipeline with `slots`

In an MLOps pipeline, data is often represented as objects for processing, feature engineering, and model training. When dealing with big data, memory efficiency is what keeps the job from crashing and the cloud bill low. Using __slots__ reduces the memory footprint of these data objects, which makes the pipeline more reliable and easier to scale.

The following full code snippet simulates a big data MLOps workflow where we process a large number of customer records. We will create two versions of a Customer data class: one with the default Python behavior and one optimized with __slots__. The code then compares the memory usage and the time taken to create a million instances of each.

Code implementation

Class definitions

import sys
import time
import random
import pandas as pd
from typing import List

# --- Part 1: Data Model Definitions ---

class Customer:
    """
    A regular Python class to represent a customer record.
    This class uses a default __dict__ to store attributes.
    """
    def __init__(self, customer_id: int, age: int, monthly_spend: float, churned: bool):
        self.customer_id = customer_id
        self.age = age
        self.monthly_spend = monthly_spend
        self.churned = churned

class OptimizedCustomer:
    """
    An optimized customer class using __slots__ for memory efficiency.
    This class explicitly defines its attributes, eliminating the __dict__ overhead.
    """
    __slots__ = ['customer_id', 'age', 'monthly_spend', 'churned']

    def __init__(self, customer_id: int, age: int, monthly_spend: float, churned: bool):
        self.customer_id = customer_id
        self.age = age
        self.monthly_spend = monthly_spend
        self.churned = churned

Data generation functions

# --- Part 2: Data Generation and Object Creation ---

def generate_customer_data(num_records: int) -> List[tuple]:
    """Generates a list of tuples representing raw customer data."""
    data = []
    for i in range(num_records):
        customer_id = i
        age = random.randint(20, 70)
        monthly_spend = round(random.uniform(25.0, 500.0), 2)
        churned = random.choice([True, False])
        data.append((customer_id, age, monthly_spend, churned))
    return data

def create_objects(data: List[tuple], class_type: type) -> List:
    """Creates a list of objects from raw data using the specified class."""
    return [class_type(*record) for record in data]

Performance testing

# --- Part 3: Performance Comparison ---

def run_performance_test(num_records: int):
    """
    Runs a performance test to compare memory and time for both classes.
    """
    print(f"--- Running performance test with {num_records:,} records ---")
    raw_data = generate_customer_data(num_records)

    # Test the regular class
    start_time_regular = time.time()
    regular_customers = create_objects(raw_data, Customer)
    end_time_regular = time.time()
    memory_regular = sum(sys.getsizeof(c) for c in regular_customers) + sys.getsizeof(regular_customers)

    # Test the optimized class
    start_time_slotted = time.time()
    slotted_customers = create_objects(raw_data, OptimizedCustomer)
    end_time_slotted = time.time()
    memory_slotted = sum(sys.getsizeof(c) for c in slotted_customers) + sys.getsizeof(slotted_customers)

    # Print results
    print("\nRegular Class Performance:")
    print(f"  - Total Memory: {memory_regular / (1024**2):.2f} MB")
    print(f"  - Time Taken: {end_time_regular - start_time_regular:.4f} seconds")

    print("\nSlotted Class Performance:")
    print(f"  - Total Memory: {memory_slotted / (1024**2):.2f} MB")
    print(f"  - Time Taken: {end_time_slotted - start_time_slotted:.4f} seconds")

    # Calculate and print the savings
    memory_saved_mb = (memory_regular - memory_slotted) / (1024**2)
    time_saved_s = (end_time_regular - start_time_slotted)

    print("\n--- Summary of Savings ---")
    print(f"Memory Saved: {memory_saved_mb:.2f} MB ({ (memory_saved_mb / (memory_regular / (1024**2))) * 100:.2f}%)")
    print(f"⏱Slotted Class Creation Time is faster by: {time_saved_s:.4f} seconds")

    # Optional: Demonstrate a simple MLOps task like converting to a DataFrame
    print("\n--- MLOps Task: Converting to a Pandas DataFrame ---")
    # This task is just for demonstration and doesn't show a direct slots benefit here
    df_regular = pd.DataFrame([c.__dict__ for c in regular_customers])
    df_slotted = pd.DataFrame([[c.customer_id, c.age, c.monthly_spend, c.churned] for c in slotted_customers],
                               columns=['customer_id', 'age', 'monthly_spend', 'churned'])
    print("Successfully converted both object lists to Pandas DataFrames.")
    print(f"DataFrame head from Slotted Class:\n{df_slotted.head()}")

# --- Part 4: Execution ---
if __name__ == "__main__":
    NUM_RECORDS = 1_000_000 # 1 million records
    run_performance_test(NUM_RECORDS)

Results and analysis

Performance comparison

This code snippet demonstrates how to optimize memory usage in Python by using __slots__ in a data-heavy application, specifically in an MLOps context. It compares the memory and performance of a regular class versus an optimized class with __slots__. On large datasets both memory use and creation time drop.

Practical benefits

The implementation shows measurable improvements in memory usage and object creation speed, which suits big data processing and MLOps workflows.

List S3 Buckets

Mohammad Abu Mattar — Wed, 25 Jun 2025 00:00:00 GMT

Overview

Multi-profile S3 management

Ever juggled multiple AWS accounts and needed a quick S3 bucket inventory across all of them? This Python script handles it.

Use case

Perfect for organizations managing multiple AWS accounts or developers working with different IAM roles and profiles.

The Problem

Multi-profile complexity

When you're working with multiple AWS accounts or IAM roles through different profiles, getting a consolidated view of your S3 buckets can be a hassle. Switching contexts or running commands repeatedly for each profile is slow and easy to get wrong. A full inventory or audit turns into a chore.

Manual process inefficiencies

Traditional approaches require manual switching between profiles and running separate commands for each account.

The Solution

Automated profile discovery

This Python script uses Boto3 to discover all your configured AWS CLI profiles. It then iterates through each profile, attempting to list its S3 buckets. It prints a profile-by-profile breakdown and handles errors like missing credentials or access denied on a single profile, so the run finishes even in complex setups.

Detailed error handling

Gracefully handles various error conditions including missing credentials, access denied, and network issues.

Features

Key capabilities

TL;DR

Lists S3 buckets across all configured AWS CLI profiles in one go.
Automatically discovers available AWS profiles.
Prints a per-profile listing of S3 buckets.
Handles errors for individual profiles (e.g., credential issues, access denied).

Output format

Organized display showing buckets grouped by AWS profile with clear visual separation.

Code implementation

Dependencies and setup

import boto3
from botocore.exceptions import NoCredentialsError, ClientError

Profile discovery function

def get_aws_profiles():
    """Get all configured AWS profiles from the AWS CLI configuration."""
    try:
        session = boto3.Session()
        return session.available_profiles
    except Exception as e:
        print(f"Error getting AWS profiles: {str(e)}")
        return []

Bucket listing function

def list_s3_buckets_for_profile(profile_name):
    """
    Lists all S3 buckets for a specific AWS profile.
    Returns a list of bucket names or an empty list if an error occurs.
    """
    buckets = []
    try:
        session = boto3.Session(profile_name=profile_name)
        s3_client = session.client('s3')
        response = s3_client.list_buckets()
        if response['Buckets']:
            for bucket in response['Buckets']:
                buckets.append(bucket['Name'])
    except NoCredentialsError:
        print(f"  Warning: No credentials found for profile '{profile_name}'. Skipping.")
    except ClientError as e:
        error_code = e.response.get("Error", {}).get("Code")
        error_message = e.response.get("Error", {}).get("Message")
        print(f"  Warning: AWS Client Error for profile '{profile_name}' ({error_code}): {error_message}. Skipping.")
    except Exception as e:
        print(f"  Warning: An unexpected error occurred for profile '{profile_name}': {e}. Skipping.")
    return buckets

Main display function

def display_all_s3_buckets_by_profile():
    """
    Fetches and displays S3 buckets for all configured AWS profiles.
    """
    profiles = get_aws_profiles()

    if not profiles:
        print("No AWS profiles found. Please configure your AWS CLI.")
        return

    print(f"\nChecking S3 Buckets across {len(profiles)} AWS Profiles:")
    print("-" * 40)

    for profile in sorted(profiles):
        print(f"\nProfile: {profile}")
        print("  S3 Buckets:")
        buckets = list_s3_buckets_for_profile(profile)
        if buckets:
            for bucket_name in buckets:
                print(f"    - {bucket_name}")
        else:
            print("    No buckets or inaccessible for this profile.")
        print("-" * 40)

    if not any(list_s3_buckets_for_profile(p) for p in profiles):
        print("\nNo S3 buckets found across any configured profiles or all were inaccessible.")


if __name__ == "__main__":
    display_all_s3_buckets_by_profile()

Benefits and usage

Operational advantages

Why This Helps

This script saves time for anyone managing multiple AWS environments. It covers S3 bucket auditing, inventory tasks, and compliance checks across your whole AWS footprint in a single run. You get one view without switching profiles by hand.

Use cases

Ideal for inventory management, compliance auditing, and multi-account AWS operations.

Community Discussion

Your S3 management approaches

Your Turn

How do you manage S3 bucket information in your Python projects? Any favorite Boto3 tricks for S3 you'd like to share?

Alternative tools

Share your preferred methods for managing S3 resources across multiple AWS accounts.

AWS Secrets Manager

Mohammad Abu Mattar — Wed, 18 Jun 2025 00:00:00 GMT

Loading secrets in a Node.js app without exposing them

If you're still storing API keys or database credentials in .env files or hardcoding them into your codebase, it's time for a better approach. Secrets should stay secret, especially in production.

The Problem

Managing sensitive values

Managing sensitive values across environments can get messy fast. Hardcoded secrets are risky, and .env files aren't ideal when you're working with teams or deploying to the cloud. It's easy to lose control over where that information ends up.

The Solution

AWS Secrets Manager overview

AWS Secrets Manager gives you a safe, centralized place to store secrets. You can fetch them at runtime using the AWS SDK, so you don't need to keep secrets on disk or in code. This snippet shows how to pull them securely in a Node.js app using SDK v3.

TL;DR

Use AWS Secrets Manager to securely access secrets in Node.js.
Avoid hardcoding secrets or using local .env files in production.
This snippet helps fetch secrets using the AWS SDK v3.

Code snippet (TypeScript)

import {
  SecretsManagerClient,
  GetSecretValueCommand,
  ResourceNotFoundException,
  SecretsManagerServiceException,
} from '@aws-sdk/client-secrets-manager';
import {config} from 'dotenv';

config();

interface CachedSecret {
  value: string;
  expiry: number;
}

const secretCache = new Map<string, CachedSecret>();

const DEFAULT_CACHE_TTL = 5 * 60 * 1000;

const defaultRegion = process.env.AWS_REGION;

/**
 * Custom error for when a secret is not found in AWS Secrets Manager.
 * This is thrown when the secret does not exist or cannot be accessed.
 */
class SecretNotFoundError extends Error {
  constructor(secretName: string) {
    super(`Secret "${secretName}" not found in AWS Secrets Manager.`);
    this.name = 'SecretNotFoundError';
  }
}

/**
 * Custom error for when a secret's value is invalid.
 * This can happen if the secret is binary, empty, or not a string.
 */
class InvalidSecretValueError extends Error {
  constructor(secretName: string) {
    super(
      `Secret "${secretName}" is binary or empty, or does not contain a string value.`,
    );
    this.name = 'InvalidSecretValueError';
  }
}

let secretsManagerClient: SecretsManagerClient | null = null;

/**
 * Initializes and returns a singleton SecretsManagerClient.
 * This prevents recreating the client on every `fetchSecret` call.
 * @param region - The AWS region to use for the client.
 * @returns An initialized SecretsManagerClient instance.
 */
function getSecretsManagerClient(region: string): SecretsManagerClient {
  if (!region) {
    throw new Error(
      'AWS_REGION is not defined. Please set it in your .env file or pass it as an argument.',
    );
  }
  if (!secretsManagerClient) {
    secretsManagerClient = new SecretsManagerClient({region});
  }
  return secretsManagerClient;
}

/**
 * Fetches a secret's string value from AWS Secrets Manager with optional caching.
 * @param secretName - The name or ARN of the secret.
 * @param options - Optional configuration for fetching the secret.
 * @param options.region - Overrides the default AWS region for this fetch operation.
 * @param options.cacheTTL - Time-to-live for the cached secret in milliseconds. Set to 0 to disable caching for this call.
 * @returns A promise that resolves to the secret string.
 * @throws {SecretNotFoundError} If the secret does not exist.
 * @throws {InvalidSecretValueError} If the secret's value is binary or empty.
 * @throws {Error} For other AWS SDK or network-related errors.
 */
export async function fetchSecret(
  secretName: string,
  options?: {region?: string; cacheTTL?: number},
): Promise<string> {
  const region = options?.region || defaultRegion;
  const cacheTTL =
    options?.cacheTTL !== undefined ? options.cacheTTL : DEFAULT_CACHE_TTL;

  if (!region) {
    throw new Error(
      "AWS_REGION is not defined. Ensure it's in your .env file or passed in options.",
    );
  }

  if (cacheTTL > 0) {
    const cached = secretCache.get(secretName);
    if (cached && Date.now() < cached.expiry) {
      return cached.value;
    }
  }

  const client = getSecretsManagerClient(region);

  try {
    const command = new GetSecretValueCommand({SecretId: secretName});
    const response = await client.send(command);

    if (response.SecretString) {
      const secretValue = response.SecretString;
      if (cacheTTL > 0) {
        secretCache.set(secretName, {
          value: secretValue,
          expiry: Date.now() + cacheTTL,
        });
      }
      return secretValue;
    } else if (response.SecretBinary) {
      throw new InvalidSecretValueError(secretName);
    } else {
      throw new InvalidSecretValueError(secretName);
    }
  } catch (error: any) {
    console.error(`[ERROR] Failed to fetch secret "${secretName}":`, error);
    if (error instanceof ResourceNotFoundException) {
      throw new SecretNotFoundError(secretName);
    } else if (error instanceof SecretsManagerServiceException) {
      throw new Error(
        `AWS Secrets Manager error for "${secretName}": ${error.message} (Code: ${error.name})`,
      );
    } else {
      throw new Error(
        `An unexpected error occurred while fetching secret "${secretName}": ${error.message}`,
      );
    }
  } finally {
    // Add any cleanup or finalization logic here.
    // For example, if you had an active connection or resource to close.
    // In this specific code, there isn't an obvious resource to clean up
    // within the fetchSecret function itself, as the client is a singleton
    // and caching is handled by a Map.
    // However, for demonstration, you could log something:
    console.log(`[INFO] Finished attempting to fetch secret "${secretName}".`);
  }
}

Why this matters

This setup helps keep your apps more secure and your secrets off disk. It fits well into cloud-native workflows, especially when you're using IAM roles or CI/CD pipelines.

Over to you

How do you handle secrets in your projects? Tried this approach before?

Check S3 Bucket Existence

Mohammad Abu Mattar — Wed, 11 Jun 2025 00:00:00 GMT

Quick Tip

Don't let your deployment blow up because of a missing S3 bucket. This Bash script lets you check if a bucket exists before anything fails.

The Problem

Missing bucket failures

The Problem

If your CI/CD pipeline tries to upload files or sync data to an S3 bucket that doesn't exist, things will break fast. You'll waste time, pipeline minutes, and possibly miss something obvious. A quick pre-check could prevent all that.

Impact on deployments

A deployment that fails on a missing S3 bucket wastes CI/CD time and delays the release.

The Solution

Bash script overview

The Fix

This script checks if an S3 bucket exists using the AWS CLI. It accepts --bucket or --b flags, gives color-coded logs, and adds a bit of fun to your terminal. Perfect for pre-checks in CI/CD jobs or local scripts.

Key features

TL;DR

Use this script to check if an S3 bucket exists before your pipeline tries to use it.
Accepts --bucket or --b to pass the bucket name.
Includes clean logs, color output, and fun formatting for better readability.

Script Implementation

Color definitions and setup

#!/usr/bin/env bash

# Color definitions
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

ASCII banner function

#######################################
# Prints a fun ASCII banner header.
# Globals:
#   CYAN
#   NC
# Arguments:
#   None
# Outputs:
#   Writes header text to stdout.
#######################################
function print_fun() {
  echo -e "${CYAN}"
  echo "╭──────────────────────────────╮"
  echo "│   S3 Bucket Checker Bash  │"
  echo "╰──────────────────────────────╯"
  echo -e "${NC}"
}

Logging function

#######################################
# Prints a log message with timestamp and log level.
# Globals:
#   CYAN, GREEN, RED, NC
# Arguments:
#   $1 - Log level (INFO, SUCCESS, ERROR)
#   $2 - Log message
# Outputs:
#   Writes formatted log message to stdout.
#######################################
function log() {
  local type="$1"
  local message="$2"
  local timestamp
  timestamp=$(date "+%Y-%m-%d %H:%M:%S")

  case "$type" in
    INFO)
      echo -e "${CYAN}[${timestamp}] [INFO]:${NC} $message"
      ;;
    SUCCESS)
      echo -e "${GREEN}[${timestamp}] [SUCCESS]:${NC} $message"
      ;;
    ERROR)
      echo -e "${RED}[${timestamp}] [ERROR]:${NC} $message"
      ;;
    *)
      echo -e "[${timestamp}] [LOG]: $message"
      ;;
  esac
}

Initialization and argument parsing

#######################################
# Initializes script by parsing CLI arguments.
# Exits with error if bucket name is not provided.
# Globals:
#   BUCKET_NAME, YELLOW, NC
# Arguments:
#   --bucket | --b <bucket-name>
# Outputs:
#   Sets BUCKET_NAME variable.
#######################################
function init() {
  while [[ "$#" -gt 0 ]]; do
    case "$1" in
      --bucket|--b)
        BUCKET_NAME="$2"
        shift 2
        ;;
      *)
        log ERROR "Unknown argument: $1"
        exit 1
        ;;
    esac
  done

  if [[ -z "$BUCKET_NAME" ]]; then
    log ERROR "Bucket name not provided. Use --bucket or --b to specify it."
    echo -e "${YELLOW}Example:${NC} ./check_bucket.sh --bucket my-bucket-name"
    exit 1
  fi
}

Bucket existence check

#######################################
# Checks if the specified S3 bucket exists.
# Globals:
#   BUCKET_NAME
# Outputs:
#   Success or error message to stdout.
# Returns:
#   0 if bucket exists, exits with 1 otherwise.
#######################################
function check_bucket_exists() {
  log INFO "Checking if bucket '$BUCKET_NAME' exists..."

  if aws s3api head-bucket --bucket "$BUCKET_NAME" 2>/dev/null; then
    log SUCCESS "Bucket '$BUCKET_NAME' exists and is accessible."
  else
    log ERROR "Bucket '$BUCKET_NAME' does not exist or access is denied."
    exit 1
  fi
}

Main function

#######################################
# Main function that runs the script.
# Calls init and check_bucket_exists in order.
# Arguments:
#   Passes all script arguments to init.
#######################################
function main() {
  print_fun
  init "$@"
  check_bucket_exists
}

main "$@"

Usage and Benefits

Integration with CI/CD

Why This Helps

Adding this check to your CI/CD steps helps avoid silly failures and gives you faster feedback if something's missing or misconfigured. It's quick to add and easy to reuse across projects.

Command line usage

Run the script with: ./check_bucket.sh --bucket my-bucket-name or ./check_bucket.sh --b my-bucket-name

Community Discussion

Your approaches

Your Turn

How do you handle S3 checks in your automation? Got a trick or tool you use often? Let's swap ideas.

Alternative solutions

Share your favorite methods for validating S3 bucket existence in different environments.