Skip to main content

Storage & Message Lifecycle

TinyMQ uses a Write-Ahead Log (WAL) strategy for disk persistence. Every message written to a topic is appended to an append-only .log file before being acknowledged. This guarantees durability without the complexity of a full database engine.

The PUT/ACK Cycle​

Every message in TinyMQ passes through a two-phase lifecycle:

  1. PUT — A publisher sends a message. The broker appends a PUT record to ./data/{topic}.log and holds the message in RAM.
  2. ACK — A consumer processes the message and acknowledges it. The broker appends an ACK record to the same log file and removes the message from RAM.

On broker restart, the log is replayed line by line. Any message with a PUT record but no corresponding ACK record is restored to RAM. This ensures zero message loss across restarts.

WAL File Format​

Each .log file is a newline-delimited JSON stream. Each line is a LogRecord:

// PUT record — written when a message is published
{"type":"PUT","message":{"id":"a1b2c3d4-...","topic":"orders","payload":"eyJ...","timestamp":"2026-06-18T10:00:00Z","retry_count":0},"timestamp":"2026-06-18T10:00:00Z"}

// ACK record — written when a message is acknowledged
{"type":"ACK","message_id":"a1b2c3d4-...","timestamp":"2026-06-18T10:01:30Z"}

The log grows with every operation. TinyMQ compacts it automatically on startup.

Auto-Compaction​

At startup, after replaying all logs to rebuild in-memory state, TinyMQ runs CompactLog() on every topic. This process:

  1. Reads the full .log file
  2. Builds a map of all PUT records, keyed by message ID
  3. Removes (deletes) any message ID that has a corresponding ACK
  4. Writes the remaining active messages to a .log.tmp file
  5. Atomically replaces the old log with the compacted version via os.Rename()

The result: the log file only ever contains unacknowledged messages after a restart. Disk usage stays bounded.

Lazy Initialization

A .log file is only created when the first message is published to a topic. Manually created topics (via the API or dashboard) do not create empty log files.

Message Structure​

type Message struct {
    ID         string     `json:"id"`
    Topic      string     `json:"topic"`
    Payload    []byte     `json:"payload"`
    Timestamp  time.Time  `json:"timestamp"`
    ExpiresAt  *time.Time `json:"expires_at,omitempty"`
    DeliverAt  *time.Time `json:"deliver_at,omitempty"`
    RetryCount int        `json:"retry_count"`
}
FieldDescription
IDCryptographically random UUIDv4, generated by crypto/rand (no external deps)
PayloadRaw bytes — JSON, plain text, binary; anything up to 2MB
ExpiresAtIf set, the message is silently dropped when a consumer tries to read it after this time
DeliverAtIf set, the message is hidden from consumers until this timestamp
RetryCountIncremented on each failed processing attempt; triggers DLQ at count ≥ 3

Safety Limits​

TinyMQ enforces two hard limits to protect the host environment. Both are enforced at the broker level and cannot be overridden at runtime.

Payload Size Cap — 2 MB​

Every publish endpoint wraps the request body with http.MaxBytesReader:

r.Body = http.MaxBytesReader(w, r.Body, 2<<20) // 2 MB

If a client sends a payload exceeding 2MB, the connection is aborted mid-stream and the broker returns:

HTTP 413 Request Entity Too Large

This prevents a single malicious or erroneous request from filling the broker's heap.

RAM Backpressure — 100,000 Messages​

Each topic has a hard ceiling of 100,000 messages held in RAM at once:

const MaxMessagesPerTopic = 100000

if len(t.Messages) >= MaxMessagesPerTopic {
    return errors.New("queue capacity reached (max 100,000 messages)")
}

If a queue fills up because no consumers are running, the broker returns:

HTTP 429 Too Many Requests

This is a deliberate backpressure mechanism. Instead of silently consuming all available RAM and crashing the host (OOM kill), TinyMQ signals the upstream producer that it must slow down.

In-Memory State and Locking​

TinyMQ minimizes lock contention with a two-level locking strategy:

  • Global sync.RWMutex — Used only to locate a topic in the map[string]*Topic map. Acquired and released immediately.
  • Per-topic sync.Mutex — Used for all message operations (append, extract, ACK). Disk I/O happens under per-topic lock, not the global lock.

This design means concurrent publishers on different topics never block each other.

Wildcard Routing​

TinyMQ supports wildcard consumption patterns using * as a glob character:

GET /consume/events.*  # Matches events.login, events.logout, events.purchase, etc.

Wildcard patterns are compiled to regexp.Regexp objects on first use and cached in b.compiledRegex to avoid re-compilation on every request.