High Availability & Clustering

TinyMQ features a completely masterless, peer-to-peer (P2P) clustering architecture written from scratch. It does not require Apache ZooKeeper, etcd, or any external Raft implementation.

Node Roles

Every node in a TinyMQ cluster assumes one of three roles:

1. Follower: Accepts read requests directly. Proxies write requests (Publish, Ack) to the Leader via transparent HTTP proxying. Pings the cluster to ensure the leader is alive.
2. Candidate: If the leader goes down, a Follower becomes a Candidate and requests votes from peers.
3. Leader: Elected by quorum (N/2 + 1). Handles all writes, replicates them to Followers, and sends periodic heartbeats.

Leader Election

TinyMQ uses a Raft-inspired election algorithm. When a Follower stops receiving heartbeats for a randomized period (8–12 seconds), it transitions to a Candidate, increments its term, and requests votes.

The randomized 8–12s election timeout is specifically designed to give orchestrated environments like Kubernetes StatefulSets enough time to start all pods before the first election occurs.

Replication

TinyMQ uses Quorum-based ephemeral replication via parallel TCP connections.

When a message is published to the Leader:

1. The Leader writes the message to its own local WAL.
2. The Leader broadcasts a REPLICATE command to all Followers.
3. The Leader waits for REPLICATE_ACK from a majority (quorum).
4. Only when quorum is reached does the HTTP API return 201 Created to the client.

State Sync

When a node goes offline and rejoins, it sends a SYNC_REQ to the Leader. The Leader streams missing state via TCP, ensuring the joining node is completely caught up before it begins participating in quorum again.

TCP Protocol Commands

The cluster communicates over a dedicated TCP port (default 7901).

Command	Direction	Purpose
PING	Follower → Any	Liveness check
HEARTBEAT	Leader → Follower	Suppress elections, announce leadership
REPLICATE	Leader → Follower	Propagate new messages and ACKs
SYNC_REQ	Follower → Leader	Request state snapshot on boot
REQUEST_VOTE	Candidate → Any	Ask for leadership vote
BIND_GROUP	Leader → Follower	Replicate Consumer Group creation

Security (HMAC-SHA256)

When TINYMQ_CLUSTER_SECRET is set, every single TCP packet is signed using HMAC-SHA256. If a peer attempts to join the cluster or send a command with an invalid signature, the connection is instantly dropped.

The Importance of TINYMQ_CLUSTER_SELF

In containerized environments (Docker Bridge networks, Kubernetes), the IP a process binds to (0.0.0.0) is rarely the address other nodes use to reach it.

If Node A announces 0.0.0.0:7901 as its address, Node B will try to connect to 0.0.0.0 and fail.

To solve this, TINYMQ_CLUSTER_SELF allows a node to decouple its bind address from its advertised identity. For example, in Kubernetes, you set this to the pod's DNS name (tinymq-0.tinymq-headless:7901). The broker binds to 0.0.0.0, but tells all peers to reach it via the DNS name.

Configuration

Environment Variable	Default	Purpose
TINYMQ_CLUSTER_ADDR	0.0.0.0:7901	TCP bind address
TINYMQ_CLUSTER_SELF	empty	Advertised identity (e.g. tinymq-0:7901)
TINYMQ_CLUSTER_NODES	empty	Comma-separated peer list
TINYMQ_CLUSTER_SECRET	empty	HMAC-SHA256 shared secret
TINYMQ_CLUSTER_HTTP_ADVERTISE	empty	The HTTP address followers use to proxy to the leader
TINYMQ_CLUSTER_REPLICATE_TIMEOUT	500ms	Use 2s or 3s in Kubernetes due to DNS latency
TINYMQ_CLUSTER_LEADER	false	Set to true to force static leadership (disables elections)