Rebuilding Redis in Go, Part 6: Replication

Section 6 is done! Replication. It started with setup that felt easy, then got pretty hard. WAIT was the hardest part of the project so far.

Also, this section put me at #377 on the CodeCrafters Go leaderboard!!

The Easy Part

The section opened with server setup work. I created a new CLICommand type for commands that configure the server and don't need a client connection, like setting the port. Then came psync and replconf, plus an info package carrying the Redis instance state:

var Redis = RedisInfo{
	Port: 6379,
	Replication: Replication{
		Role:             "master",
		MasterReplID:     "8371b4fb1155b71f4a04d3e1bc3e18c4a990aeeb",
		MasterReplOffset: 0,
		MasterHost:       "127.0.0.1",
		MasterPort:       6379,
	},
}

Hardcoded defaults, some of which will change and some that won't. There was also an empty RDB file to serve, which they gave me as base64, so I just decode it to binary and return it.

It felt like setup for something harder coming later, which turned out to be very true XD.

Writing to Replicas

Then came actual replication which involved keeping a list of replica connections and replaying write commands to all of them. The first version was simple. Commands got a Replicates flag so reads don't get sent for no reason, and this moved into goroutines so one slow replica can't hold up the client:

for _, replica := range replication.GetReplicas() {
	if !command.Replicates {
		continue
	}
	go func() {
		replicaCommand := command.ToReplicaCommand()
		replicaCommand.SendTo(replica)
	}()
}

ToReplicaCommand rebuilds the command in wire format, name and args as bulk strings, since the command already knows everything about itself.

Another Refactor

Somewhere in here I hit trouble with command processing and ended up refactoring the connection setup. Functions got split into clearer pieces across main and connection.go: dispatchCommand, handleMasterConnection, setupPort, parseArgs, checkReplicationRole.

I also called in an AI suggestion which pointed me at bufio.Reader, which replaced my old hardcoded 1024-byte buffer.

ACKs

"ACKs with no commands" was marked easy, but I had built replication as one-way, so the master sends, the replica receives, nobody talks back. ACKs need the replica to answer, so I had to rework that before I could continue.

It didn't need as big a redesign as I thought. I special-cased REPLCONF GETACK * in the master connection handler: when the master asks, the replica answers with its current offset. Everything else goes through normal dispatch.

The offset comes from byte counting. parseArgs returns how many bytes it consumed, and every command adds that to a running offset.

This also forced a distinction between handshake OK responses and one-way ACK messages. When a replica reports its offset, the master shouldn't reply at all, so I made a Nothing response type:

if args[0] == "ACK" {
	replicaOffset, err := strconv.Atoi(args[1])
	if err != nil {
		return resp.SimpleString("ERR invalid ACK offset"), nil
	}
	SetReplicaOffset(c, replicaOffset)
	return resp.Nothing(), nil
}

That way the code shows what the protocol means instead of needing a comment.

WAIT

Then came WAIT, and it gave me the most trouble of anything so far.

I first thought of it as "send a request and count replies." When it was really tracing the whole protocol, the client sends WAIT to the master, the master sends REPLCONF GETACK * to every replica, and each replica reports its replication offset with an ACK. So it's the offset, not the reply count, that tells you whether a replica caught up. It sounds a lot easier explaining it now that I've finished this lesson, but it took some time for me to wrap my head around.

A few edge cases made this clearer. With no writes before WAIT, the target offset is zero, and replicas initialized at zero already count as caught up. And the requested replica count is a threshold for how long to wait, not necessarily the number to return.

The waiting went from a busy-wait to synchronized waiting using a mutex and condition variable guard the replica offsets, and the waiter sleeps on the cond until enough replicas catch up or the deadline hits. Same sync.Cond pattern from blpop! I also changed the name of the cond variable to something that makes a little more sense to me, replicaStateChanged.

for {
	caughtUp := 0
	for _, c := range replicas {
		if replicaOffsets[c] >= targetOffset {
			caughtUp++
		}
	}

	if caughtUp >= required {
		return caughtUp
	}

	if !time.Now().Before(deadline) {
		return caughtUp
	}
	replicaStateChanged.Wait()
}

A timer broadcasts on the deadline so the waiter always wakes up.

What's Next

Replication tested everything I built so far. Most of it held up, and I refactored the parts that didn't.

Next up is RDB Persistence, which sounds like another fun section!