It was the best of times, it was the worst of times. That’s when I hit a performance mystery that sent me down a multi-day rabbit hole of adventure. I was writing some code to take some entries, append them into a fixed size in-memory buffer, and then flush that buffer to disk when it was full. The main bit of code looked a little something like this:

type Buffer struct {
	fh  *os.File
	n   uint
	buf [numEntries]Entry
}

func (b *Buffer) Append(ent Entry) error {
	if b.n < numEntries-1 {
		b.buf[b.n] = ent
		b.n++
		return nil
	}
	return b.appendSlow(ent)
}

with the idea being that when there’s space in the buffer, we just insert the entry and increment a counter, and when we’re full, it falls back to the slower path that writes to disk. Easy, right? Easy…

This post written jointly with JT Olds.

Infectious — The Reed-Solomon Forward Error Correcting library for Go

storj/infectious — Reed-Solomon forward error correcting library

Infectious is a high-performance forward error correcting library for the Go programming language. Error correcting codes in general are widely used for reliability and resilience, and even for decreasing latency and increasing throughput. Reed-Solomon specifically is used all the way from QR codes or barcodes to satellite communication and is why scratched CDs and DVDs still work. If you haven’t guessed yet, we use Reed-Solomon, too.