CrowdSuse

Depending on your needs you can also break it into a columnar format with some standard compression on top. This allows you to search individual fields without looking at the rest.

It also compress exceptionally well, and "rare" fields will be null in most records, so run length encoding will compress them to near zero

See fx parquet

Pet peeve: don't use string keys. It invites key collision errors. Use the fact Go supports structs as keys in maps. Safer and more efficient.

Not sure. But I think starting out early in the growth fase of the company and having a strong core of senior enginnees that can push for it and ensures everyone learns good practices