We're now using our own autonomous system and IP space for 3 of our networks. We run 2 entirely separate anycast DNS networks for our authoritative DNS and have a simpler unicast setup on a bare metal server at Xenyth which we'll be using for more soon.
Our ns1 network has 11 locations on Vultr (Piscataway, Miami, Los Angeles, Seattle, São Paulo, London, Frankfurt, Singapore, Mumbai and Tokyo).
Our ns2 network has 8 locations on Misaka.io (Ashburn, Miami, San Jose, Seattle, London, Berlin, Singapore and Tokyo).
Vultr and Misaka.io both have very good transit and peering for anycast due to having matching transit providers within regions and globally.
Both anycast networks needed a lot of configuration with BGP communities for traffic engineering and are working very well.
Our anycast networks are deployed with 2x IPv4 /24 obtained we quickly obtained for free from ARIN via NRPM 4.10 + NRPM 4.5.
We could use our own IPv6 space everywhere we have BGP if we wanted to do that since we have a /36 which can be expanded into more space reserved for us.
ARIN has approved our request for an IPv4 /22 via their waitlist but it will take around 18 to 36 months for the waitlist to progress to our request. For now, we're using an IPv4 /24 loaned to us for free by a Romanian LIR supporting GrapheneOS for our unicast Toronto IP space.
Our current bare metal server at Xenyth is sponsored by them and used as an update mirror which is using our IP space. However, our main use case for the IP space in Toronto is for our mail server which we're planning to host on-premises and tunnel the traffic through Xenyth.
Xenyth has support for routing to multiple servers announcing the same publicly routable IP space by announcing smaller blocks from specific servers so we can also pay for additional Xenyth bare metal servers or VPS instances. We'll likely be using it a fair bit in the future.
Our plan for our IPv4 /22 from the ARIN waitlist is deploying a single /24 in each of Toronto, Miami, Los Angeles and Seattle. Once we have a /22 deployed for North America, we'll qualify for getting out-of-region space on ARIN via the waitlist or transfers for Europe, Asia, etc.
The interesting parts of our BGP setup can be seen in https://github.com/GrapheneOS/ns1.grapheneos.org where we have our BGP community configuration for each ns1/ns2 location along with our setup for region steering via GeoDNS + anycast server location and failover via health checks from our DNS servers.