Ask Experienced Devs

1452 readers

1 users here now

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 2 years ago

MODERATORS

Ategon@programming.dev

How does indexing work at huge scales? (programming.dev)

submitted 4 days ago by ertai@programming.dev to c/ask_experienced_devs@programming.dev

4 comments fedilink hide all child comments

Youtube would be a prime example: I'm guessing the storage required for the metadata of all videos is too large to be stored on a single server, so how do they achieve millisecond-level performance on searches and handle millions of queries routinely?

What kind of infrastructure and technology is required for this?

Do you have any resources I could use to learn more on this subject?

you are viewing a single comment's thread
view the rest of the comments

[–] Shadow@lemmy.ca 3 points 4 days ago

Often it's distributed databases like Google spanner, amazon aurora, cockroach db, etc. Google has a public whitepaper on spanner you can read.

https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf