ProleWiki

901 readers

1 users here now

ProleWiki

A community related to the ProleWiki project.

Post in this community to request articles, provide suggestions and discuss ways to develop our project

founded 4 years ago

MODERATORS

felipeforte@lemmygrad.ml

ProleWiki needs to expand its storage! (lemmygrad.ml)

submitted 5 months ago* (last edited 5 months ago) by felipeforte@lemmygrad.ml to c/prolewiki@lemmygrad.ml

10 comments fedilink hide all child comments

To expand our operations, we will have to eventually scale our storage. To simply pay for a better VPS server is not scalable and very costly for us

The way forward is to pay for external storage I believe. Do you have any experiences with cloud storage you could share with us? We're probably opting for Amazon S3 based on what they are able to deliver to us

top 10 comments

sorted by: hot top controversial new old

[–] poo_22@lemmygrad.ml 19 points 5 months ago (2 children)

I'm not sure why you are considering giving money to Amazon, please don't, they can go fuck themselves. They lock you into using their services forever by having large egress fees for moving your data out. The popular open source NixOS project was hit with this recently and it highlighted this predatory issue: https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672

Do you really have more than a few TB of data? I would be surprised if you do. Just use a cheap VPS for storage: https://contabo.com/en/storage-vps/ has 2.4TB for 18$/month. Contabo has famously shitty support but they still have it and my experience with it has shown me that the cheap makes up for it.

I actually don't understand how a wiki with 22,000 pages of text and 4,000 files can be "very costly" to host. Please don't go the wikipedia route with the expenses.

[–] felipeforte@lemmygrad.ml 19 points 5 months ago

This is precisely why I asked for advice beforehand. Thank you!

I actually don’t understand how a wiki with 22,000 pages of text and 4,000 files can be “very costly” to host. Please don’t go the wikipedia route with the expenses.

It's not, we're fine, but we need to think long-term for growth. We serve more than just one language, and they are growing. Plus, we are thinking of hosting our own communication server, something like Matrix or better. And we need to have storage ready for that.

[–] poo_22@lemmygrad.ml 16 points 5 months ago

I also noticed you're not behind a CDN. There are free ones that will cache your content on servers around the world which serves your pages faster, saves you bandwidth and prevents you from getting DDOS'd. I would consider using a CDN.

[–] ksynwa@lemmygrad.ml 4 points 5 months ago

There are A LOT of S3 compatible object storage providers. I don't know about the market to suggest one but do look into them. AWS is extremely overpriced in comparison.

[–] johnrobbespiere@lemmygrad.ml 4 points 5 months ago

How much storage does Prolewiki need? I have experience with some forms of storage, but no provider comes to mind. Otoh for glacial backups, I could volunteer a section of my NAS.

[–] comrade_nomad@lemmygrad.ml 3 points 5 months ago (1 children)

I think it is safe to say we might need a bit more information to make a good suggestion here. Like how much storage do you currently need?

As said by others AWS(also GCP and Azure) can get really nasty with lock in(both egress fees and their specific offerings). All 3 are also US based so there is a chance they shut you down because they don't like the content.

Happy to discuss it a bit more and see where I can help.

[–] felipeforte@lemmygrad.ml 4 points 5 months ago (1 children)

Thanks for the reply!

We need to store 30 GB (safe estimate for 2-3 years) for publicly accessible images and documents. These would be constantly accessed by users. As far as I've measured, this could peak up to 2GiB/hour of network transfer

We also need to store up to hundreds of GBs (but less than a TiB in total) for files related to our server and backups, but these would have slow transfer rates, as they are not meant to be constantly touched. We plan on expanding towards other operations besides merely the wiki, so that would take extra space eventually

[–] comrade_nomad@lemmygrad.ml 3 points 5 months ago

Thanks for that additional info. I'll poke around a bit and see what I can find since you also mentioned hosting Matrix(I run an instance personally so will check usage info on mine). I'll fully admit most of my experience is with the big 3 cloud companies, at least professionally.

I'll also echo what another comrade mentioned in that a CDN would be helpful here.

[–] knfrmity@lemmygrad.ml 2 points 5 months ago

What about Hetzner? I'm not sure how they integrate with other VPS providers but their storage is very very cheap and bandwidth is good enough.

I use a storage box for remote backups and have never had an issue. They do take the storage box servers offline for maybe 3 hours per year all together for maintenance, which can be annoying but at least it's announced ahead of time. In nearly two years there has been one unplanned outage which was solved in minutes.

[–] bobs_guns@lemmygrad.ml 1 points 5 months ago

Consider Wasabi Hot Storage instead of S3. Significantly more cost effective.