1111
submitted 2 years ago* (last edited 2 years ago) by ernest@kbin.social to c/kbinMeta@kbin.social

Currently, on the main instance, people have created 40191 accounts (+214 marked as deleted). I don't know how many are active because I don't monitor it, but once again, I greet all of you here :) In recent days, the traffic on the website has been overwhelming. It's definitely too much for the basic docker-compose setup, primarily designed for development use. I was aware of the possible consequences of the situation happening on Reddit, but I assumed that most people would migrate to one of the Lemmy instances, which already has an established position. I hoped that a few stray enthusiasts would find their way to kbin ;)

The first step was to upscale the VPS to a higher version (66.91EUR). It quickly turned out that it wasn't enough. I had to enable CF protection just to keep the website responsive, but the response times were still very slow. At this stage, the instance was practically unusable. The next step was a full migration to a dedicated server (100EUR, the current hardware). It can be done relatively quickly, so it resulted in a 5-minute technical break. Despite the much higher parameters, it didn't get any better. It became clear that the problem didn't lie there. I'm really frustrated when it comes to server administration. That was the moment when I started looking for help. Or rather, it found me.

A couple days ago I wrote about how kbin qualified for the Fast Forward program. To be honest, I did it out of pure curiosity and completely forgot because a lot was happening during that time. During the biggest fire incident, Hannah ( @haubles ) reached out with a proposal to help. I outlined the situation (in short: the server is dying, I don't even know what I need, help! ;). She quickly connected us with Vlad ( @vvuksan ) and Renaud ( @renchap ). I was probably too tired because I don't know if the whole operation lasted 60 minutes or 6 hours, but after a series of precise questions and getting an understanding of the situation, the guys themselves adjusted the entire job. I love working with experts, and it's not often that you come across individuals so well-versed in the fediverse. Thanks to Hannah's kindness, we will be staying there a bit longer. Currently, fastly.com handles the caching layer and processes images. Hence those cool moving thumbnails ;)

Things were going well at that point. I could disable Cloudflare protection. Probably thanks to that, many of you are here today, and we got to know each other a bit better :) However, even then, when I tried to enable federation, the server would stop working.

Around the same time, Piotr ( @piotrsikora ), whom I already knew from the Polish fediverse, contacted me. He is the administrator of the Polish Mastodon instance pol.social, operates within the ftdl.pl foundation, and specializes in administering applications with a very similar tech stack. I made the decision to grant him server access. It only took him a few moments, and he came back to me with a few tips that allowed us to enable federation. In the following days, there was more of it, and we managed to reach the current level. I think it's not too bad.

Nevertheless, managing the instance has taken up about 60% or more of my time so far, which prevents me from fully focusing on current tasks. That's why I would like to collaborate with Piotr and hand over full care of the server to him. Piotr will also take care of the security side. Now I have to take this much more seriously. We still need to work out the terms of cooperation, but I want you to know the direction I intend to pursue.

We also need to migrate to a new environment because one server will sooner or later become insufficient. This time, I want to be prepared for it. This may be associated with transient issues with the website in the coming days.

The next two updates will still be about project funding (I still can't believe what happened) and moderation. The following ones will be more technical, with descriptions of changes and what contributors are doing on Codeberg. I would like to be here more often, but not as an admin, just as myself.

Thank you all for this.

P.S. In private messages, I also received numerous offers of help that I didn't even have a chance to read and respond to. You are the best!

top 50 comments
sorted by: hot top controversial new old
[-] Whirlgirl9@kbin.social 276 points 2 years ago

hi all, you can buy ernest a cup of coffee aka give him some cash for his incredibly hard work! https://www.buymeacoffee.com/kbin

[-] DrChickenbeer@kbin.social 70 points 2 years ago

Yassss I'm supporter 666๐Ÿค˜

[-] BigVault@kbin.social 14 points 2 years ago
[-] I_Miss_Daniel@kbin.social 53 points 2 years ago

That 100 Euro server also needs some coins inserted :)

[-] freebrick@kbin.social 19 points 2 years ago

Done! Thanks @ernest! This site is so cool!

[-] HipHoboHarold@kbin.social 14 points 2 years ago

Unfortunately someone just tried to use my bank account the other day, so I'm waiting for a new card to come in, but I got this bookmarked for when I do. I think it's worth the cost of a cup of coffee every now and then.

[-] aeternum@kbin.social 13 points 2 years ago

Ernest should set up a patreon. I'd pay a monthly sub to him for this great website.

It's hard ot believe it's only a month or two old.

load more comments (1 replies)
load more comments (5 replies)
[-] Nougat@kbin.social 71 points 2 years ago

I hoped that a few stray enthusiasts would find their way to kbin

They did. "A few" as a portion of the whole, where the whole is much larger than you ever expected it would be.

One thing I have learned in my own technical career is that you can't do everything. Even if there was enough time in a day for you to do everything, you can't know everything, at least not sufficiently enough to be most effective. You have to depend on other experts, and delegate to them what you are not qualified to do. This requires a pretty high level of trust, and makes it so that you need to develop people management skills.

Lots of technical people are short on people management skills. I don't know where you sit on that spectrum, but you may need to consider bringing on an "overseer," kind of like a project manager, to keep tabs on all of the technical resources involved - yourself included. This will help ensure that concerns are prioritized appropriately, and that communication and messaging about those priorities are consistent and clear.

I've been in that kind of position, and I take a bit of pride in my use of words. I'm happy to give any advice you like.

[-] bourbonmakesitbetter@kbin.social 18 points 2 years ago

I sense an opportunity here: managed hosting of Fediverse services. I don't mean a managed host where you can install/run the services. I mean a top-to-bottom setup, management, backup, upgrades, monitoring, etc. so the only thing you need to do is administer the community. I'd love to set up several Fediverse services for my local community, and I know there is an audience that would also love that, but I also know I do not want to invest the time it takes to manage the technical side of that.

Having easier to set up instances would help in relieving the pressure from the more popular current instances. We're starting to see those hosting options come on line for some of the services, but not fast enough for my liking.

load more comments (12 replies)
[-] haubles@fosstodon.org 56 points 2 years ago

@ernest @vvuksan @renchap @piotrsikora Thank you, Ernest ๐Ÿ’ž we're so glad we can help and support you, and the project.

[-] MeowdyPardner@kbin.social 41 points 2 years ago

It's cool to see renchap helping out here in addition to his work with mastodon.social! The amount of cross-pollination and coordination within the fediverse is so cool to see.

load more comments (1 replies)
[-] Saturdaycat@kbin.social 30 points 2 years ago

Thank you for the update @ernest ! Do you have a webpage for donations ?

[-] Badabinski@kbin.social 22 points 2 years ago

Hey @Ernest and @piotrsikora,

I haven't looked too closely at how kbin is architected yet, but would it benefit from horizontal scaling? I do full-time development of tooling to administrate very large k8s clusters for a company that you've probably interacted with today without knowing it. Not sure if k8s is the right orchestration system for you, but I'd be more than happy to provide some input on a potential migration to k8s (if kbin is a good fit there). I know there's a community on Matrix as well โ€” I'll try to reach out there too, although it may be a bit.

[-] VerifiablyMrWonka@kbin.social 19 points 2 years ago

If the post is anything to go by it's using the included "mostly for dev work only, mostly" docker-compose files. It would absolutely be able to be scaled out since at it's core it's just a webapp with workers. The app is already configured to use Redis for session storage so should be able to go super wide.

Only limitation is how performant you could make your postgres cluster.

[-] ernest@kbin.social 16 points 2 years ago
[-] piotrsikora@pol.social 14 points 2 years ago

Hi @Badabinski

K8S is one of option, but we decided to use some mix of bare-metal and docker swarm.

Almost everything is prepare to grow horizontally. Only (like always) problem is in database, and also we want to have flexible software that run on big cluster and small node without changes in code.

Give us few days, and after that we will show something ;)

@haubles @vvuksan @renchap @ernest

load more comments (4 replies)
[-] trent@kbin.social 18 points 2 years ago

Thank you ernest... and fastly... and welcome aboard Piotr!

OSPs make me so happy :)

[-] brandonsh@kbin.social 16 points 2 years ago

Thanks for the update, and keep up the amazing work!

[-] EROLoLICON@kbin.social 12 points 2 years ago

I read the whole thing with a smile on my face. So wholesome!

[-] s804@kbin.social 11 points 2 years ago

Thank you for creating this space for us :)

[-] Calcharger@kbin.social 10 points 2 years ago* (last edited 2 years ago)

I'm so confused. We only have 3500 accounts on Kbin? So we are miles and miles less than Lemmy?
edit No, I just need coffee

[-] lixus98@kbin.social 22 points 2 years ago
[-] Calcharger@kbin.social 9 points 2 years ago

I need coffee

[-] Sausage@kbin.social 10 points 2 years ago* (last edited 2 years ago)

Thanks for the update. Sounds like you're having an "exciting" time. You've done a great job so far and I hope that the additional help you are receiving means that you might be able to take some time off in the near future, for your own sake.

Edit: buy Ernest a coffee for those able to contribute

[-] thanksbrother@kbin.social 9 points 2 years ago

ITT: When your little hobby completely blows up in your face and takes over your life.

Thanks @ernest - we all appreciate it!

[-] Very_Bad_Janet@kbin.social 9 points 2 years ago

Kudos, @ernest. What you've put together in such a short time is amazing. I'm glad you're getting the help you need.

[-] huskola@kbin.social 8 points 2 years ago

Keep notes. This will make a great documentary some day!

I am not well versed with the way all this works so this might be a stupid question: Is there one central server in one location for this site? Could outside, remote server space be used to share the work?

load more comments (11 replies)
[-] lixus98@kbin.social 8 points 2 years ago

Nice work! I can't imagine what would happen if a hobby project of mine is suddenly used by 35k users, it would be insanely stressful, keep it up, we are loving it here.

[-] frasassi@kbin.social 8 points 2 years ago

@ernest Fantastic news and big thanks to all that are helping! @ernest I feel you will keep seeing several thousand users join in daily as the reddit shitshow continues to play out. There are now 10 apps being developed that will further support the transition especially as the July 1st shutdown takes place. Thatโ€™s probably when a much bigger wave of new users will come in. Is your infrastructure scalable to allow for this onslaught? Thank you and everyone else again for your hard work!

load more comments
view more: next โ€บ
this post was submitted on 21 Jun 2023
1111 points (100.0% liked)

/kbin meta

38 readers
1 users here now

Magazine dedicated to discussions about the kbin itself. Provide feedback, ask questions, suggest improvements, and engage in conversations related to the platform organization, policies, features, and community dynamics. ---- * Roadmap 2023 * m/kbinDevlog * m/kbinDesign

founded 2 years ago