18
25
[-] xtremeownage@lemmyonline.com 82 points 8 months ago

Don't make the same mistake reddit did, by assuming active users = engagement.

Look at reddit's stats, active users didn't drop very drastically when everyone left. However, engagement/comments dropped drastically.

195
submitted 10 months ago* (last edited 10 months ago) by xtremeownage@lemmyonline.com to c/technology@lemmy.world

Both CloudNordic and Azero said that they were working to rebuild customers’ web and email systems from scratch, albeit without their data.

Yea.... Don't bother. But, do expect to hear from my lawyers.....

CloudNordic said that it “had no knowledge that there was an infection.” CloudNordic and Azero are owned by Denmark-registered Certiqa Holding, which also owns Netquest, a provider of threat intelligence for telcos and governments.

Edit-

https://www.cloudnordic.com/

2
[-] xtremeownage@lemmyonline.com 58 points 10 months ago

Was a contractor for Walmart.

Got hired on as a lead dev, getting compensated 150k/yr.

2nd day, they told me I needed to switch contracts in order to stay on. New contract paid 50k salary.. with lots of required OT.

But, it's OK they said, you get benefits and PTO.

Fuck that.

[-] xtremeownage@lemmyonline.com 51 points 10 months ago

Guess, I am out of the loop.

After Linus blasted people using ad-blockers (fucking hypocrite), I uh, just blocked his channel.

[-] xtremeownage@lemmyonline.com 53 points 10 months ago

yea... every modern IDE makes it extremely obvious of the unicode character.....

SO.... yea...

If you really wanted to be evil, zero-width space is the worst.

https://symbl.cc/en/200B/

You don't see it.

423

Knock on wood, I have not used them in quite a while.

26
submitted 10 months ago* (last edited 10 months ago) by xtremeownage@lemmyonline.com to c/selfhosted@lemmy.world

My adventures in building out a ceph cluster for proxmox storage.

As a random note, my particular instance (lemmyonline.com) is hosted on that particular ceph cluster.

4
2

Well... That didn't last long...

15
7
244

I can't say for sure- but, there is a good chance I might have a problem.

The main picture attached to this post, is a pair of dual bifurcation cards, each with a pair of Samsung PM963 1T enterprise NVMes.

It is going into my r730XD. Which... is getting pretty full. This will fill up the last empty PCIe slots.

But, knock on wood, My r730XD supports bifurcation! LOTS of Bifurcation.

As a result, it now has more HDDs, and NVMes then I can count.

What's the problem you ask? Well. That is just one of the many servers I have laying around here, all completely filled with NVMe and SATA SSDs....

Figured I would share. Seeing a bunch of SSDs is always a pretty sight.

And- as of two hours ago, my particular lemmy instance was migrated to these new NVMes completely transparently too.

[-] xtremeownage@lemmyonline.com 77 points 11 months ago

I mean... as a software developer, Sorry, I will not be returning to the office.

You need me, more than I need you. The market is HOT right now.

Companies will learn, the hard way.

[-] xtremeownage@lemmyonline.com 53 points 11 months ago* (last edited 11 months ago)

As someone who hosts a small instance, nothing massive....

Honestly, nothing.

The only benefit I get- is full control over weather my instance is up or two. I know for absolute certain that my instance isn't going to randomly shutdown, and not come back online.

I also have the benefit of having a lot of control over how fast my instance is, and performance optimizations as needed to make it perform as I would like. As such, for me, the performance is outstanding.

With that said,

Basically everything else is downsides.

Having to proactively moderate content originating from your server, is a drag. The moderation tools in Lemmy are absolute dog-shit. Your only option here is to use either 3rd party tools (lemmy-helper), or to just run database queries.

PictRS just keeps growing and growing. pictures gets synced to every instance, and those take up room. Lots of room. PictRS has even less moderation tools then lemmy. If you want to make sure your user aren't uploading illicit/illegal content, is a major pain in the ass. My solution was to run a few scripts to fetch all of the content, and just run it through some AI scanning software to attempt to detect bad content. But, still, a pain in the ass.

Those attacks you read about here on lemmy world. Those happen to our smaller instances too. Every time you hear @ruud@lemmy.world doing an update here- we are also working on plans for updating the instance. Granted- my small user base makes these upgrades much easier and faster. But- we will have to do these updates. (At least on the plus side, my instances isn't constantly under a DOS attack, due to a disgruntled member, or due to a pissed off instance which was defederated)

And, lastly, one downside of lemmy- things don't really go away or get cleaned up. Your database and storage will continue to grow and grow, and grow. Again, to restate, There are basically no moderation or administration tools included with lemmy. You can see reports. You can ban users. And, you can delete posts. Thats about it.

There isn't an easy way to even list users, comments, posts, or activity happening on your instances.... through lemmy itself.

On top of those other issues, lemmy is very chatty, network wise.

Here are the incoming stats, from my "small" instance.

In terms of outgoing, it's very chatty there too. You will find all sorts of weird and random outbound DNS records.

tldr; Its prob not worth hosting your own instance, unless you just really like playing around with infrastructure, networking, databases, and digging through application issues.

Personally though- I enjoy the challenge, and that is one reason I keep doing it.

30

So, last month, my kubernetes cluster decided to literally eat shit while I was out on a work conference.

When I returned, I decided to try something a tad different, by rolling out proxmox to all of my servers.

Well, I am a huge fan of hyper-converged, and clustered architectures for my home network / lab, so, I decided to give ceph another try.

I have previously used it in the past with relative success with Kubernetes (via rook/ceph), and currently leverage longhorn.

Cluster Details

  1. Kube01 - Optiplex SFF
  • i7-8700 / 32G DDR4
  • 1T Samsung 980 NVMe
  • 128G KIOXIA NVMe (Boot disk)
  • 512G Sata SSD
  • 10G via ConnectX-3
  1. Kube02 - R730XD
  • 2x E5-2697a v4 (32c / 64t)
  • 256G DDR4
  • 128T of spinning disk.
  • 2x 1T 970 evo
  • 2x 1T 970 evo plus
  • A few more NVMes, and Sata
  • Nvidia Tesla P4 GPU.
  • 2x Google Coral TPU
  • 10G intel networking
  1. Kube05 - HP z240
  • i5-6500 / 28G ram
  • 2T Samsung 970 Evo plus NVMe
  • 512G Samsung boot NVMe
  • 10G via ConnectX-3
  1. Kube06 - Optiplex Micro
  • i7-6700 / 16G DDR4
  • Liteon 256G Sata SSD (boot)
  • 1T Samsung 980

Attempt number one.

I installed and configured ceph, using Kube01, and Kube05.

I used a mixture of 5x 970 evo / 970 evo plus / 980 NVMe drives, and expected it to work pretty decently.

It didn't. The IO was so bad, it was causing my servers to crash.

I ended up removing ceph, and using LVM / ZFS for the time being.

Here are some benchmarks I found online:

https://docs.google.com/spreadsheets/d/1E9-eXjzsKboiCCX-0u0r5fAjjufLKayaut_FOPxYZjc/edit#gid=0

https://www.proxmox.com/images/download/pve/docs/Proxmox-VE_Ceph-Benchmark-202009-rev2.pdf

The TLDR; after lots of research- Don't use consumer SSDs. Only use enterprise SSDs.

Attempt / Experiment Number 2.

I ended up ordering 5x 1T Samsung PM863a enterprise sata drives.

After, reinstalling ceph, I put three of the drives into kube05, and one more into kube01 (no ports / power for adding more then a single sata disk...).

And- put the cluster together. At first, performance wasn't great.... (but, was still 10x the performance of the first attempt!). But, after updating the crush map to set the failure domain to OSD rather then host, performance picked up quite dramatically.

This- is due to the current imbalance of storage/host. Kube05 has 3T of drives, Kube01 has 1T. No storage elsewhere.

BUT.... since this was a very successful test, and it was able to deliver enough IOPs to run my I/O heavy kubernetes workloads.... I decided to take it up another step.

A few notes-

Can you guess which drive is the samsung 980 EVO, and which drives are enterprise SATA SSDs? (look at the latency column)

Future - Attempt #3

The next goal, is to properly distribute OSDs.

Since, I am maxed out on the number of 2.5" SATA drives I can deploy... I picked up some NVMe.

5x 1T Samsung PM963 M.2 NVMe.

I picked up a pair of dual-spot half-height bifurcation cards for Kube02. This will allow me to place 4 of these into it, with dedicated bandwidth to the CPU.

The remaining one, will be placed inside of Kube01, to replace the 1T samsung 980 NVMe.

This should give me a pretty decent distribution of data, and with all enterprise drives, it should deliver pretty acceptable performance.

More to come....

[-] xtremeownage@lemmyonline.com 58 points 11 months ago

Honestly.

Stackoverflow is a horrible place to ask anything.

I have had 100% legit, well documented questions, closed as duplicate of unrelated other question.

Its... honestly, just not a friendly place to go. Full of a bunch of assholes....

Most of the answers actually suck too. Many times, you will find the correct answer downvoted, and incorrect or bad answers upvoted.

[-] xtremeownage@lemmyonline.com 60 points 11 months ago

I give credit to the verge- They have been doing a fantastic job of grabbing the details for the current reddit issues, and doing a fine job of reporting in a non-clickbaity way.

[-] xtremeownage@lemmyonline.com 69 points 11 months ago

Nah, fuck that.

Until reddit releases their IPO, all attempts at "peace" are secretly just trying to keep the boat floating

[-] xtremeownage@lemmyonline.com 130 points 1 year ago* (last edited 1 year ago)

Sigh....

All of those ideas are bad.

  1. Captchas are already pretty weak to combat bots. It's why recaptcha and others were invented. The people who run bots, spend lots of money for their bots to.... bot. They have accessed to quite advanced modules for decoding captchas. As well, they pay kids in india and africa pennies to just create accounts on websites.

I am not saying captchas are completely useless, they do block the lowest hanging fruit currently. That- being most of the script kiddies.

  1. Email domain filters.

Issue number one, has already been covered below/above by others. You can use a single gmail account, to basically register an unlimited number of accounts.

Issue number two. Spammers LOVE to use office 365 for spamming. Most of the spam I find, actually comes from *.onmicrosoft.com inboxes. its quick for them to spin it up on a trial, and by the time the trial is over, they have moved to another inbox.

  1. Autoblocking federation for servers who don't follow the above two broken rules

This is how you destroy the platform. When you block legitimate users, the users will think the platform is broken. Because, none of their comments are working. They can't see posts properly.

They don't know this is due to admins defederating servers. All they see, is broken content.

At this time, your best option is for admin approvals, combined with keeping tabs on users.

If you notice an instance is offering spammers. Lets- use my instance for example- I have my contact information right on the side-bar, If you notice there is spam, WORK WITH US, and we will help resolve this issue.

I review my reports. I review spam on my instance. None of us are going to be perfect.

There are very intelligent people who make lots of money creating "bots" and "spam". NOBODY is going to stop all of it.

The only way to resolve this, is to work together, to identify problems, and take action.

Nuking every server that doesn't have captcha enabled, is just going to piss off the users, and ruin this movement.

One possible thing that might help-

Is just to be able to have an easy listing of registered users in a server. I noticed- that actually... doesn't appear to be easily accessible, without hitting rest apis or querying the database.

[-] xtremeownage@lemmyonline.com 58 points 1 year ago* (last edited 1 year ago)

It doesn't.

If you are on my instance, for example, its hosted out of my own pocket.

I have a TON of spare compute resources laying around, and I am more than happy to donate them to this cause.

Edit- lets also be perfectly honest- hosting lemmy costs FAR less then it costs me to host plex for friends/family.

[-] xtremeownage@lemmyonline.com 52 points 1 year ago
{ "type": "comment response", "message", "I too, am certainly a human, and not a robot"}
view more: next ›

xtremeownage

joined 1 year ago
MODERATOR OF