3216
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

top 50 comments
sorted by: hot top controversial new old
[-] LuckyLu@lemmy.world 320 points 1 year ago

Very impressed by how quickly action has been taken by this and other instances to patch the issue.

[-] trouser_mouse@lemmy.world 51 points 1 year ago

Very, seems like great work.

load more comments (27 replies)
[-] ThisIsMyLemmyLogin@lemmy.world 282 points 1 year ago

I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.

[-] riskable@programming.dev 27 points 1 year ago

Deleting hospital fees/debt is very dangerous... In many HUGE regions in the US there's only one hospital and if that hospital suddenly can't pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.

In fact, something like this already happened:

https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/

Hospitals are special in that they're often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).

Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.

Before that can happen though people need a backup payment method that doesn't go through their systems and no: Cash won't work (there's not enough in circulation and it's dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).

As soon as people have a widely accepted payment option that doesn't go through Visa and MasterCard's middlemen (e.g. First Data) then hackers can take their gloves off! Until then though... Let's keep the payment infrastructure working, OK? Thanks!

There's no limit to the amount of good deeds hackers can do though. So let's encourage that! For example, there's plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍

load more comments (2 replies)
load more comments (30 replies)
[-] Ahmed@lemmy.world 138 points 1 year ago

Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.

load more comments (10 replies)
[-] TheVampireSaga@lemmy.world 99 points 1 year ago

what steps are being taken to ensure it doesn't happen again? was any personal data compromised for users?

[-] ruud@lemmy.world 90 points 1 year ago

Good point, I'll update the post.

[-] TheVampireSaga@lemmy.world 50 points 1 year ago

Also I am curious, what's the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?

load more comments (14 replies)
[-] wazoobonkerbrain@lemmy.world 96 points 1 year ago

IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.

load more comments (2 replies)
[-] trouser_mouse@lemmy.world 88 points 1 year ago* (last edited 1 year ago)

First - really good summary and sounds like everyone is working hard.

Cross posting the below comment.

Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

[-] ruud@lemmy.world 48 points 1 year ago

Thanks for the info. We're looking into this.

load more comments (16 replies)
[-] HexesofVexes@lemmy.world 74 points 1 year ago

Thanks for letting us know - this is the kind of transparency that I wish the world had more of!

[-] MarekKnapek@lemmy.world 72 points 1 year ago

So what happened:

  • Someone posted a post.
  • The post contained some instruction to display custom emoji.
  • So far so good.
  • There is a bug in JavaScript (TypeScript) that runs on client's machine (arbitrary code execution?).
  • The attacker leveraged the bug to grab victim's JWT (cookie) when the victim visited the page with that post.
  • The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.

Am I right?

I'm old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

  • User provided content (post using custom emojis) caused havoc when processing (doesn't matter if on server or on client). This is lack of sanitization of user-provided-data.
  • JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
  • How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
  • The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Am I right? Correct me if I'm wrong.

Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.

Marek.

load more comments (17 replies)
[-] FartsWithAnAccent@lemmy.world 63 points 1 year ago

Damn, I go to bed early and I miss everything! Thanks for the quick resolution and transparent disclosure, this place is great!

load more comments (1 replies)
[-] alaxitoo@lemmy.world 61 points 1 year ago

Thank you for your work 🙏

[-] archy@lemmy.world 56 points 1 year ago

This is really good to see such transparency from admins

load more comments (1 replies)
[-] phillycodehound@lemmy.world 55 points 1 year ago
[-] Caminsky@lemmy.world 33 points 1 year ago* (last edited 1 year ago)

We need to learn more about the exploit. Lemmy essentially runs on donations, we all need to support this site. I have a lot of faith in lemmy. I am a reddit refugee and i think it is reminiscent of the early days of reddit and not the pos that it became.

load more comments (1 replies)
[-] dylanTheDeveloper@lemmy.world 51 points 1 year ago

Good thing we all use randomly generated passwords for every account and always remember to change them every few months.

load more comments (6 replies)
[-] bluemellophone@lemmy.world 42 points 1 year ago* (last edited 1 year ago)

Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

[-] ruud@lemmy.world 53 points 1 year ago

Good point. I did post about this on Mastodon @mwadmin@mastodon.world

load more comments (5 replies)
[-] nosut@lemmy.world 40 points 1 year ago* (last edited 1 year ago)

Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

EDIT: it has been added back to the block list.

load more comments (1 replies)
[-] SirPorthos@lemmy.world 39 points 1 year ago

So thats why MalwareBytes gave me this message yesterday.

load more comments (6 replies)
[-] AlmightySnoo@lemmy.world 38 points 1 year ago* (last edited 1 year ago)

the details of the vulnerability are already known now anyway since there's a fix that was proposed on the Lemmy GitHub so I don't think it will hurt others to talk about it

load more comments (13 replies)
[-] Sam1232188@lemmy.world 34 points 1 year ago

Thank the heavens the meme community stayed safe through this without my daily dose of cybersecurity memes idk how I would function ;)

load more comments (1 replies)
[-] gnarly@lemmy.world 31 points 1 year ago* (last edited 1 year ago)

Had to clear my browser catch to log in, Jerboa still shows as not logged in even after logging out which you do by clicking the hamburger menu then click the top banner to change/log out of accounts. This post is a test to see if my account works again via browser lol.

Edit: clearing app data/cache for Jerboa fixed the login issue.

load more comments (18 replies)
[-] Aceticon@lemmy.world 31 points 1 year ago* (last edited 1 year ago)

FYI: I had to clear my lemmy.world cookies in order to be able to successfully log back in.

(This was with Firefox)

(Edit: I also shift-clicked reload, which somebody pointed out does clean the cache for that page, so I also cleaned the cache).

load more comments (6 replies)
[-] NausetJF@lemmy.world 30 points 1 year ago

Hopefully with more attention on the source code scary hacks like this doesn’t happen again.

[-] AlmightySnoo@lemmy.world 29 points 1 year ago* (last edited 1 year ago)

Do we have any details on how Michelle's account was compromised? Right now in the GitHub issue about the vulnerability they're clueless about how the custom emoji exploit could be performed without first an already compromised admin account.

EDIT: yeah here's how: https://github.com/LemmyNet/lemmy-ui/issues/1895#issuecomment-1629326627

You do NOT need an admin account to do that. Any normal user could have done that.

load more comments (9 replies)
[-] CaptObvious@lemmy.world 29 points 1 year ago

One wonders how much spez paid for that hack.

load more comments (1 replies)
[-] Gili@lemmy.world 28 points 1 year ago* (last edited 1 year ago)

Took me a bit to realize I actually had to log out and log back in on Jerboa since it looked like I was still logged in but some interactions didn't work

load more comments (1 replies)
[-] MCForTheBest@lemmy.world 27 points 1 year ago

How do we know that this isn't a fake announcement as well, trying to give us a sense of security???

Just kidding, thanks for letting us know! Thank god I haven't been too active the last few days! Can't afford my credentials being leaked, maybe I should be proactive and change my password anyways.

load more comments (2 replies)
[-] SnowFoxx@lemmy.world 27 points 1 year ago

You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.

Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.

[-] PagingDoctorLove@lemmy.world 26 points 1 year ago

Can I ask some possibly dumb questions?

  1. What is JWT?
  2. Was any private user data compromised, and if so will users be informed?
  3. Is there anything regular users can do to avoid their data being compromised? For example, not accessing lemmy on certain web browsers?

Thank you!

[-] cley_faye@lemmy.world 41 points 1 year ago
  1. JWT stands for JSON Web Token. It's basically a way for a server (lemmy's) to put a piece of information in your browser in a way that makes sure it come from the server. It (usually) uses some form of digital signature. You can think of it as a note someone gave you with their signature, assuming said signature is very hard/impossible to counterfeit. The next time you see that person, they don't have to remember you, they just have to check the signature. If it is valid, anything written on the note is taken at face value.

When you connect to a site, there are a few steps to validate that you are who you say you are (identification and authentication). Something like inputing you login/password. Since it would be tedious to do that on every requests, the first time you give your login/password to the server (this is the simplified version, this exchange is a bit more complex usually) the server gives you that JWT. For every subsequent requests, your browser automatically send that JWT that is simple to handle but hard to counterfeit, and the server safely knows that you're whoever is written in that JWT.

  1. I assume there will be a post here when more details are known, or that this post itself will be updated. As with any online service, it's up to the service to decide if they want to communicate. (it may also be a legal requirements in some places to tell user when such an event occurs). Since we're talking about obtaining other user's authentication token including an admin, it is safe to assume that whatever an admin can see has leaked. This can range from basic user informations to more private stuff, although I am not familiar with the software behind lemmy. Note that this is a worst-case scenario; an admin impersonator could have access to anything an admin could see, it does not mean they immediately dumped everything. It depends on their motivation.

  2. Protection against this kind of stuff Compromission of the JWT can happen in many ways and I don't know which way was used. But if there's a flaw in the software used (the lemmy's client-side code, for example) there is not much you can do. JWT can leak through many things :

  • server compromission (out of your control)
  • client-side compromission (only happens when using a browser; applications that uses API should be less susceptible to that)
  • vulnerable extension: if you have browser extensions, they can easily peek into what's happening in any given page (that's their whole purpose). Malicious extensions, or extensions that allow outsider's some kind of control over them can leak data
  • browser vulnerability: keep your browser up-to-date, and (this is controversial) stick to a family of well-enough known browser. That obscure browser that have 20 users worldwide and is based on a three years old version of chromium is not the best thing to use
  • keep your data safe: only put the minimum required amount of data on any service. For lemmy, I assume an email address and your login/password is the bare minimum (well the email is already extra, but it's very convenient to have). Some services really likes to get everything they can out of you.

Basically, stay up to date and don't use shady stuff. Easy to say, I know.

load more comments (2 replies)
load more comments (2 replies)
load more comments
view more: next ›
this post was submitted on 10 Jul 2023
3216 points (99.3% liked)

Lemmy.World Announcements

28381 readers
4 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 1 year ago
MODERATORS