73
A NSFW detector with CoreML
(github.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 28 Aug 2023
73 points (91.0% liked)
Open Source
31021 readers
645 users here now
All about open source! Feel free to ask questions, and share news, and interesting stuff!
Useful Links
- Open Source Initiative
- Free Software Foundation
- Electronic Frontier Foundation
- Software Freedom Conservancy
- It's FOSS
- Android FOSS Apps Megathread
Rules
- Posts must be relevant to the open source ideology
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
- !libre_culture@lemmy.ml
- !libre_software@lemmy.ml
- !libre_hardware@lemmy.ml
- !linux@lemmy.ml
- !technology@lemmy.ml
Community icon from opensource.org, but we are not affiliated with them.
founded 5 years ago
MODERATORS
By definition you can't have some of these things open source, CSAM/NSFW detection needs to be closed source because people are constantly trying to get around it.
Security through obscurity doesn't work. These systems need to be actually robust, which is only trustworthy with open source
That is literally not the problem, it's not security. It's obfuscation on purpose so things can't be reverse engineered. I agree with you in most other cases, but this is one I don't. It's the same reason there aren't public hash lists of these vile images out there, because then the people out there will change them. Same with fuzzy hashing and other strategies, these lists and bits of code must remain private so they aren't tipped off to their stuff tripping the content.
This can't be a cat and mouse game all the time when it comes to CSAM, it must work for a while. So I'm fully on board with keeping it private while we can, it's the one area I am okay with doing that. If it's open bad actors will just immediately find a way to get around detection and all modes of knowing it will be obsolete until we find another way, and in that time we're waiting to find another way they're going around posting that shit everywhere, then it doesn't matter how open source Lemmy is, because all of our domains will be seized.
Because any detector has to be based on machine learning you can open source all code providing you keep model weights and training data private.
But there's a fundamental question here, that comes from Lemmy being federated. How can you give csam detecting code/binaries to every instance owner without trolls getting access to it?
Some instances will be run by trolls, and blackbox access is enough to create adversarial examples that will bypass the model, you don't need source code.
That discussion is happening, right now the prevailing idea is that it's an instance admin opt-in feature, where you can host it yourself or use a hosted tool elsewhere to prevent it. on top of that, instance admins should be allowed to block federating images, so things uploaded on other instances are not federated to us and instead those images are requested directly from your instance. That would help cut down on the spread of bad material, and if something was purged on the home instance it could be purged everywhere
Just chiming in here to say that this is very much like security through obscurity. In this context the "secure" part is being sure that the images you host are ok.
Bad actors using social engineering to get the banlist is much easier than using open source AI and collectivly fixing the bugs when the trolls manage to evade it. Its not that easy to get around image filters like this, and having to do wierd things to the pictures to be able to pass the filter barrier could be work enough to keep most trolls from posting. Using a central instance that filters all images is also not good, because now the person operating the service is now responsible for a large chunk of your images, creating a single point of failure in the fediverse(and something that could be monetised to our detriment) Closed source can not be the answer either because if someone breaks the filter, the community cant fix it, only the developer can. So either the dev team is online 24/7 or its paid, making hosting a fediverse server dependent on someones closed source product.
I do think however that disabling image federation should be an option. Turning image federation off for some server for a limited time could be a very effective tool against these kinds of attacks!