Opensource

4626 readers

148 users here now

A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!

Credits

Icon base by Lorc under CC BY 3.0 with modifications to add a gradient

⠀

founded 2 years ago

MODERATORS

pylapp@programming.dev

I didn't want to use AI, so I made something that detects it! (lemmy.world)

submitted 1 week ago by fccview@lemmy.world to c/opensource@programming.dev

6 comments fedilink hide all child comments

Lately I have been using AI more and more in my codebase and that's been a bit of a hit and miss if you ask me.

I reckon it's an amazing tool that allows developers to truly optimise their workflow, however at times laziness take over and code reviews are not as frequent as they should be.

I really REALLY wanted to build something without using AI after having spent months and months getting a bit too complacent. And this is the first idea that came to mind. Of course to celebrate this occasion I went for a technology I hadn't used in almost a decade: jQuery!!!

You can find the repo here: https://github.com/fccview/ackchyually-ai
You can play around with the tool on fccview.github.io/ackchyually-ai

Few disclaimers:

I am a full stack javascript developer (with extensive knowledge of php and a very good grasp of shell). This means majority of the detection will be revolving around javascript, if you develop in a different language and you feel like you know AI patterns in that language PLEASE do create a pull request so we can make the detection even more accurate for other type of syntax!
The tool uses MY OWN metrics, this is doing what I usually do to see if something is (in my opinion) been made with AI and barely reviewed by a human. It's not gospel, it's not a scientific method, it's most definitely going to be flawed, so keep it with a grain of salt, it can always be improved and collaboration/community support will definitely help with it
I don't hate AI, on contrary I enjoy using it and I find value in not needing to "google it" whenever I hit a blocker, more times than not, AI has a valid solution that can be used, refined and applied properly. What I have mostly an issue with is people randomly generating code, not reviewing it and trying to sell you the moon when they barely understand the architecture of what they built
Please do not make this post a fight between pro/against AI individuals, that's not the aim of it

All the patterns I am using to identify possible AI code are here: https://github.com/fccview/ackchyually-ai/tree/main/data if you come up with more patterns/ideas and have more ideas on what to look for please reply here, open an issue and/or make a pull request, but at ONE CONDITION, do NOT use AI for this. It'd go against the core principle of this one pure little app I decided to build <3

top 6 comments

sorted by: hot top controversial new old

[–] PiraHxCx@lemmy.ml 9 points 1 week ago* (last edited 1 week ago) (2 children)

So I tested a bunch of AI-made .js scripts I'm using, they are all kinda "modular", AI didn't write them all at once, I added more parts as I needed, and I kinda heavily edited and formatted them to my liking manually, but the code itself is like 95% to 99% AI, and they all scored 1%, the highest was 6% because it had "emoji" (which is stuff I added manually, and funny thing, a few days ago I tried using AI to check if the code looked AI-made and they pointed emoji as something that didn't look like AI). Two scripts scored 0%...

However, one scored 100% and this one was the least AI-made of the scripts lol. Your tool basically complained about inline style (this code builds an ajax menu and as there would be different styles inside the same div and I wasn't going to use it anywhere else I went for inline style, and when I used AI to create some html templates before, in my experience, AI didn't like inline style at all, it would rather create a dozen of classes and ids and see which lines are using the same style to group them... I find it easier to just use inline). Again, it used "emoji" as an example that it was AI-made, but in this case I doubt AI would ever use it even on command, the "emoji" is actually one line using zalgo text and another with special characters like 🄾🄻🄳 🄼🄰🄽, I think it would need a very complex prompt to get AI to add those to a text inside a script successfully, imo this kinda stuff should count as not-AI-looking. So, another reason it scored 100% is because of the amount of "bar" mentioned, but in this case the menu is a sidebar, the script name is sidebar.js, so the name sidebar appears in various parts of the code and is a css class as well, and there is also a "seekbar" because the script also generates a music player... I don't get why every time the word "bar" is used in any context it counts as AI-looking, because in this case those were 100% my name choices haha - as were all the other things pointed as AI-looking, so funnily enough, it was the human part of the code that made it score 100% AI in your tool, else it would probably score 1% like the others :P

[–] fccview@lemmy.world 8 points 1 week ago (1 children)

wow thank you so much for the extensive testing! Yeah it's far from perfect, but can't stress it enough, the aim of the site is not to tell you how much is made by AI but how likely it is that something was reviewed by a human!

I'll most DEFINITELY sort out the bar situation, did not think that one through as much lol

I did put inline styling as a AI choice, most devs wouldn't do that, AI loves doing that haha

I'll try and fine tune things more and more, this is super helpful stuff!!

[–] PiraHxCx@lemmy.ml 2 points 1 week ago* (last edited 1 week ago)

Oh, that code I said scored 6% (now 5% because the bar in seekbar is not counting), it's code for a widget and it contains the css in it, and all the # in are counting as comments - but they are id selectors.
I'm using characters for buttons, ⏸ and ⏭ count as emoji, ▶ doesn't.

[–] PiraHxCx@lemmy.ml 4 points 1 week ago* (last edited 1 week ago)

Oh yeah, and this last code is extremely flawed, I was having trouble with loading order because each page accessed by this menu should also load other scripts and sometimes it wasn't working properly so I added a bunch of redundant stuff that I'd be embarrassed to show to an actual javascript developer. As I said, this was the least AI-made of the bunch, and how bad the code is should be proof humanity haha - even though I could hardly call it AI-assisted, it was AI-made and poorly human-edited.

[–] jasory@programming.dev 5 points 1 week ago (1 children)

I would look at stylometric analysis rather than just matching for certain phrases.

[–] fccview@lemmy.world 2 points 1 week ago

I wanted to keep this simple, but my next step would be something that may be considered similar to a stylometric analysis, just have no idea how i'll tackle it, can't remember last time I went on stackoverflow before working on this project lol

Something like checking patterns in length of variable names, patterns in maximum line lengths.

My worry with it is the tool starting to flag good coding standards lol

How would you apply stylometric analysis in a simple html/js/css project like this? As I don't want to compromise the nature of it