Not sure about JS, but from some cursory experimentation there don't seem to be any limitations on what can be done with the custom CSS. It seems to just load whatever CSS code you put there into a style block at the top of the finalized page source, so anything goes really. I wouldn't be surprised to see limitations on this in the future since this could obviously lead to some annoyances from making the entire page invisible to overlaying a seizure gif onto everything.

As far as documentation I don't know of anything official, but /m/kbinStyles seems like your best bet for asking questions, alongside using inspect element to identify the elements you want to style while consulting a CSS reference like w3schools. Also iirc, browser extensions like Stylus let you create custom themes for sites while previewing them on the fly which would be pretty helpful here. As long as you're not trying to do anything crazy complex you could probably learn what you need to style a magazine in an afternoon!

It's awesome that such open custom CSS and JS is allowed.

Next question, though; how do I disable that? :) I just came across /m/boycottreddit with customized CSS, and wow, I can't read it. It's got dark gray text on a black/dark orange background. It'd be nice to be able to disable it on a per-community basis in case I like most of the customizations but occasionally come across a community like this one that I'd like to read but can't.