I don't really bike much when it's wet, and never in mud, but the main downside of a slicker (but not slick) tire I've noticed is on steep gravel trails. I have a knobbly in front, so descending is ok, but ascending is definitely harder.
Mind that tread isn't everything, tire size, air pressure, rubber hardness etc. all play a part. A wider softer tire might not struggle as much even without tread. (Speaking of which, I should try letting out some air next time.)
Hi!
After a lot of thinking, reading reviews, and fondling gear at my local shop, I got a completely new setup recently. 8" board, 8" Thunder hollow lights, and 52 mm 93a Dragon wheels.
Having such a small, light, and low board feels really good (coming from 8.5 indys), and those wheels let me skate some spots that used to be too crusty for anything but my cruising wheels.
In fact the 93 durometer wheels (Powell Peralta Dragons and Spitfire Soft Sliders) are probably the most interesting development of the last few years. They handle rough ground and pebbles well, but still slide and bounce almost like hard wheels.
There's still some trade off, but I'd say they're perfect for skating actual street in any place that gets weather.