Yeah, the "Mine" part of Minecraft was always rather lame. Because the worlds are so insanely flat, caves have no real verticality, so it's hardly worth descending them to get to deeper levels. As such, you also don't need to secure a path through them with ladders, minecarts or intermediate bases.
The monsters are the only interesting part, but if nothing interesting happens while fighting those, then it's going to be "off-camera".
Well, or you could save yourself the trouble and just dig a staircase to diamond level.
Or you could save yourself the trouble and just cheat the resources.