Yeah I tried Buildroot and Yocto and got frustrated. Buildroot did work, but it does a ton of stuff with a ton of not-very-well-documented options, and it definitely wasn't integrated into a proper build system. Like there's no way I'd trust any kind of incremental build.

I ended up doing it myself. Tbf I was just benchmarking kernel boots so I didn't need userspace or even real hardware. I only needed OpenSBI (a RISC-V firmware thing) and the kernel.

Frankly OpenSBI is a rather undocumented mess too. Incremental OpenSBI builds don't work, and I did try and contribute fixes but despite the repo being on GitHub they auto-close PRs and tell you to use git send email from the command line to a mailing list like it's the 90s. I shit you not. How to keep away contributors.

Anyway my conclusion is that the whole Linux OS building story is a shit show. If you only want a basic image with busybox etc. you can probably do it yourself and then you will actually understand it, but you will just be recreating Buildroot.