Yep, this works well. I used to have a cheap, old, secondhand ultralight notebook that I used for work, and I installed Lakka to a tiny bootable USB drive so that on evenings when we planned to gather at one colleague or another's house after work, I could just throw the drive and a couple of controllers in my bag and we could have a nice, clean, 100% emulation-focused system to game on. Even a 10-year-old laptop with Lakka should smash a Raspberry Pi 4's cost/performance ratio.

I usually plugged in power and a display, but of course as a laptop it was fine for using on the go, too. Not quite a handheld, but still very portable.

Another option is one of the cheaper ARM-based handhelds with a USB port and HDMI output, so you can still play on a big screen at home. I later got a Retroid 2 for this, which also worked well, but needed a bit more technical fiddling than the Lakka laptop, and couldn't emulate a few things at full speed. The Retroid 3 should be better, or there are other options such as Ambernic, Game Park, and PowKiddy.

Now, I just use my phone with a Bluetooth controller, and optionally an HDMI output dongle when I'm at home. If your phone doesn't support HDMI over USB-C natively (mine doesn't), look into a DisplayLink compatible USB dongle. If you check specs carefully you can even find some that work over USB 2.0 for older/cheaper phones. They have a free app for Android phones.