what are your system specs? at a low effort you should be getting a lot more FPS, what cli command are you using? but I guess it would be best for you to export to /tmp given enough ram and then go from there
EDIT: for context, when encoding libjxl I would do -distance 0 -effort 2
for lossless output
Have you considered using av1an? it supports vaporsynth which has a large amount of upscale and frame interpolation tools, AI or not. If your upscaler supports vapoursynth, it could be a lot better option.