You'd probably want to use a Canny or SoftEdge ControlNet to get it to match your face contours, and possibly use img2img at low denoising strength with loopback (repeats img2img on the output of the previous run, for however many loops you specify). The latter is always a good way to change the visual style of an image without changing the content too much.

You could also use a reference-only ControlNet to get it to stay more consistent, but you'll usually have to set it to "my prompt is more important" or it'll seriously mess with the brightness of your output image (near-black or near-white)

Inpaint?

Could try a Lora like this https://civitai.com/models/20832/kbeau-korean-beauty-lora and put in anime in the negative prompt