I work in the assisted living field. There's frequently 1 nurse tending 40+ beds for 8 hours. If the next nurse is late, that's 1 nurse for 8+ hours until the next one shows. You can bet your ass that nurse isn't providing high quality medical advice 12 hours into a shift. An ai can take a non partial perspective and output a baseline level of advice to help the wheels moving.

How is a language based approach that completely abstracts away actual knowledge, and just tries to sound “good enough” any kind of useful in a medical workflow?

A LLM cross-referencing a list of symptoms against papers and books could be helpful for example. There is so much medical literature available these days and in so many languages that no one person can hope to gain a somewhat clear overview, much less keep up with all the new stuff coming out.

Of course this should only be in assistance to a trained medical professional, as all neural networks are prone to hallucinations. You should also double-check results of NNs that interpret medical images, they may straight-up hallucinate or just pick up on correlation instead of causation (say all the cancer images in your training set having a watermark from the same lab or equipment manufacturer).