Programming

First, my condolences.

Second, I think something you can do as "head of AI" is push back on the benchmarks your execs are expecting you to measure. They got them straight from whoever your AI vendor is, and if your executive team is even halfway competent they should understand "these metrics are designed by the vendor to make us spend more", but a lot of exec teams won't listen. Still, you should institute your own benchmarks around code quality and delivery speed and talk about them with the exec team even if you have to shoehorn them into the discussion.

The next and probably more important thing that comes to mind is managing how your devs use their new tools. They'll be able to churn out more lines of code than ever before and "complete" some features much more quickly. Your metrics should not incentivize this if your goal is code quality and stability. I don't really have much in the way of solutions (other than "manage expectations" and "set the narrative"), because that's about where in my "head of AI" career I got laid off. Once the numbers came in that we only needed 30% of our current staff they crunched the numbers and I was in the 70%.

Good luck. You're starting from a deficit because your management team probably already has entertained the thought of trimming some expensive devs from the payroll, and that's a tough thing to argue against.