Yup, just get SE. Mods need to be updated to work with the Anniversary Edition, and a lot of authors haven't done that yet. You'll need to downgrade SE, but there's an easy tool to do just that.

If you wanted to buy Special Edition for Steam, GMG (key seller with great reputation) has it on sale for roughly US$9.

You may see some people refer to the baseline Skyrim as the AE as when the AE released the SE got baseline upgrades and some creation club content is included by default (fishing, survival mode, Saints and Seducers, and Rare curios) so it has been changed since the SE release.

The paid AE DLC only adds the rest of the creation club content. There are some interesting things in it, but nothing mandatory. You can pick and choose what you want though, so you're not forced to run the entire pack. Just think of it as a bundle of paid mods that you'll have to check compatibility to your other mods.