You should say nothing. If he still works at the place you used to work at I think its technically illegal for him to badmouth you when they call to verify employment.
Usually companies have an HR dept that will only say "Yes, Mr. Propaganda worked here between and " for this very reason. I'm not sure entirely what recourse you have for this but there's some liability involved on his part if this is the case.
If he doesn't work there and these aren't verification calls but you're instead listing him as a reference, leave this manager off your reference list since it sounds like its going more harm than good. Get a friend to pose as a manager to call.