There must be a huge variety of Indian accents given the large geographical area and bonkers number of people, but we (in America at least) only think of one stereotypical accent. I presume it annoys people to be lumped in with whichever group actually has that accent! Like Austrians probably don't love that the only Austrian voice anybody knows is Arnold Schwarzenegger, who comes from a very rural area and has that accent, apparently.