I'm working on something similar in swift and started my project with the assumption that I'll need different API classes for every type of "instance" I want to gather data from (mastadon, lemmy, kbin, etc) and custom server model objects for each instance (as while some properties might be standard across all instances there will probably be some that are specific to an instance(s)).

Then I'll need those objects to be wrapped in a unified protocol to access and display them. Certainly not ideal, and does seem counterintuitive to the idea of the fedi-verse; but hopefully something for activity pub to work towards.