You can paradoxically build trust with cats by making a point of ignoring them. When they're around you just calmly meet their gaze for a second so they know you're aware of them, then turn back to whatever you were doing. With people or dogs this might be disrespectful but cats interpret it as a gesture of non-threatening-ness, you're announcing to them that you're aware of them and comfortable with them being around doing their thing.
If he approaches you on his own, you might just meet his gaze again, turn a little bit away again and set your hand down next to you where he can reach it if he wants. It's slow going and he probably won't ever completely shake old habits, but knowing how to "talk to" cats is important and not always intuitive.