If you're doing casual hookups that's going to be collateral damage a lot of the time. Just don't do repeats with guys like that and you're pretty guilt free. In the scope of guys sleeping around, you being just one of the guys that gets with him is not going to be the thing that breaks things.