Sure, life would be easier if people didn't care about the ones who hurt them. But that's not how it works. You can't just tell yourself to feel one way or the other.

[He withdraws at last, not wanting to, but needing to look at Loki, get a feel for his expression. He keeps his hands on the man's shoulders, though, reluctant to break contact.]

...Come on. Keeping it all to yourself isn't going to help anything. Sit down and tell me what's going on.