Yes he is indeed affected. He reads her extremely well. Heck even Farhad reads her well, whereas her so-called family are just useless.
I think he loves to see her fighting spirit and when she is down, he doesn't like seeing her like that. It makes him uncomfortable. Today, after the women left, another person may have consoled her yet he took out his frustration of seeing her in that difficult spot but asking her why she always has so much drama in her life. He was quick to point out he sent the ladies away for his own reasons, not for her.
he liked fit lioness not wounded one
He has seen her lows and highs more than her he knows her well enough
Her eyes speaks volumes but the moment it changes he get shaken