I dont know why this confusion over mistress has arisen. He did not call her a mistress BUT he hinted at more than just being a gold digger. Please understand the following statements made by him -
1. Ok sanki baba apke samne apke beti ko hath nahi lagayega, lekin jab wo mera sath akele hoti hai tab? - he was hinting of a physical relationship
And the most explicit line was
2 . Kuch rishtey shadi ke bagair maza dete hain, aur wo maza lene ka haq sabko hai Pallavi ko bhi
He was talking about physical relationship.
He waa not just implying that she is a gold digger ..