When you are labeled as a tyrant for not committing tyranny, a criminal for not committing a crime, bad for doing good, guilty for not making a mistake, unfaithful for being faithful, and arrogant for being loving, the heart becomes very weary. So much so that one doesn't even feel like giving explanations anymore, so much so that one is neither ignored nor laughs, and sometimes one doesn't even feel pain; it's as if all sensations have ceased.
These are the times when some forget Allah, and some get closer to Allah. And the one who overcomes these trials by following Allah's commands is the patient one. 
At this time, be patient and remain silent when people's words are such that they should be thrown back in their faces.
And one good thing about patience is that when it comes, there is no desire for anyone else.