A person usually says “don’t judge me” when they know that what they are doing is wrong. So a person must be confident of the choices they are making to invite judgment on their life. 
Lydia, a newly converted Christian had recently won her family to Christ. She knew that Paul and Silas needed a place to stay. She said to the men “if ye have judged me to be faithful to the Lord, come into my house, and abide there.”
Are you centered in the Lord’s will so much so that you would welcome people to look at you, weigh the facts and decide who they think you are serving? 
Instead of the mindset of “that’s none of your business” maybe it should be “watch what Christ is doing in me”. 
Scripture taken from: Acts 16:14-15