What would you do if someone grabbed your chest at work to see if they were "real"?

This guy at my work did the above mentioned, it made me feel dirty and hurt my feelings alot that this man would do this, however I kept it to myself because I am the kind of person who knows that employment is a big deal, and without it your life could be ruined. SO I choose not to say a word about it and let the weirdness fade from my head and all in all tryed to forget about it. Today one of the Asst's pulled me aside and said that I had made a racist comment, and I should watch my comments, I know the person that said I did was the SAME person that had grabbed me. I am the furthest thing from racist, and it hurt my feelings to think that someone would even think that, I guess i assumed and that is my fault that since this person says things about themselves as a joke it was ok to laugh at it and carry on a conv. But I guess not! My issue is How dare this person sit there and try and get me in trouble for saying something that was not meant in any way but jokingly after he knows that I could have got him fired on the spot for doing what he did. I almost want to say something but I really dont want to drag it on. I am afraid though that I am going to get in some sort of trouble for that and I think that is pretty dirty if you ask me. What would you do?