Joseph wasn't a saint who was immune to the same temptations that we all struggle with.
He was, however, a tzadik - a righteous man - who was able to overcome his own potential for downfall by staying focused on who he was.
He saw himself as a son of the Hebrews - and stayed focused on the plan that G-d had assigned him to fulfill in life.