You are Crown of Doom’s owner if it started the game in your deck. Therefore, once you hand off the Crown, your opponents cannot give it back to you by activating its ability (although other effects, such as the one Zedruu the Greathearted’s ability creates, may do this).