If the creature targeted by Ancestral Vengeance's enchant creature ability is an illegal target as Ancestral Vengeance tries to resolve, Ancestral Vengeance won't resolve. It won't enter the battlefield, and its enters-the-battlefield ability won't trigger.