If the target creature is an illegal target when the ability tries to resolve, it won’t resolve and none of its effects will happen. No creature will leave the battlefield and no library will be shuffled.