If the artifact or enchantment is an illegal target as Unravel the Aether tries to resolve, it won’t resolve and none of its effects will happen. No library will be shuffled.

If the artifact or enchantment is a token, it will be shuffled into its owner’s library when Unravel the Aether resolves, then will cease to exist. For practical purposes, whatever item is representing the token should not actually be put into the library, though the library is still shuffled.