If you choose at least one target for Serene Remembrance, and all of its targets are illegal when Serene Remembrance tries to resolve, it won’t resolve and none of its effects will happen. You won’t shuffle any library and Serene Remembrance will be put into its owner’s graveyard.

You may choose zero targets when you cast Serene Remembrance. If you do, you’ll shuffle Serene Remembrance into its owner’s library when it resolves.

If you give Serene Remembrance the flashback ability and cast it from your graveyard, it will be exiled instead of being shuffled into your library. Any other targets will still be shuffled into their owners’ library, however.