Unattaching Razor Boomerang is a cost to activate the equipped creature’s ability.

If the targeted creature is an illegal target by the time the equipped creature’s ability resolves, the entire ability doesn’t resolve. Razor Boomerang remains on the battlefield unattached.

The source of the damage is Razor Boomerang, not the equipped creature. However, the equipped creature’s ability is what targets the permanent or player. If Razor Boomerang is equipped to a red creature, for example, the ability couldn’t target a creature with protection from red. It could target a creature with protection from artifacts, but all the damage would be prevented.

If Razor Boomerang is no longer on the battlefield by time the equipped creature’s ability resolves, it’s not returned to its owner’s hand. The rest of the ability resolves as normal, so Razor Boomerang will still deal damage to the targeted creature.