Did Mercury and Uranus Have a Rough Youth After All?