That would be a bit jarring. I suppose a woman could have a baritone voice, but it would be an exceptionally low-pitched voice for a woman, and certainly wouldn't pass without comment. If you wake in bed next to a woman with a baritone voice, it would be reasonable to think that it's not really a woman at all.