You're not crazy at all. And I think Civil War buffs will support you in your idea, so long as you are generally faithful to the times and the unit. Just make it clear that this is fiction. Historical fiction. Not history. This does not mean you won't get a grouse or two along the way from a purist ("His rifle would have had such-and-such a sight on it, not the one you've got."), but you'll never satisfy the most niggling, even if you would write a straight, non-fiction history. In other words, stay as faithful as you can, and then take whatever liberties you need, would be my advice.
I've got one historical fantasy out and I'm working on my next, so I know the issues you're dealing with. Good luck!
I've got one historical fantasy out and I'm working on my next, so I know the issues you're dealing with. Good luck!