Nightmare is the closest you can get to a player, and should really be the default player replacement yes. If you aren't good enough to beat a Nightmare AI, you would have gotten your ass handed to you by a human player in any case. Even if you have the natural gaming talent of a heavily sedated hobo, once you've played the game a while, you'll be better than a Nightmare AI.
Normal AI like a conveyor bringing you free gold and xp. Maybe it would be challenging if you had two broken arms.
For custom games, there should be an option to set the replacement AI difficulty level, but the default should still be Nightmare.