I'm a big fan of sensible city behaviour - where a city doesn't just concentrate on a single megaproject for six hundred years whilst it doesn't even have a marketplace. This makes the "natural roads" concept attractive - a dirt track between locations within a certain number of squares that eventually becomes a full fledged road, with inns serving as waypoints for roads between distant cities.
But if that's not viable, I'm not opposed to a worker unit, creating a peasant workforce is absolutely fine by me (and gives tactical considerations to defend them), especially since it avoids the aforementioned "single city focus" problem that I hate.
But I'd hope to avoid certain things associated with them:
1: Optionally automatic.
Everything should have an automatic setting, from unit design to city build queues. Sometimes you have to micromanage, but otherwise, their movement shouldn't even be displayed and their existence should be ignored unless something's nearby and about to murder them. Likewise, units should have the automation to guard an area and engage anything within X squares so I don't have to babysit them militarily either.
2: Not road-specific.
There's more to life than roads. Peasants do so much more, from building fortifications for their lords to just plain farming the land. More improvements makes them more worth the inclusion. Inns are a start, but so are forboding towers of doom, temporary siege equipment (invasion towers, battering rams and the like for destroying fortifications would not be created in some city a kazillion miles away, they'd be contructed on-site).
3: Avoid periods of uselessness.
If my serfs have finished their building, they shouldn't sit there with nothing to do. I'm sure anyone who's played Civilization will know what I'm talking about with workers clustered in their cities with their thumbs up their jacksies waiting for "steel mongering level 2" to be researched.
That's dumb. Civ is dumb.
When there's nothing to create, workers should, without fuss or notification, go back to the nearest city, find the nearest, highest value tile with no other workers on it, and start farming/mining/lumberjacking/unicorn training to give a small bonus to production from that tile.
When a task becomes available, these workers all stop their auto-harvesting and perform whatever tasks they have to, then resume being helpful. Peasant populations don't just build improvements, they also staff them. A state too large for centralisation needs to have a sizeable peasant population to harvest in the regions between cities. The more territory, the more workers, so they should never run out of things to do, ever.
4: Not be defenceless.
This is pretty obvious. In a world where the average citizen can be abducted by faeries and stepped on by wandering giants at a moment's notice, these folks whose basic tools include scythes, hammers, saws and presumably high explosives will not hold up their hands because a regular recruit with a sharpened stick so much as glances at them.
5: Be revolting.
Taking care of the peasantry is a must. Overtaxation, letting too many worker units die (bandits, invaders or wandering monsters... or sacrificed to the dark gods for mana), or starve (and they always get food after all the military is fed), or just plain instigated by an enemy, and your peasants will revolt.
This can be something to encourage and cause in enemy nations where suddenly some of their worker units are turning neutral aligned and attacking everything in sight - including nearby worker units that they convert into more revolting peasants.
Not only does this make more sense than a cities randomly revolting and switching sides - a ruling class of a city rarely revolts, but is conquered by its own working classes. This also opens up several solutions in keeping with the Game Mechanics:
Diplomats might spend GoodWill (diplomatic resource is a silly name. Diplomacy generates goodwill, and using it gives you things in return, hence the name) to "buy" the revolting units - using their talents to "talk down" the uprising.
Conquerors can simply demolish them with their armies (which increases "fear", the anti-revolutionary drug that may bring other revolting peasants in line)
Heroes can talk them down, or take out the Rebel Leader to decapitate the movement.
Mages can drop a thermonuclear armageddon spell on them, or simply mind control the populace for the lulz.
Imperialists can spend gilder, just as a diplomat uses goodwill, essentially meeting the peasant's demands or bribing the leaders to stop their fussing.
Conquerors have the overall easiest time crushing revolutions (for valuable experience no less), but have the disadvantage of killing, rather than recovering, their workers.
Revolts should be infrequent, costly to cause for someone else, but potentially devastating if they manage to accumulate enough worker units to both cripple your economy and pose a viable military threat to your nation. Ideally a revolution should be a cause for interest, worry and excitement, not a minor uprising you quell every turn.
These would make workers hopefully interesting, valuable, and worthwhile additions rather than some tedious, arbitrary and largely irrelevant mechanic just as easily done with a city-window build task.