Based on this season and listening to Robin's graphic novel podcast, I wanted to offer some thoughts on how scary the walkers need to be.
IMHO, they have to be pretty close to the graphic novel walkers. Any of our able bodied people have to be able to beat a lone walker pretty much 10 times out of 10, and to have the advantage over even 2 or 3 when they have room to maneuver.
The first reason is that if you assumed that there's even a 10% or 5% chance the walkers win any particular fight, then our heroes have had enough fights that they should all be dead.
Instead, like the graphic novel, our guys are pretty much OK until they make a mistake - get careless, let their guard down, assume they're safe, or get surrounded and outnumbered.
Of course, sooner or later, everybody makes a mistake, which brings up the walkers as a metaphor of mortality. You can exercise, eat right, buckle your seat belt, and everything else, but sooner or later, death gets you, just like on TWD.