I was spending way too much time hanging out backstage in theatres when I started to take writing seriously, so I began with playscripts, scribbling down notes while listening to directors and actors chopping text around to make it flow. Watching a professional playwright make edits during rehearsals is an exercise I'd recommend to anyone who has the chance. When actors hit a bad line, they'll simply change it, and the playwright will often incorporate this into the final script.
In a novel there's no buffer between your words and the audience: if you can't make the dialogue sound natural, no-one else is going to do it for you. "Natural," though, is a misnomer. As an academic I've studied real dialogue extensively, and that's really not what you want to aim for. Let's take a closer look.
Classic "Bad Dialogue"
Let's start with a short exchange from the beginning of Dan Brown's The DaVinci Code, a book which is panned for its writing style about as often as it's praised (usually by different people) for its gripping plot.
The albino drew a pistol from his coat and aimed the barrel through the bars, directly at the curator. "You should not have run." His accent was not easy to place. "Now tell me where it is."
"I told you already," the curator stammered, kneeling defenseless on the floor of the gallery. "I have no idea what you are talking about!"
The albino, we're told, has an accent that's hard to place, which gives us chance to excuse him for almost any style in which Brown may choose to make him speak. He's also in control. He's the man with the gun, and he might well have rehearsed his speech in his head.
By contrast, Sauniere has probably never been held at gunpoint before. He should be shocked and terrified; indeed, we're even told that he's stammering. Yet he doesn't even contract "you are" to "you're." Does that strike you as realistic? For me, it's implausible enough to be jarring.
Moving seamlessly between narrative and dialogue, without a change of style, is a common mistake of inexperienced writers. But putting quote marks around a sentence doesn't make it speech.
There are plenty of other ways in which dialogue can fail, for example when it's used to force exposition from the mouths of the characters, but for the purposes of this post we'll stick to discussing the stylistic issues.
So How Do People Really Talk?
You've been listening to people talking all your life, but unless you've had cause to transcribe a conversation in detail, you may never have thought about the mechanics of it.
Frequent features of real conversation include backchannels ("yeah," "mhmm"), interruptions, overlapping speech, hesitations (whether these are silent pauses or "um"s), mistakes, almost-instantaneous corrections of mistakes (often called self-editing), repetition of words, and so on. Speakers may even change the syntactic structure of an utterance halfway through, but somehow this doesn't impair communication. These little mistakes are commonly called 'disfluencies' by linguists.
Let's take a look at an example from conversational analysis, transcribed from a real mother-daughter exchange:
M: but we've got the...Paul Norton and his wife coming round on the evening time for a meal(credit)
K: oh but the only problem is I need to get dad's present
K: and we...so I can either do that on the Saturday and Sunday but I think one of the shops might not be open
M: on Sunday...you, you're home all week...all...
K: oh yeah
M: from Monday
That Wouldn't Read Well...
Stilted dialogue breaks the reader's focus and takes their attention away from the story, but an accurate portrayal of human speech would also read horribly. The secret is to find the balance point where characters sound "natural but better."
Most people have never studied conversation, and have no idea about how real conversations work. They believe in fairytales like "it's rude to interrupt" and think that people speak in sentences. Here's an interesting fact: if you give a non-expert a piece of speech to transcribe, they'll usually edit out all the disfluencies, repetitions, and minor self-edits. But if you ask them, they'll swear that they've written down just what they heard.
You can take advantage of this mental filter. If you can write dialogue that sounds like it's come out of the other end of such a filter, you'll get exactly what the layman considers to be natural speech: usually informal and contracted, possibly slang-filled and ungrammatical, but still fundamentally fluent. If in doubt, read aloud, and then get friends to act out a scene for you.
Many writing books recommend noting down conversations that you overhear - I'll go one step further. Make recordings. (If you're not brave enough to tape your friends, try a live, unscripted interview from the radio or TV.) Then have a go at writing down what you hear. Listen again with your notes, and you'll quickly learn what you're filtering out.
You also have the option to use any kind of mis-speaking as a genuine feature in your writing, when it adds to the story or the characterization. You don't want to pepper your manuscript with transcription-style error notation, but there are times when it may be useful. Imagine the case where someone almost gives himself away, corrects himself before finishing the word, but maybe gives his interlocutor enough to guess that he's hiding something. And because these minor corrections happen all the time in real conversations, your readers aren't going to mind it happening once or twice in your novel. The key is to hold it back for the times it really matters.
I could easily write a book about individual speech patterns, but for now, I'll leave you with a brief thought on characterization through dialogue. I've heard it suggested that you should be able to take any line of dialogue from your novel, in isolation, and be able to identify the character by his style. I think that's an exaggeration: most utterances are too short to exhibit that much variation, and most people have only the most subtle of stylistic quirks. There's nothing wrong with marking one or two characters with distinctive vocal tics or accents (again, listen to real people then tone it down) but if you studiously choose idiosyncrasies for every character then you'll end up writing the dialogic equivalent of a circus freak show. For most characters, all you need is an occasional turn of phrase that only they would use.