I have been pondering this issue for awhile, ever since I decided to take a break from Becca's Story. (Now awesomely titled: The End of Summers)
My main motivation behind this pause: I was running out of steam for the plot. I had set up some crazy plot twisty things and I didn't feel ready to make decisions about the ultimate end of the book. There are lots of them: Does Becca actually die? Does someone sacrifice himself? Does Kevin get killed off? Does Jeff kill Kevin or vice versa? How does Elsie tie into it?
I didn't really intend for Elsie to be in the story this long, but she kind of just hopped on board. Now, I'm stuck with her. I could ax her, but that feels too easy. Something has to happen that is plot pivotal to break up their little team or to get her or Kevin dead. Or arrested. Arrested would be alright. Maybe she gets captured and all her power sucked out or something...hmmm...
Anyway, back to topic. The way I recognized I was having plot troubles with Becca is she suddenly became incredibly whiny. I could sense it in the few chapters leading up to the place where I stopped. They took a long time to write and everything got all personal and she became super annoying. Then I wrote a bunch of short scenes that all sounded lame in which she basically regressed into a spoiled twelve year old, flopping around and complaining about everything.
I have come to realize this is my brain telling me to take a break. This usually happens when the character in question is about to go through some sort of mental transformation. They have reached a pivotal point where they need to make decisions and perhaps alter their way of thinking. They can't be swept along by external events any longer. They become the driving force behind the story, more than the circumstances that got them into this story in the first place.
Sometimes this whininess is a good thing. In my zombie Apocalypse book, the female character is pretty much useless. The book is partly about how she bucks up takes charge of her life, even if her life is now fleeing from cannibals. She realizes how useless she is/was towards the end of the book and resolves to be more like the male main character. Just in time for her to rescue her man. Convenient? Why, yes, yes it is...
I know it sounds crazy, but I'll say it again and again. Many times my characters DO NOT do as I wish them to do. They plow ahead on their own. If I try to force them into my plot, they either baulk completely or become the aforementioned twelve year old girl. They are indecisive, they make stupid mistakes, they sit and talk about their problems forever like an especially terrible episode of Star Trek.
I have a couple ways I have played with to break through this problem. One is to simply take a break, as I did with Becca's story. Give it some time to breathe.
Another is to jump ahead in the story, skip over the decision entirely. I have had mixed results with this technique. If I jumped ahead and write a whole bunch, I am essentially trapping myself into a plot, which as we know doesn't work very well. But sometimes I can write the place where I want my characters to go and then bridge the gap between.
Related to this is to jump backwards. Do some character development, especially if one of the characters has had some trauma in their past. Write about that, then see how you can fit it into the story. I wrote a ton of background stuff for the main male character of my zombie book and ended up using almost all of it in the book itself. It showed why he was so capable and unwhiny, while the female was still struggling. He had already gone through something so personally traumatic, that the reduction of humanity to roving bands of cannibals was no biggie.
Another way I've tried with some success is to keep going with the external forces aspect of the plot. If the characters have reached a decision forced on them by events outside their control and I can't get them to do something, a final event can take the decision away from them and make them to act. Sometimes this clears things up and sometimes it comes out sounding really stupid and I just erase it.
And sometimes nothing helps. Sometimes it is just a bad plot. Or bad characters. Or even just a plot and characters I haven't thought through enough. Why are they doing these things? If I don't know, even subconsciously, then my characters sure as heck don't. And if they have no motivation for saving the world or chasing down a murderer or fighting zombies, then why should they try hard at it?
It is painful, but sometimes I have to ditch a plot and start over from scratch. It hurts. A lot. Especially if I've put lots of time into it. But it has to be done. I don't usually erase a large body of words, just hide it somewhere deep in my files and let it stew until I can face beginning again. I have several that are in that stage right now and I am avoiding looking at them. I can feel the potential there, but I just can't face all redoing all that work just now.
Which ironically, makes me sound whiny and indecisive. Art reflects life, I guess...