Books can be seen as architecture. They have a foundation – the basic situation and the stakes. They have bearing beams – characters, plots and subplots, setting. Then there are room dividers – scenes and chapters and, if you use them, sections. (My books usually seem to divide themselves into four sections.) And finally, there's decoration: description, dialog, the cute plot twist.
Ideally, all these things cohere to make a whole. The house, or whatever it is, is sturdy; it's not built on quicksand, the walls are plumb-straight, there's some sort of sense of proportion – the living room, for example, isn't bigger than the whole rest of the house put together. The entrance is attractive, the floor plan is easy to follow. There are no missing stairs, no unexpected holes in the floor.
I can go on with this longer than you'd like me to, so I'll cut to the point.
A problem can arise in any of these areas. The ones that are easiest to deal with are decorative problems. They're relatively easy to redo, and they don't (usually) require changes a hundred pages on either side of them. Scenes and chapters can be rewritten and reorganized; that's a little harder, but it doesn't need to threaten your entire existence.
The big problems arise when something's wrong with the foundation and/or the bearing beams.
These are the problems you try to anticipate by walking around your basic idea a few times, by writing about it, by making sure that the stakes are something you care deeply about. By doing lots of warmup on the characters.
During the writing of the book, you try to avoid these problems by making certain the stakes are always in mind, that your characters drive the story rather than the other way around. By asking yourself repeatedly: Is this true to my characters? Do I care about this? Am I being honest with my reader?
You can do all these things, but there still may come a time when you become aware that something is basically, dreadfully wrong. You're hopelessly lost. No matter what direction you move in, it's the wrong one. Your ten-mile sentence, which was supposed to head due east from the beach at Malibu, has led you somewhere several miles offshore in the Pacific Ocean. You're adrift. If you're lucky, you're floating. Your boat (for purposes of the metaphor, let's say it's a houseboat) lacks a compass, there are no sails, and water is rising in the bilge, but you're floating. Just barely.
This is big problem time.
About the only thing you can do (or, at least, the only thing I know of) is to review the entire manuscript, find the things that most closely reflect what you were trying to do in the first place, and salvage them. Then scuttle that boat. Let it sink, and start over.
And I mean from the beginning.
Go back to your most basic idea. Look at it all over again. Make sure it's an idea that means something to you. Re-evaluate the stakes. Do you really care about them? Passionately? If you don't, that's your problem. As Raymond Chandler says, Technique alone is never enough. You have to have passion. Technique alone is just an embroidered pot holder. If you haven't got passion about the issues in your book, it's no wonder the story you were weaving turned out to be a potholder.
This is a bleak situation to be in. I know, because I've been there. But I'm going to say again something I've said before, and something I've had to say to myself.
Get over it.
This is a book, not a lung x-ray. The sun will rise in the morning, and you'll still be here to see it. And you're not really starting from scratch. You have some important tools. First, you've got some good material from the discarded manuscript. (It may not actually end up fitting in the new book, but by the time you discover that, you'll already be writing and it won't matter so much.) Second, you've learned one way not to write your book. You won't make that particular mistake again. God willing.
So steel yourself. Start the warmup all over again. Write about the idea, the characters, the stakes, the setting. When you're ready, when you really know where the center is, start your new first scene. Stick with your routine. Keep your fingers on the keyboard. Open yourself to the things your characters say and do. Keep them true. And don't be overwhelmed by the magnitude of the task. Keep your eyes on the next word, the next sentence, the next scene. This is how books are written.
Be brave. Keep writing. If there's a book in that idea, you'll eventually find it. And you'll have learned something invaluable. You'll have learned that you can recover from a complete and utter shipwreck. You'll have survived the worst thing that can happen to a writer.
It'll take a lot to frighten you in the future.