Obama seems to genuinely believe that he needs to have this fight now once and for all. Conceding buys him nothing. There is no 'compromise.' Like the 'Red Line' over chemical weapons in Syria, the red line over using the debt limit as a standard negotiating tool has been crossed. The explicitness with which Republicans have pursued this as a strategy (see this piece by Republican strategist Marc theisen on "Hostage Taking 101" and Representative Eric Cartman of Indiana insisting that Obama must 'respect GOP athorita!') probably helped, as does the fact that he doesn't need to run for re-election.
The problem is that all that crap people make fun of about not appearing weak has a germ of truth in it. I can say from personal experience that the sad natural tendency in policy is to treat reasonableness as weakness. It takes a long time to establish a reputation for meaning what you say, and most people don't bother to invest that time and effort. Obama's continued efforts to be 'reasonable' and being willing to compromise to achieve what he considered the most critical outcome consistently read as weakness. His failure to do anything to establish reputation to offset that impression fueled this conventional wisdom. It also fit the stereotype of the Democratic leadership, and the Democratic party generally.
And why not. To the uninitiated, "strength" is what the Tea Party has been doing. Bold, brazen and in your face -- insisting on your version of reality and browbeating anyone who says otherwise. Furthermore, the Tea Party was only the most logical extension of trends that have been working almost 20 years.
The amount of time this is going on is critical as well. The vast majority of reporters simply don't know how to write a story that breaks with the traditional narrative. And even if they are starting to have their doubts, they don't dare break the groupthink. Breaking groupthink has very real social and career consequences. Trust me on this one.
So everything keeps re-enforcing itself into the predetermined narrative, which is how you miss the signs that this time is different. And if you ask 'how is it possible for people to miss things this big,' I remind you that most people have trouble noticing when a gorilla walks across their field of vision if they are focused on other things. When all anyone talks about is horse-race and polling and minute-by-minute political advantage, because it is widely accepted that this is what drives behavior, then that is all anyone sees.
And here we are. And the key actors will continue to misjudge things, at least for the next couple of days, unless the heavy-duty fundraisers lay down the law to Boehner that there must be no default even if it means political disaster for Republicans in the short term (which appears to be happening). Even then, enough people simply cannot conceive of how badly this can go to make the possibility of a default (and resulting panic) real.
From an analytics perspective, it's kinda fun to watch. I think of it like wonk crossword puzzles. They tickle the right part of the brain, but people who aren't into them don't understand what makes them entertaining. Except that screwing up your crossword puzzle doesn't generally result in crashing the world economy.