Old Québec City is a UNESCO World Heritage Site for a reason. It’s the only North American city to have preserved its fortifications and ramparts, and is second to none in terms of its architecture, historic charm and overall awesomeness. If you’ve never been, you must go.
However, there’s a whole lot of charm in Québec City outside the walls of the Old Town. The “haute ville” and the “basse ville,” particularly in the St-Jean-Baptiste, Montcalm, and St-Roch neighbourhoods (just to the west of the ramparts).
The old buildings in these neighborhoods—not quite as old as Old Québec, but old enough—have generally been well-maintained and ooze charm. These areas are mostly residential with a dash of commercial here & there. Formerly working-class, they are gentrifying quickly and you can feel the tension between the traditional residents and the monied classes moving in. It shows in the grafitti, and it shows in the vary states of renovation.