Mark Dixon asks a great question — one I've been asking myself.
I've used Dave Gamache's excellent Skeleton framework for my past few projects, and it has made the aspect of responsive design quite a bit easier. But I've also had to pare it down, stripping out all the bits I don't need. I've wondered about using something more lightweight, but the reality is, I'd be better off creating something of my own.
Like jQuery plugins, it can be easy to use a CSS framework without fully understanding the fundamental concepts that it employs. That's where I feel I need to create my own starting point for website projects — it's one of the best learning opportunities I have.
Like most things, there is no hard line that works for everyone. Like many commenters on this post suggest, each project will have its own needs. Use frameworks where they make sense. But don't be afraid to spend some time making something to suit your own needs.