Over the past few months I've been working a fair bit on optimizing certain areas of Drupal core, mainly
node_load(). While I've used devel and ab pretty much exclusively in the past, I've found myself needing to see exactly
why a page is slow, and devel and benchmarks only get you so far with that. So I ended up using xdebug and kcachegrind/webgrind, which I'd never used before, and had regarded as voodoo before trying them out; they looked complicated and hard to set up. Turns out that they're not to hard to set up, quite easy to use, and very, very helpful.
While Drupal is being used for some very large, very busy websites, there's still room for improvement in core in terms of performance, so we can do better. And part of doing better is knowing where the problems are and how to measure improvements.
Before you profile, you need to know what to profile. Optimizing either the database or PHP is quite labour intensive, and particularly when working on Drupal core (but ideally in contributed modules or specific projects too), you also have to keep the code readable, extensible, and maintain functionality. Making something a couple of milliseconds faster at the cost of readability won't fly. So in my opinion it's best to focus on things that happen often - loading nodes, viewing users, or the code Drupal loads for every request. When optimizing a real site, this usually means your front page, and traffic logs will tell you which other pages are used a lot. Or the mysql query log analyzer will tell you which queries are taking the most time overall. If one administration page takes 3 seconds to load, then that's not great, but that page could take much more effort to optimize than shaving 20 ms off every page request on your site - which in terms of server load and user experience for most of your users makes a much greater difference.
The next thing to do is to identify the actual bottlenecks. yslow and firebug can help you for front end performance - which are often a bigger issue in the wild, but while Drupal core can make those problems easier to solve (reverse cache support, CDN support etc.), they need real world situations to be measured in, and the actual solutions are often site specific. So here I'm going to focus on what you can find on your local development environment - and three tools, devel module, ab, and xdebug/cachegrind.
