I just attended a day-long workshop on Responsive Web Design at the User Interface Engineering (UIE.com) UX Immersion Mobile conference in Seattle.
A comment by the presenter, Jason Grigsby (of cloudfour.com) caught my attention. It was his opinion that if you had to choose between optimizing a site for speed in loading and making it responsive, that he'd go for the speed. A site that might FIT on a small screen, but that still takes too long to load, will lose more viewers than one that loads quickly, even if they have to do some pinching and zooming to see the content.
For Joomla! sites a plugin can help with the speed thing: JCH Optimize. Here's a blog entry about that: http://bit.ly/11oiaKA.