If you have a look here:
https://trello-attachments.s3.amazonaws.com/4f0680090bd2eb29720c5203/4f19acb1a63c9d76680115fe/15d8c3dfa13479c10918b94c80952280/upload_2012-08-29_at_3.23.54_pm.png
You'll notice that in our setup the funtion GFCommon::replace_variable() is taking a lot of time. Over 2s total in a request. I'm sure there are many ways that you could optimize this and you guys would probably be much better at this than I am, but one simple thing we've done to help this is create an index on Lead_id and field_number. It seems to have helped us. Perhaps it would help others as well.