Are you running SQL Server 2005 SP2, and do you regularly use (or use a 3rd-party app that regularly uses) the sp_OA... procedures? If so then you should look to apply the latest service pack of SQL Server as there is a known memory-leak issue relating to the use of the sp_OA... procs.
Have a look here for further info: http://support.microsoft.com/kb/937277
We recenly had a call open with Microsoft about this same problem and it turned out to be the use of these stored procs that caused the symptoms. Ironically it was the monitoring software that we were using to monitor the problem that was using the sp_OA... procs.
Another cause of memory (VAS) pressure can be the use of sp_xml_preparedocument without a corresponding sp_xml_removedocument.
Hope this helps.