You probably haven't really got a problem at all. Your application, being a service, is running in Server GC mode. If there was contention for your memory, the GC would run, clean things up and then you would push on again.
You need to look and see whether you are actually experiencing any problems, rather than just that number looking large.
.NET Tools Division