I think we have worked out the source of the error using a debugger.
When Profiler starts the w3wp.exe process, it first tries to launch w3wp.exe using the current user security context. Sometimes this fails, and it tries to launch it from the ANTS Performance Profiler 6 service, which runs as SYSTEM. If any errors occur, they happen in the service, and tries to send them through the .NET remoting channel that connects the service to the profiler program.
In this case, the actual error class was not marked as serializable, so instead of getting the actual cause of the problem, we get a serialization error. When we debugged the service, we found that the Profiler could not determine the IIS Site ID based on the URL in the address box. This was because the URL included the port that we wanted to profile on rather than the one IIS normally uses.
So it is important to note that the address you enter is the one you would normally use in IIS.
Additionally, I've found that the profile on original port option should work, but it will restart IIS and a lot of the time the restart fails because of problems with security or one of the dependent services launched by IIS failing to start. Running the Profiler As Administrator may make the process work if the Profiler trigger process can't elevate.
Rest assured I'll log a few bugs around this and try to come up with some better documentation and diagnostic steps to troubleshoot the most common IIS profiling failures.