I don't know of any way, published or non-published, to increase the timeout of the SQL Backup mail module. I guess that what is happening is that the mail has succeeded, but SQL Backup is timing out while waiting for an acknowledgement from the SMTP server. If that's the case, I think the desired behavior would be to report an error, since the software can't determine whether the mail was sent successfully or not.
You can try running an SMTP test from the SQL Server and actually read the messages that the mail server is sending -- maybe that would help you work out what the mail server is doing. Please see Microsoft's website for the instructions on how to test an SMTP connection using telnet: