For QTP 11, the default support is only for with Protected Mode "OFF". For Protected Mode "ON" support, a patch required: QTPWEB_00073. You can download it from the link below:
Internet Explorer's protected mode is a feature that makes it more difficult for malicious software to be installed on your computer.
In addition to helping protect your computer from malicious software, protected mode allows you to install wanted ActiveX controls or add-ons when you are logged in as an administrator.
Protected mode is turned on by default in the Internet, intranet, and Restricted sites zones and an icon appears on the status bar to let you know that it's running. For details about protected mode, please refer to the document below: