As an aside, InstallShield goes on to say the problem manifested itself when Data Execution Protection is enabled on a dual-core processor on Vista. At about the same time a new post also appeared on the Windows Installer Team Blog. It seems the root cause of this problem actually stemmed from security changes in Windows Vista.
A future KB article is summarized as:
Custom Action code that is not NX compliant will not run on
Software developers should be aware that starting
with Vista, msiexec.exe is NXCOMPAT. (i.e., it is compiled with the /NXCOMPAT
switch.) This ensures that non-executable code is not run by msiexec.exe.
Msiexec.exe is the image for Windows Installer service, a Windows Installer
client and any custom action sand-box processes. What this means is that any
custom code that is not NX compliant would not be run and would result in a
forceful shutdown by the Vista NX sub-system.
If your Windows Installer (MSI) package use libraries that are not NX compatible, then they might not work on your Vista systems. It is recommended that you recompile your setups using the new NX compatible libraries. Older versions of ATL
(pre-Visual C++ 8.0) are known to be not NX-Compatible. Using ATL shipped with
Visual C++ 8.0 is NX compliant.