InstallShield's InstallScript CA refactoring has led to one tradeoff: There seems to be about a 2 second transaction cost for invoking a CA on a typical machine.
With the old ( and very unreliable ) model, the InstallScript engine was initialized at the beginning of the installation and maintained for the live of the installation. With the new ( and much more reliable ) model each invocation of an InstallScript Custom Actions results in the initialization of the InstallScript engine, execution of the custom action and cleanup of the engine.
So let's talk about ways to mitigate this:
Use WindowsInstaller Properly
Now let's be brutally honest about custom actions. I believe that a setup developer should strive to have as few custom actions as possible. The built in patterns of WindowsInstaller should be leveraged as much as possible.
Combine Related Custom Actions
Think of ways of creating combining multiple custom actions into a single custom action. If you are calling more then one custom action in a row in the a given sequence, create a new InstallScript function that acts as a driver to the other InstallScript functions. The custom actions that are being grouped together don't have to have the same conditions. You can use MsiGetProperty and if/else/case constructs within the driver function to conditionally execute functions. If you have more custom actions elsewhere in the sequence, ask yourself if they could be rearranged for greater effect. Each combined CA will save you about 2 seconds.
Let's say you implement AppSearch but then you have to use a InstallScript Custom Action to perform some business logic to complete the search pattern. There is a 2 second cost there, but the user is unlikely to notice because he's look at a splash dialog telling him to please wait while setup initializes. Same thing applies to deferred Custom Actions since the user is watching the InstallationProgress dialog and the various status messages and progress bar updates occurring. Frankly they expect this step to take some time to execute.
ControlEvents on dialogs is where you can get in trouble with the user very quickly. When a user clicks the next button on a dialog they expect to be taken to the next dialog very quickly. A two second delay is borderline for even the most patient user. I've recently had a discussion with a setup developer who had several DoActions on the control which resulted in as much as a 20 second delay. Consolidating those custom action calls as described would reduce this by 90% but it would still be 2 seconds.
Refactor to C++
While I really believe that InstallScript is a robust domain specific language, there may be scenarios ( like the above ControlEvent scenario ) where performance really is critical and in that case you will have to write a C++ Type 1 custom action.