Rob Mensching just posted a blog giving insite into why the Windows Instalelr team doesn't support managed code custom actions. It's a really good read to get a view of their inside thinking, but I have to admit I completly disagree with them. Microsoft ( or Macrovision for that matter ) doesn't write my pay checks but I'm still going to give some free advice here....
Regarding the strategic direction, I completely agree with that built-in Windows Installer stories should be updated to eliminate the need for custom actions.
Unfortunately I've lost confidence that the Windows Installer team will get this job done. The last generally available release of MSI isn't available down-level and only adds stories related to Vista UAC/RM functionality. Cool stuff, but not the additional standard action patterns that many people have been begging for over the last several years. The release before that seemed to focus mainly on patching. Even if the Windows Installer team was to pull a rabbit out of the hat and support these stories, I doubt they would port it down-level to a 3.x branch.
Please, tell me I'm wrong with that assessment. Please tell me that Bill G is going to throw so money at that team and we are going to get the framework improvements we have all been begging for.
Until that day comes, Custom Actions aren't only tactical; they are strategic because there is no other viable solution. You raise some interesting points about AppDomains and the CLR and yet I've seen InstallShield address these issues with their CoCreateObjectDotNet() pattern. I've also done some work on my own with .NET 2.0 IPC remoting and I've found it to be a very reliable way of hosting custom actions out of process and yet still giving them the ability to communicate with MSI.
For the rest of Microsoft, managed code is clearly marketed as the future of software development. Windows Installer should adopt this strategic initiative and simply overcome the technical hurdles. This way we can have really good custom action patterns to play with until the day comes that MSI natively supports the patterns that we are writing custom actions for. Then we can gladly dump our CA's and refactor using the built-in standard actions.