WiX DTF Mystery Solved

Last October a (then) former ( and now current ) coworker also named Chris asked me the following in an email:

I am running into a problem with a DTF custom action. When the action tries to run I get:

"SFXCA: Failed to create new CA process via RUNDLL32."

This on is weird because I am using the exact same module in another product on the same machine and it works like a champ. Bizarre.

My initial reply was:

Are any CA's running before it?

I hadn't seen this error before and I was a little concerned because I was such a proponent of DTF and managed custom actions I was afraid that perhaps DTF had an Achilles' heel. I also knew that where my friend worked was the home of a very restrictive environment so if anyone would find a problem with DTF, it would be him!

I introduced Chris to the creator of DTF, Jason Ginchereau, and the two attempted to troubleshoot the problem with little progress. I didn't really follow up on it because at the time I was busy cranking through sprints for a new product at Multimedia Games.

Fast forward five months and I'm working with Chris again refactoring build automation, creating installation tools and writing custom actions for installs. Suddenly the ugly RUNDLL problem rears it's head again only this time it's MY problem because I'm now the install team lead and I'm always saying how good DTF is.

Rats.

So I email Jason again and log a bug (SF210297) at SourceForge to make sure it gets proper attention this time. This time Jason sees some problems on the unmanaged side of DTF. He was assuming that the current working directory of the recycled process hadn't been mucked with ( remember my comment about other CA's scheduled prior? ) when he made his call to RUNDLL32.

So to make a long story short, I just completed an effort with the build team today to update 30 some virtual machines to have WiX 3.0.5210.0. Between this bug fix and many others related to DTF, I really suggest that anyone using DTF custom actions in builds prior to 5210 upgrade ASAP and don't ship your product until you do. You might never come across this problem but I know I sleep better knowing I won't either now also.