Dealing with very large number of files

Allan Bommer recently asked on the WiX-Users list:

The one item I am worried about is managing a very large number of files. Our application’s Help directory contains about 1500 files (html, gif, etc). It seems the only way to get these files installed is to list them explicitly (with some help from Heat). It also appears that it is suggested to make each file a component.

Explicitly dealing with 1500 files and their component GUIDs seems likely to have a very low ROI for the programmer and company. I’m sure my clients would rather have me crank out a few useful features in the application than spend countless hours on all those individual entries. Perhaps if you have a million clients, then dealing with all those files individually makes sense, but with 100, 1000 or 10,000 clients, the cost-benefit ratio doesn’t look very good.

Allan hits on one of the weaknesses of the declarative design of Windows Installer. Rich metadata describing the behavior of the contents and behavior of the installer has the huge downside of having to author that metadata and keep it in sync with the ever changing world. The upgrade servicing patterns also require that you be absolutely perfect in how you do this. One component rule violation or primary key column change across dozens of builds will result in undesired servicing behavior.

Allan then asks ( in his own way ) about dynamically linking the files. Dynamically file linking will generally work but it has several downsides:

The most important downside is the possibility that a file should be in the package, but that it was not where it was supposed to be at build time. If the file was described declaratively this would be a build break. But with dynamic file linking, it simply isn't linked and now we have a potential runtime break.

The second downside is that it's very difficult to perform minor upgrades and patches due to the dynamic nature of the component authoring. Some tools like InstallShield have patterns for mitigating this, but it's still a crap shoot.

The third downside is that your build will take A LOT longer to perform since it's not only creating your package but that it's also authoring and validating your component definitions.

But I think there is a third way to go about this and I probably spent at least one of my five hours with Sinan Karaca touching on this subject: Use authoring tools with very rich component wizards. InstallShield has one, but it's not nearly good enough in my opinion. Here are a few attributes that would make it better:

`Deprecated Component` instead of Deleted Component. Minor upgrades can't remove a component and it's keyfile, so instead you take advantage of Transitive Components and NOOP conditional expressions along with a 0 Byte file in the media. This allows a developer to safely remove a component without worrying about which upgrade pattern will be used to service the installation.

Better definition and control over the concept of `Best Practices` when authoring the components. On MSDN you will find 2 very different definitions of best practices for componentizing files. The MSI team breaks things by directory and COM servers with all other files being allowed to share a component as a designated key file and companion files. The Visual Studio team takes a different approach that every file should be a key file. To be honest, I see ways in which they are both correct. A properly behaving component wizard shouldn't simply ask `do you want to follow best practices` as if it is a black and white decision. It should offer you a choice of those two best practices and a custom mode where you get to decide for yourself. The point is when I have some ASP.NET developer adding files I don't want him to have to understand the component rules. I don't want him to need to understand that default.ascx is the keyfile and background.png is the companionfile. He just needs to know that he can update either file and he can expect to see his changes on the server after the installation is executed.

Another feature that a good wizard should have is the ability to automatically detect changes between the currently authored components and the directory structure that it is syncronizing. We are talking easy button here: Rescan Components followed by "I see these 5 new files... would you like to add them?" or "These 5 files are missing, would you like me to deprecate them?"

MSI might be complex, maybe more then it needs to be, but there is value in it's patterns. It's just time that we have authoring tools that makes it a hell of a lot easier so that Allan doesn't have to be pointing out the obvious ROI problems associated with it.

Flashback: A trip down memory lane

I was reading over on Aaron Stenber's blog about Microsoft using Virtual disks to ship windows and applications and it reminded him of the old days when Microsoft would just ship preconfigured hard drives instead of installs.

I recall that at my last job that they really weren't that far removed from the practice. Going back further ( 1996 Windows 3.1 days ) I once invented a tool while I worked at Electronic Data Systems called `The Cookie Cutter`. This tool was a bootable ISO9660 ( not easily done back then.... you had to be comfortable reading the El Torrito spec and using a Hex editor ) that started up MSDOS and ran a compiled QBasic program that I wrote.

The program would present a series of configuration screens to select an image to load and needed TCP/IP settings. We could select from 18 different configurations ( Banyan, FTP PC-TCP, SmartTerm IP, Novell, MS-TCPIP ), then it would automatically fdisk / format / extract and configure all the needed INI settings to configure the machine. Funny, even back then I wasn't afraid of configuration data being a pain. The only real pain was when 10+ people lost their jobs because of the efficiency improvements in our work flow. I travelled a lot back then and this is what started me down the path of setup. It was sad that we choose to clone our workstations instead of install them because our installs were so bad back in those days.

The other funny thing is that Microsoft seems to think that virtualization is new. The fact is I used to be a rabbid Amiga addict. I actually learned how to install and configure MSDOS/Win 3.1 by using PC-Task to emulate an x86 PC. Sorry American Megatrends, please don't ask where I got a BIOS image from..... :)

Going way back, my first computer experience ever was actually quite embarrassing. My older brother had a friend who owned a Commodore Vic 20. He gave me a book to read and sent me on my way. When I returned he tested me with a simple question. You have a program in memory and you want to erase it, what do you do? The correct answer was to type NEW. I hadn't read the book so I simply reached for the power switch and toggled the power. Not too bad for a 7 year old I suppose.

Despite my less then stellar start, my parents decided to get a Commodore 64 and it all started with Microsoft Basic 2.0 & 6510 Assembler on a Commodore 64. Ahh the Commodore 64 Advanced Programmers reference manual.

It's funny to think that I've been coding for over 25 years and that Microsoft has always been there... whether I wanted them or not. How things change and yet stay the same.

Yes WiX, I really DO want to do that...

Consider this MSI fragment( created in Orca and transformed to XML by MSI2XML ):

<table name="InstallUISequence">
<col key="yes" def="s72">Action</col>
<col def="S255">Condition</col>
<col def="I2">Sequence</col>
<row>
<td>Dialog1</td>
<td/>
<td>1</td>
</row>
</table>

It's not a real installer, it's a `fake` MSI that was created just for fun and just to show what Windows Installer is capable of doing once you free yourself from coloring between the lines.

Now MSI2XML can easily round trip the table data without passing a whole lot of judgement. Just for fun I thought I'd see what it's like in WiX to do this. I take the XML and reconstitute the MSI and then use Dark to create a wxs project. Now WiX takes lot's and lot's of issues with the source even though it is a perfectly formed MSI database that will actually do what it was intended to do. Let's ignore most of this for a moment and talk about the Sequence tables.

In my XML you'll notice that only the InstallUISequence table is defined and it's pretty sparse..... it only contains a single action, a call to a UI Interface Wizard called Dialog1 (yes, unique name I know...)

Now note the XML that dark creates to represent this:

<InstallUISequence>
<Show Dialog="Dialog1" Sequence="1" />
</InstallUISequence>

This snippet sounds like what I describe, right.... a single sequence with a single action. But that's not what candle/light does with it. It automatically goes and creates:

AdminExecuteSequence ( 8 actions )
AdminUISequence ( 4 actions )
AdvtExecuteSequence ( 7 actions )
InstallUISequence ( 6 actions instead of 1 )
InstallExecuteSequence( 13 actions )
In order to get a successfully round trip'd resultant MSI ( pedantic error checks for the time being ) the XML actually needs to look like:

<AdminUISequence>
<CostInitialize Suppress="yes"/>
<FileCost Suppress="yes"/>
<CostFinalize Suppress="yes"/>
<ExecuteAction Suppress="yes"/>
</AdminUISequence>
<AdminExecuteSequence >
<CostInitialize Suppress="yes"/>
<FileCost Suppress="yes"/>
<CostFinalize Suppress="yes"/>
<InstallValidate Suppress="yes"/>
<InstallInitialize Suppress="yes"/>
<InstallAdminPackage Suppress="yes"/>
<InstallFiles Suppress="yes"/>
<InstallFinalize Suppress="yes"/>
</AdminExecuteSequence>
<AdvertiseExecuteSequence>
<CostInitialize Suppress="yes"/>
<CostFinalize Suppress="yes"/>
<InstallValidate Suppress="yes"/>
<InstallInitialize Suppress="yes"/>
<PublishFeatures Suppress="yes"/>
<PublishProduct Suppress="yes"/>
<InstallFinalize Suppress="yes"/>
</AdvertiseExecuteSequence>
<InstallUISequence>
<CostInitialize Suppress="yes"/>
<FileCost Suppress="yes"/>
<CostFinalize Suppress="yes"/>
<ValidateProductID Suppress="yes"/>
<ExecuteAction Suppress="yes"/>
<Show Dialog="Dialog1" Sequence="1"/>
</InstallUISequence>
<InstallExecuteSequence>
<CostInitialize Suppress="yes"/>
<FileCost Suppress="yes"/>
<CostFinalize Suppress="yes"/>
<ValidateProductID Suppress="yes"/>
<InstallValidate Suppress="yes"/>
<InstallInitialize Suppress="yes"/>
<InstallFinalize Suppress="yes"/>
<PublishFeatures Suppress="yes"/>
<PublishProduct Suppress="yes"/>
<ProcessComponents Suppress="yes"/>
<UnpublishFeatures Suppress="yes"/>
<RegisterUser Suppress="yes"/>
<RegisterProduct Suppress="yes"/>
</InstallExecuteSequence>

Now I can understand that for a real MSI would want all of those actions/sequences by default, and I also understand that some pople are going to want to quickly dismiss this use case as `fake` and `invalid`, but consider this prospective:

Shouldn't a successful decompiler/compiler story result in the creation of a binary that is identical ( or as close as possible ) to the original binary? To me it seems like Dark should respect ( and warn of course ) that this was the original configuration of the database and author XML to represent it. Instead dark seems to try to read between the tea leaves and say `well I know this is what you are, but I really think you should be this...`.

In the past I've worked on some pretty big, complex installer frameworks that delivered dozens of installers by consuming dozens of product assests in different configurations with different brandings. The frameworks are so big and so active that it's not practical to just dump it all at once and replace it with an entirely different system. You have to refactor individual layers and components while maintaining the existing contracts. While doing this refactoring I've attempted to incorporate WiX in my solutions, but unfortunatly it's been issues like what I just described that have forced me to use other tools like MSI2XML which are much more respectful of what I'm trying to accomplish.

Disagreeing with the Benevolent Dictator

I've been partially following the WiX-Users list recently when I noticed this question from Leo:

How do you define the InstallExecuteSequence table in WIX so that the standard actions PublishFeatures, PublishProduct, RegisterProduct and RegisterUsers won't show up? thx!

Oh my, I knew right away that this was stepping on a hornets nest. Anyone who has followed WiX / Rob Mensching / Aaron Stebner ectera for awhile knows that this group is pretty self confident that their way of doing this is the right way and everything else is wrong. However this is the WiX-Users list and I really don't want to troll for an argument. I decided to sit back and watch. Sure enough, Rob Mensching replied:

Why would you want to do that?

Ok, so maybe this is a case of email not conveying emotition, but I'm pretty sure what Rob was really saying that you have to somehow convince him that you have a valid reason for doing this. I've had several reasons in the past, but again, I don't want to go trolling... Then someone replies with an answer to the actual question of HOW you would do it, but continues to lecture about why you shouldn't do it. Then yet another person replies to lecture about how you shouldn't even use MSI if this is your need, go use NSIS. Ouch... yuck.

Now our original poster Leo comes back with a pretty good piece of logic... the MSI team specifically has a help topic on MSDN explaining how/why you might want to not register your product with these standard actions.

MSDN does not indicate any problems or recommand not doing this. As a matter of fact MSDN provided the instruction for omitting the registration of application at Exactly what kind issues I might get into if I do this. Please advise. Thx!

Now I've not waited long enough to see additional replies, but I already know from experience that this kind of questioning isn't going to go over well in the Wix-Users space because you have to keep in mind that what the Windows Installer team considers a `valid` MSI and what Rob considers a `valid` MSI are entirely different.

My opinion is as long as you understand what your doing and the related impact..... go do your thing. I follow most generally accepted best practices, but on certain issues I've had business requirements that require me to do something different. I personally won't listen to anyone try to dictate ( no matter how benevolent ) what my install should or should not do nor will I listen to complaints like `fake` or `trojan` MSI and most importantly I don't accept something as some immutable, undeniable best practice just because one person says so.

Mailbag: Have you seen this?

I guess thinly veiled ambigious references aren't enough because I keep getting emails from friends asking if I've seen the news about Macrovision. So yes, I know about the 7% RIF and I know about the director who picked up a nice $200,000 dumping 15,000 shares of MVSN stock. I also know about various people suggesting that a sale is in store for Macrovision. However, until people actually start flooding my mailbox with real inside scoop, I'm just an outsider looking in wondering what's going on. That said, here are some interesting quotes from a Yahoo! investment board:

From `candor2001us`

Why is this a good sign? you usually cut employees in bad times, when there is no work for them. This is not a sign of a growing company.

From `maxiumboostage`

exactly, with mvsn dropping safedisc , ripguard, hawkeye, the flex stuff not being developed any further i think its soon gonna be time that fred makes a move. he has been known to break companies apart and this is the first sign that vsn is being broken into nice purchasable chunks for any interested parties that want to buy the technology. i also heard on the grapevine that electronic "safediscs main customer" have dropped the product thus rendering it non profitable. well ill wait and see but i am standing by my predictions that its gonna be broken up and sold into chunks. bye bye macrovision.

From `josedelacruz150`

EA may or may not have dropped safedisc. That's an old product anyway in a market that's moved on.
Its never been a money maker, and its end of lifed already.
Macrovision doesn't have any real technology but it does have IP.
Wouldn't be surprised if he sacked them all and sold that.

From `candor2001us`

that's not necessarily true. You cut employees to strengthen the bottomline especially when the top line is remaining stable.....I think Fred just wanted to cut deadweight especially with a couple of Execs and Im pretty sure there will be another one cut soon once he proves his incompetency.....

you also start to slim down when getting ready to look for a buyer.....

It's been great working here though....

An afternoon with InstallAware's Sinan Karaca

It was a really nice father's day today: My smiling girls, the sweet cards and presents, the bacon wrapped Fillet Mignon and best of all, being given a pass to go enjoy some free time without the responsibilities of parenthood.

So what better to do on an afternoon off then to sit at a Starbucks for 5 hours chit chatting with the founder of InstallAware, Sinan Karaca. We both went into extreme detail over practically every aspect of the installation space.... way too much to write about here. I think a lot of good information flowed and I've come to realize that the two of us actually disagree on very few topics. I'm looking forward to see if InstallAware decides to implement a few of my thoughts in WiXAware and I'm equally interested in seeing if I can fit WiXAware into my installation solutions.

Either way it was a very productive session and as always, I always enjoy meeting with people who have an interest in setup. If you find yourself in Austin, feel free to look me up.

Wondering about the future

I've been wondering about the future lately... both personally and professionally. Sometimes the way things appear to be and the way things actually are can be unrelated. You often have clues to the coming changes but you either want to ignore them or just not speak of them. Maybe it's just hard to talk about something when you are only dealing with suspicions.... you just don't know the exact facts.

I know of a couple companies that are going through some pretty difficult times right now. Both have given clues that they aren't making the numbers that their investors are looking for and both have recently gone through some pretty serious layoffs and `reorganizations` as people scramble to figure out a way to make money. Both are also giving some pretty strong clues that they will be on the selling block in the near future. I don't want to name names since I don't want to hurt the innocent, but I'm guessing that by the end of the year, time will reveal all. It has been stressful lately because one of these companies I currently work for, and the other is a company that I've very recently considered working for.

Then there is the big unknown that strikes closer to home... Will my wife's cancer come back?