To my dismay, I discovered that MSI doesn't natively expose the ProductType values returned by the GetProductInfo Function. [Hey MSI team, how about adding this in 4.5?]
Given this fact, the proper way to detect a Windows Server 2008 OS installed with the Server Core option would be to write a custom action that queries the aforementioned API and sets an MSI property when any of the *_CORE* values are returned by the pdwReturnedProductType parameter (such as PRODUCT_DATACENTER_SERVER_CORE or PRODUCT_ENTERPRISE_SERVER_CORE or PRODUCT_STANDARD_SERVER_CORE or PRODUCT_WEB_SERVER_CORE, etc.).
As Chris pointed out when I discussed this with him, if you're using a vendor tool to create your MSI installation then the tool may have an alternate method of detecting Server Core (InstallShield can do this via an InstallScript CA).
However, if you're not using a vendor tool or if creating a CA to query GetProductInfo isn't desirable in your situation, you can detect a Windows Server 2008 installed with the Server Core option by the fact that the Windows Explorer user interface isn't installed on Server Core.
To do this, add a search for "explorer.exe" located in the %windir% folder (see MSI example table records below). Then use the public property set by your search in combination with the VersionNT and MsiNTProductType MSI private properties to detect Server Core.
AppSearch Table Record:
- OS_SUPPORTS_UI findWindowsExplorer
DrLocator Table Record:
- findWindowsExplorer [WindowsFolder] 0
Signature Table Record:
- findWindowsExplorer explorer.exe 5.0
Conditional Statement (true if Server Core running)
- VersionNT=600 AND MsiNTProductType>1 AND NOT OS_SUPPORTS_UI
In fact, assuming MS doesn't depricate explorer.exe in a future version [big assumption], just remove the OS version and product type values from the condition and specify 'NOT OS_SUPPORTS_UI' for a condition that should work on both WS2K8 Server Core as well as future versions of Windows w/o UI support).