Bug#916496: xcom-ufo: cannot build package from GOG 2.0.0.4 installer

[Tim Allen]

I've done some more research, and come up with more details.

As I understand it, the original X-COM for DOS was released as version
1.0, then had a bunch of fixes up to version 1.4. Later, a "collector's
edition" was released which ported the game to Windows and included even
more fixes.

PCGamingWiki says[1] that the GOG build is straight up version 1.4,
bundled with DOSBox to make it run on modern OSs. It also says that the
Steam build uses the 1.4 engine for DOS, but with some Collector's
Edition data files.

[1]: https://pcgamingwiki.com/wiki/X-COM:_UFO_Defense#Availability

I found some details on the files that shipped with the Collector's
Edition, so together with the official patches I acquired earlier, I
believe I can trace all these variations.

Going back to the original list of missing files...

# assets not in setup_xcom_ufo_defense_2.0.0.4.exe:
#   group_members: |
#     44288     6a2b1ec8182f5025ee505c6949356925 GEODATA/BIGLETS.DAT
#     12456     5ae490e16959e1961071f0172c793c94 GEODATA/SMALLSET.DAT

These files come from the Collector's Edition. Compared to the 1.4
versions, they add extra characters that weren't in the original DOS
character set[2][3]. I imagine the DOS engine doesn't actually use them,
but OpenXCOM might.

[2]: https://www.ufopaedia.org/index.php/BIGLETS.DAT
[3]: https://www.ufopaedia.org/index.php/SMALLSET.DAT

#     46601     1770cf9f3a18350c7afa54b0e271c359 GEODATA/ENGLISH.DAT
#     52672     eacbbca8d2de655cda7aa403702bec46 GEODATA/FRENCH.DAT
#     53587     1cd8dd5069970d0f2a6983339057a34b GEODATA/GERMAN.DAT

These files come from the Collector's Edition. Comparing them to the
1.4 versions, the only differences are that the message "Quit to DOS"
(or its translated equivalent) has been changed to "Quit" (or
translated etc.).

#     93853     7194c1480e6ce2d3e188133ece4fdefb SOUND/ROLAND.CAT

As discussed previously, this file comes from the 1.4 patch.

Looking at the original list of obsolete files:

# obsolete:
#   group_members: |
#     32768     9f20ed66008a2fbc055c10db74c44307 GEODATA/BIGLETS.DAT?orig
#     9216      3943027ebeab34adfa3d31c04adc886d GEODATA/SMALLSET.DAT?orig

These are the versions from the GOG release, and since they doesn't
appear in any other patches, presumably from the original 1.0 release.

#     46608     a28031075f0e531d5896ffca8125d7bc GEODATA/ENGLISH.DAT?orig
#     52679     1538fc2f7d85aa81d06b4bff319d9902 GEODATA/FRENCH.DAT?orig
#     53594     342570230da352242219d6fea289d8b1 GEODATA/GERMAN.DAT?orig

These are the versions from the 1.2 and 1.3 patches.

Lastly, the mysterious extra ROLAND.CAT. As I understand it:

  - 1.0 through 1.3 came with music data for AdLib (in ADLIB.CAT) and
    Roland CM-32 (in ROLAND.CAT) devices.
  - 1.4 switched to a different music engine[4], which used different
    formats. ADLIB.CAT was updated to use the new format, and a new
    GM.CAT was supplied to support General MIDI devices, but ROLAND.CAT
    was left untouched.
  - Therefore, if you tried to use Roland CM-32 support in 1.4, the game
    would hang[5] as the new music engine tried to interpret the
    ROLAND.CAT file in the old format.
  - A fan reverse-enginered the old and new file-formats and repackaged
    the Roland music data from 1.3 in the format 1.4 expects[6],
    which is the version bundled with the GOG release.
  - However, OpenXCOM does not use or care about ROLAND.CAT, only
    GM.CAT, so there's no point packaging it at all.

[4]: https://www.ufopaedia.org/index.php/SOUND
[5]: https://pcgamingwiki.com/wiki/X-COM:_UFO_Defense#Game_hang_when_attempting_to_use_the_MT-32_in_version_1.4_.28DOS.29
[6]: https://www.vogons.org/viewtopic.php?t=21542l&start=32

On Sat, Feb 02, 2019 at 11:06:29AM +0000, Simon McVittie wrote:
> On Sat, 02 Feb 2019 at 20:17:38 +1100, Tim Allen wrote:
> > I found a site[1] claiming to host the official 1.1, 1.2, 1.3 and 1.4
> > patches, alongside a bunch of other unofficial patches. It turns out
> > that the ROLAND.CAT that game-data-packager was looking for (MD5:
> > 7194c14...) is part of the UFO 1.4 update. The site says[2]:
> > 
> > # This patch fixes many stability issues, some game stopping research
> > # bugs And removed the startup copy protection. However it also replaces
> > # the sounds with ones that are not as good as the originals.
> 
> Should we be preferring to package the unpatched ROLAND.CAT (143866 bytes, as
> shipped by GOG) rather than the v1.4 ROLAND.CAT (93853 bytes, as shipped by
> Steam) if we can see both, then? The YAML syntax lets us prefer one version
> over others.

Now that I know OpenXCOM doesn't use it, there's probably no point
packaging it at all. Perhaps OpenXCOM might use it at some point in the
future, but that seems very unlikely given that GM.CAT is almost exactly
the same thing, and ADLIB.CAT is probably the one people care about for
nostalgic reasons.

> > The BIGLETS.DAT and SMALLSET.DAT files don't appear in any
> > patch, official or unofficial, expected hash or otherwise, so I have no
> > idea where they came from, or why the Steam and GOG releases differ.
> 
> *shrug* if they work, they're probably good enough...

Knowing what I know now, I expect the Collector's Edition/Steam versions
should probably be preferred over the 1.4/GOG versions... but either one
should be fine.

> > The "obsolete" ENGLISH.DAT, FRENCH.DAT and GERMAN.DAT that the GOG
> > release includes come from the 1.2 and 1.3 official patches. The
> > versions that game-data-packager expected aren't present.
> 
> The addition of xcom-ufo.yaml in commit a5fd5cb5 had details of the Steam
> release (Steam app ID 7760), so I assume the versions now tagged as "updated"
> are from Steam. (Alexandre, can you confirm this?) It isn't unheard of for a
> Steam rerelease to have new changes that weren't in any previous official
> patches.

As discussed above, the Collector's Edition/Steam versions are probably
slightly more appropriate for OpenXCOM, since quitting OpenXCOM no
longer takes you back to DOS. ;) The 1.4/GOG versions should still be
acceptable, though.