Bug#916303: steam: installs to ~/.steam, which is not as intended by upstream

[Simon McVittie]

On Wed, 12 Dec 2018 at 20:39:00 +0000, Simon McVittie wrote:
> The steam package in Debian has traditionally unpacked the Steam
> bootstrapper directly into ~/.steam, which gives that directory a dual
> role: the installation root, and also the symlink farm.

In this email I'll refer to the installation root as $STEAMDIR and
the symlink farm as $STEAMCONFIG (those are the names used in Valve's
/usr/bin/steam). The current /usr/games/steam in Debian uses $config to
represent both; my proposed patches used $root and $config respectively
(but I might revise them to use the same names that Valve do, in an
attempt to make the situation easier to understand).

Michael Gilbert commented in a private email that the intention of the
current script's conflation of $STEAMDIR and $STEAMCONFIG was to reduce
clutter in the user's home directory by redirecting everything into a
single location, and said that it would only be acceptable to switch
away from that if the Steam client was changed so that it used the XDG
basedir specification for everything.

Michael, I would like to ask you to reconsider this, from several angles:

- Steam can't stop using ~/.steam for $STEAMCONFIG because that would be
  an API/ABI break
- The current installation layout breaks functionality
- The proposed installation layout does not result in any more clutter
  than the one that is currently implemented, so there is no detriment
  from that point of view

API/ABI implications of $STEAMCONFIG
====================================

The Steam client is a proprietary binary, so we can't change its
behaviour: we can only change the behaviour of the Debian-specific
wrapper, or ask Valve to change the behaviour of the proprietary bits
(but they are happy with how it currently works in the .deb that they
provide, and there is little or no benefit to them from accommodating
our requests, so we'd need a really compelling reason for them to be
willing to change it).

My understanding is that Valve also *can't* change their use of ~/.steam,
because they're constrained what they have historically offered as an
API/ABI - similar to the way Debian needs to continue to have /lib64
even though we'd prefer not to, because it's part of the x86_64 ELF ABI,
and we have to contiue to provide /etc/rmt even though it makes little
sense, because that was also historically API "surface area" for tar.

If I'm understanding what Valve developers have said correctly, the split
between $STEAMCONFIG and $STEAMDIR has two purposes:

* The existence of some (most? all?) of the following symlinks and files
  under ~/.steam is a filesystem-based API "surface area" for
  the Steamworks API, in much the same way that /etc/machine-id,
  /usr/lib/os-release, /run/systemd etc. are filesystem-based APIs
  for systemd:

  - ~/.steam/bin -> bin32
  - ~/.steam/bin32 -> $STEAMDIR/ubuntu12_32
  - ~/.steam/bin64 -> $STEAMDIR/ubuntu12_64
  - ~/.steam/registry.vdf
  - ~/.steam/root -> $STEAMDIR
  - ~/.steam/sdk32 -> $STEAMDIR/linux32
  - ~/.steam/sdk64 -> $STEAMDIR/linux64
  - ~/.steam/steam -> $STEAMDIR
  - ~/.steam/steam.pid
  - ~/.steam/steam.pipe

  I haven't yet been able to get a canonical list of how much of that
  is API, but at least some of it is. These are used by the Steamworks
  binary blob, which if I understand correctly is bundled in games and
  cannot be replaced by updating the Steam client; so the files on disk
  have to stay as they are to avoid breaking older games, and this would
  remain true even if newer versions of the Steamworks binary blob used
  $XDG_DATA_HOME (or a different environment variable) to locate Steam.

  Having Steam continue to populate ~/.steam, so that games can
  dereference the symlinks to find Steam, is analogous to having
  Apache continue to use ~/public_html and having sshd continue to use
  ~/.ssh/authorized_keys: we need to continue to do what works, because
  if we change it, then third-party software relying on the current path
  stops working.

* Having the indirection through ~/.steam also means that if a user
  changes their XDG_DATA_HOME without moving the files from their old
  $XDG_DATA_HOME/Steam to the new location, their Steam installation
  doesn't disappear or cease to work, causing all their games to appear to
  vanish. I don't personally think this particular reason is particularly
  compelling (because changing XDG_DATA_HOME without relocating the files
  is surely user error, and no other XDG basedir implementation tries to
  cope with users who do that), but if ~/.steam needs to exist for the
  Steamworks API *anyway*, then it's a reasonable way for Valve to avoid
  tech support burden from users who have broken their installations in
  that way.

There might be other reasons, although I'm not currently aware of any
that would be relevant to Debian. (I think Valve use ~/.steam/steambeta
for parallel-installation of internal betas, which wouldn't work with
our directory layout, but Valve aren't going to use our packaging for
their internal betas anyway.)

Current layout breaking functionality
=====================================

Having $STEAMCONFIG and $STEAMDIR be identical causes collisions, because
each is designed to have its own internal structure, and unfortunately
both internal structures include a member named "steam" with different
purposes.

As designed, $STEAMDIR/steam is a real directory (not a symlink) which
is meant to contain cached resources for the Steam client, among other
things.

As designed, $STEAMCONFIG/steam is meant to be a symlink to
$STEAMDIR. However, the Steam client cannot create that symlink (which
it does on startup) if $STEAMCONFIG = $STEAMDIR and $STEAMDIR/steam has
already been unpacked, which is true in practice.

As a result of the symlink, parts of the Steam client and the Steamworks
API can normally read and write $STEAMDIR/foo, $STEAMCONFIG/root/foo
and $STEAMCONFIG/steam/foo interchangeably. However, because the
Debian-specific directory layout can't create the STEAMCONFIG/steam
symlink, those paths are not interchangeable on Debian. If modules are
inconsistent about which one it uses, then we can easily get a situation
where a component writes to $STEAMCONFIG/steam/foo (~/.steam/steam/foo
on Debian) and the same or a different component reads $STEAMDIR/foo
(~/.steam/foo on Debian), or vice versa, resulting in data loss.

In open source software, this would already be bad, but given enough bug
reports, in principle we could fix all the occurrences of inconsistent
paths. However, in proprietary software controlled by multiple parties
(the Steam client and games), we can't do that, because a lot of it is
outside our control.

If you insist on continuing to use a root inside ~/.steam, we could
address this problem without moving the root outside ~/.steam by
using a *subdirectory* of ~/.steam for the $STEAMDIR, for example
~/.steam/debian-installation-path. However, it would have to be a
subdirectory that Valve will never use for a different purpose, and the
~/.steam "namespace" is under their control. If we ask them to reserve
a directory for our use, they're most likely to just tell us to do what
I proposed here (or use their own /usr/bin/steam wrapper, which in this
respect is more or less equivalent to what I proposed) instead of doing
our own thing.

Amount of clutter remains constant
==================================

I don't think clutter in the home directory is as important a
consideration as "it has to work". However, while it's true that the
proposed paths are not a net reduction in the amount of clutter, they
are not a net increase either.

In the current Debian implementation of /usr/games/steam, a user's
home directory will normally contain the XDG basedirs (~/.local/share,
~/.config, ~/.cache), plus ~/.steam, plus any paths hard-coded by
unrelated software like bash.

In the proposed patch on this bug, a user's home directory will
normally contain the XDG basedirs, plus ~/.steam, plus any paths
hard-coded by unrelated software - the same. The difference is that
for new installations ~/.steam will be a lot smaller, and its internal
structure will be different.

(The result of installing Valve's Steam client .deb is the same again:
no more or less clutter than ours.)

---

I hope these various factors persuade you that the patches I attached,
or something similar, would be a net improvement. If you would like
more information about any of these points, please let me know, and I will
try to get more information from Valve developers if it would be helpful.

Thanks,
    smcv