[Aptitude-devel] Bug#834402: Bug#834402: aptitude: search loses column format when redirected or piped

[Manuel A. Fernandez Montecelo]

Hi,

2016-08-16 19:18 Javier Cantero:
>On Mon, Aug 15, 2016 at 11:25:40PM +0100, Manuel A. Fernandez Montecelo wrote:
>> The reasoning for the change was that with pipes/redirections the
>> concept of "terminal" and consequently "width" is lost.  If it's
>> redirected it's possibly/likely that it's because it's moved and
>> processed elsewhere (where the new terminal size will likely be
>> different), or that the further processing doesn't bother with terminal
>> columns or uses smarter methods like using '|' as field separators.
>
>Counterexamples: any PAGER (more, less, ...), grep, sed, and virtually any
>filter can interact with the terminal in a later stage of the pipeline.
>In some cases like pagers there is *always* a terminal at the end.

That is the fundamental difference: the tool interacting with the
terminal is at the end, so the output is the terminal, so it can know
its dimensions.

If the tool is at the beginning, or at any point in the middle, the
output is a file or pipe (== a file for all purposes, from the tool's
POV), so it cannot know the dimensions.

For example, with a file with long lines such as apt's history.log:

  # less /var/log/apt/history.log > /tmp/test
  # md5sum /var/log/apt/history.log /tmp/test
  420a7acb08a4e58c9c6b97654aa1bb6a  /var/log/apt/history.log
  420a7acb08a4e58c9c6b97654aa1bb6a  /tmp/test

  # less --chop-long-lines /var/log/apt/history.log > /tmp/test
  # md5sum /var/log/apt/history.log /tmp/test
  420a7acb08a4e58c9c6b97654aa1bb6a  /var/log/apt/history.log
  420a7acb08a4e58c9c6b97654aa1bb6a  /tmp/test

  # less --chop-long-lines /var/log/apt/history.log | cat | md5sum
  420a7acb08a4e58c9c6b97654aa1bb6a  -


So they are exactly the same file.  "less" (at the start of the command
chain) doesn't insert newlines in the file or does any processing on the
input file with the "--chop-long-lines" option.  It treats the width of
the terminal as infinite.  This is what aptitude does now as well.


(Disclaimer: the tools or other tools that you mention perhaps still do
some trick to adapt to the terminal size even if the output is a
file/pipe, but well, my point is that the position and access to the
terminal is the fundamental difference between "cat | less" and
"aptitude | less"; and that "less" when not the last in the chain
behaves the same as the new behaviour of aptitude).


>> In other words, trying to columnize the output when the width is unknown
>> (pipes or redirections) is a bit hacky and doesn't make much sense to
>> me, and forcing it to be 80 for a lack of better default is not always a
>> good solution as it might have been back in ~2000 (I think).
>
>I agree that forcing to be 80 is hacky. But there is a better solution:
>if the output isn't to a terminal (and -w is not passed), write the
>entire output without truncating to any width size and let the next
>process in the pipeline deal with it. If it's the last process before
>going to terminal, I'm sure that program will have code to properly
>adapt its output to the actual terminal size. In one sentence: delegate
>the job to the process dealing with the terminal.

Actually, aptitude /now/ works as you say and for the same reasons that
you say, so we could be in violent agreement.  Only that I suspect that
we are not, because that's why you submitted the original bug report
after all... :)

So as you say, what aptitude does is that "if the output isn't to a
terminal (and -w is not passed), write the entire output without
truncating to any width size and let the next process in the pipeline
deal with it".  It does not truncate the output now when it's at the
start of a chain of commands.

Only that to columnize the output, it has to have some reference to the
width that it's going to "render" the output to, because most fields can
shrink and expand -- they are not of a fixed size.  And knowing the
width is exactly what it's lacking to columnize the output when
piping/redirecting.  If aptitude considers width as infinite, same as
"less", all rows are of infinite width (infinite being 65536 in the
implementation).

So the new behaviour is that, instead of columnizing to 80 and
truncating, it doesn't truncate (but doesn't columnize).


If it doesn't truncate and also doesn't have a width and doesn't want to
use the infinite, the only solution for columnizing would be to do two
passes: one collecting the maximum size of the columns for all packages
that match the search ("search" and other commands that use the same
columnizer code), and another pass to actually write the text and fill
with blanks until the max width of the column is filled in.

In this case, if using "aptitude search ${pattern} | less", searches
would have different widths -- the output would depend on the max length
of the fields for the packages that match.  Like this:

  |---------------------------------------------------------|
  i A aptitude an awesome package manager that everyone loves
  |---------------------------------------------------------|

  |----------------------------------------------------------------|
  i A aptitude        an awesome package manager that everyone loves
  i A aptitude-doc-en doc for an awesome package manager
  |----------------------------------------------------------------|

And the above is not usually a big problem if width it's narrower than
the terminal, but if not truncating and the width is wider than the
terminal, each row would "spill" into the next line, like this:

  |------------------------------------------------------------------
  -----|
  i A aptitude                       an awesome package manager that
  everyone loves
  i A aptitude-doc-en                doc for an awesome package manag
  er that everyone loves
  i A aptitude-plugin-with-long-name plugin for an awesome package ma
  nager that everyone loves
  |------------------------------------------------------------------
  -----|
  

...

Leaving all that aside.

So the possible solutions are:

- force a width (as it was done before, to 80) and truncate

- abandon columnization, do not truncate (what happens now)

- somehow try to use the same width of the current terminal, with
  possible truncation (would work fine in pipes for which the ultimate
  output is the same terminal, but not if redirected and processed/seen
  from another terminal/computer, unless the width is the same)

- don't truncate and columnize, with the result of double-passes (and
  consequent slowness) and the other sizing problems mentioned above


(in #1 and #2, the user can control it with -w if desired anyway, #3 and
#4 could maybe also use that)

I somehow am not convinced that any of the solutions is /much/ better
than the others, and now I kind of regret to have changed it from #1 to
#2 in the first place.

So I am reluctant to change it again because I don't see any clear
winner.  There are workarounds to tell exactly how to render if one uses
the output for further processing or for visualisation in external
tools, etc, or uses it quite heavily or is very picky about the
formatting for some other reason.


Cheers.
-- 
Manuel A. Fernandez Montecelo <manuel.montezelo at gmail.com>