Bug#854149: libvte-2.91-0: x-terminals not properly compose unicode characters

[Egmont Koblinger]

Hi,

This is a truly interesting bug.

If you highlight and copy-paste only the printed "1" or "2" digits, you'll
notice that "1" carries the combining strikethrough with it. This is one
possible way of making sure that VTE's belief about where that combining
accent is is correct. The problem "only" occurs when it displays it.

To make it more complicated, the behavior:
- depends on which regular characters are used instead of "1" and "2";
- depends on the combining accent
- depends on the font;
- is consistent with gedit.

Let's replace "1" and "2" with "a" and "b", as well as "x" and "y". Let's
also try '\u0301' (accent acute) in addition to the strikethrough.

echo -e '1\u03012'
echo -e 'a\u0301b'
echo -e 'x\u0301y'
echo -e '1\u03362'
echo -e 'a\u0336b'
echo -e 'x\u0336y'

and also redirect the output and open the file in gedit.

And let's try a couple of fonts:

DejaVu Sans Mono Oblique =>
Only the 2nd out of the 6 commands appear correctly ("a" with accent).
The rest: in gedit there's another cell in between containing the accent
only, in vte the accent is on top of the second letter.

FreeMono Regular, Noto Mono Regular, Tlwg Mono Regular, Ubuntu Mono Regular
=>
In both apps: The accent is always on the first character, as expected.

Liberation Mono Regular, Nimbus Mono L Regular =>
In vte: The accent is always on top of the second letter.
In gedit: The acute accent is always on top of the second letter. The
strikethrough is always in an additional cell between the two letters (as
DejaVu in 5 out of 6 cases).

Monospace Regular =>
Like DejaVu, except that the 3rd case also renders correctly (x with acute
accent).

Some conclusions:

It's broken in VTE if and only if broken in Gedit as well.

In Gedit it can be broken in two different ways; either the accent appears
over the second letter, or combines into a glyph that looks double wide
(but actually the cursor or mouse highlight jumps through it in a single
step) and contains the base letter in the left part and the combining one
in the right part. Which of these two broken variants occurs might depend
on the combining character itself.

Whether broken or not might depend on the base character and might depend
on the combining character too.

The only rendering difference between gedit and vte can be explained very
well. Gedit renders the entire flow in one step. VTE instead renders every
cell separately (putting into it the contents it wishes to see there, and
it knows correctly that these combining characters belong to the first
cell). Whatever is painted there might overflow out of the cell (and is
still painted there), but for the next character it's another painting step
started at the location VTE believes is the correct starting position. So
wherever gedit seems to paint three columns looking like "a-b", vte instead
renders "a-" beginning at column 1, and then paints "b" beginning at column
2, effectively looking as if the letter "b" was striked through.

For all the rest, it's a total mystery to me. I have no clue if it's all
these fonts that are buggy, or Gtk+'s font rendering has some fundamental
flow, or what else.

Okay, so let's check Konsole, KDE's terminal emulator build on top of Qt
and KDE libs. Konsole is notorious for not rendering each cell
independently, but rather a continuous run of text in a single step (as
most apps typically do except for terminal emulators).

KDE's behavior is almost the same as gedit's. The only exception I've found
was that with Monospace it rendered exactly as with Dejavu, that is, only
the accent on top of "a" was correct, the one on top of "x" wasn't (it was
on top of "y").

At this point my primary suspects are all these monospace fonts, although
there's sure much more to this story, e.g. there might be some mapping to
precomposed characters or whatnot...

It would be great to get someone heavily experienced in fonts involved in
this thread.

cheers,
egmont
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-gnome-maintainers/attachments/20170929/e749a7a8/attachment.html>