[Pkg-fonts-devel] Bug#974537: Bug#974537: fonts-noto-core: Fallback font selection changed and incorrect glyph displayed

Thu Nov 12 15:58:00 GMT 2020

Jonas Smedegaard:
> Hi astian,
>
> Thanks for a detailed bugreport!
>
> Quoting astian (2020-11-11 21:31:00)
>> With version 20200323-1, when attempting to render code points such as
>> 0x3001 and 0x3002, fontconfig would choose "Noto Sans CJK JP" [0] as
>> fallback for "Monospace".  This was expected behaviour, I want to see
>> Japanese punctuation glyphs.
>>   0: /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
>>
>> Binary packages for version 20200323-1 seem to be gone from the archive
>> but version 20181227-1 also shows the wanted behaviour.
>
> Some are here: https://snapshot.debian.org/binary/fonts-noto-core/

I forgot about that site.  Unfortunately the APT package lists in
stable/testing/unstable no longer have a record of that version and
consequently I no longer have a trusted path to those files.

>> After updating to version 20201027-3 and later also 20201109-1,
>> fontconfig chooses "Noto Sans Mongolian" [1].  This results in
>> unintended glyphs.
>>   1: /usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf
>
> I understand that this changed.  But is it a bug?  I mean, is it
> universally preferred to use Japanese over Mongolian for those
> characters?

Universally?  How could I say?  I know nothing about Mongolian, but
taking U+3001 as example, it's described as "ideographic comma" and
fontconfig (FC) recognises it as part of the charset of various CJK
languages (see for example "fc-lang/{ja,ko,zh_tw,zh_hk}.orth" in FC
sources).  Interestingly, FC does not think it is part of mn_cn (see
"fc-lang/mn_cn.orth").

It seems an odd change to me.  Here are some cases I can imagine:

  - No bug in fonts-noto-core, no technical bug in FC, no policy bug in
    FC: I should explicitly configure FC to prioritise a Japanese
    fallback font.  I should still want an explanation for why is
    Mongolian preferred.

  - No bug in fonts-noto-core, no technical bug in FC, but policy bug in
    FC: Lacking explicit configuration, a Japanese (or some other CJK
    language) font should be generally preferred over Mongolian.

  - No bug in fonts-noto-core, but technical bug in FC: The changes in
    fonts-noto-core have exposed a bug that causes FC to incorrectly
    change the fallback selection in favour of this Mongolian font.

  - Bug in fonts-noto-core: Either the glyphs don't belong or there is
    some incorrect metadata or configuration, or bad interaction with
    other fonts, or any combination thereof, which causes FC to
    correctly (in technical terms) change the fallback selection in
    favour of this Mongolian font.

  - Bug in fonts-noto-cjk: As above.

I don't currently have the expertise to decide myself and at present
don't have the time to investigate in depth.  Maybe a FC expert's
opinion could help.

>> STR:
>>
>>   a) Run:
>>        $ LANG=en_US.UTF-8 pango-view --font monospace -t $'\u3001'
>>      Or even:
>>        $ LANG=en_US.UTF-8 pango-view -t $'\u3001'
>
> Oh, that's a neat way to render unicode characters, didn't know that
> one.
>
> Failed for me at first, however, so here is another little trick: locale
> en_US.UTF-8 is not generally enabled, but locale C.UTF-8 is.

Right.  I used "en_US" because I wanted an invariable locale, otherwise
the test results would be unreliable because FC determines the default
language from the user's locale and that can affect its decisions.  I
briefly looked at the source now and I see that C works too, it also
becomes "en".  Thanks.

>> Expected behaviour:
>>
>>   a) The pango-view window shows the Japanese comma glyph (see for
>>      example "Noto Sans CJK JP" in fontforge).
>>
>>   b) A Japanese font is preferred:
>>        Noto Sans CJK JP:style=Regular:file=/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
>>        Noto Sans Mongolian:style=Regular:file=/usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf
>>
>> Actual behaviour:
>>
>>   a) The pango-view window shows a different glyph (from "Noto Sans
>>      Mongolian").
>>
>>   b) A Mongolian font is preferred:
>>        Noto Sans Mongolian:style=Regular:file=/usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf
>>        Noto Sans CJK JP:style=Regular:file=/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
>
> When I try the above with packages in unstable as of today, I get what
> looks to me as the comma glyph, even though fc-cache indeed shows
> Mongolian as prioritized.

Here's a Japanese comma (base64, PNG):

  iVBORw0KGgoAAAANSUhEUgAAACQAAAAsCAIAAACCMY2iAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA
  i0lEQVRYw2P8//8/A70AEwMdwahlo5aNWjZq2ahlo5aNWjZq2ahlo5ZRw7IfNxfEOoRNu/SVHpax
  CcrI/NyT5+zVdOQ95e30/wTBv4+nu13FWARtmk98+vefEsBAlKp/H080WvGzaVec/EF7y/7////3
  xZmjt75R5LH/jKNdplHLRi0btWzUsoG1DADyRkmF4+MjugAAAABJRU5ErkJggg==

Here's what I get instead (base64, PNG):

  iVBORw0KGgoAAAANSUhEUgAAABsAAAAxCAIAAACkgUVSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA
  d0lEQVRIx2P8//8/A1UBEwO1waiJoyaOmjhq4qiJoyaOmjjQJv79+fMPFU3893R9mrFR+sYX//Cp
  +k80+Hau1Ypf0DRv/f2f+JQRbeLf2/328q49Zz4TUshIfEvq58cP//gFOAkpYxxtm42aOGriqImj
  Jo6aCAcAgcywLcj1CDYAAAAASUVORK5CYII=

Indeed it looks like a comma, but not like a typical Japanese one.

To compare the fonts you can also try fontforge or gucharmap (tick
"View->Show only glyphs from this font").

>> This looks like a regression and it is one for me, but I guess it
>> could also be a configuration issue involving fontconfig.  I have no
>> custom fontconfig configuration, though, so if somehow this is not
>> considered a regression, perhaps you could recommend a configuration
>> that would restore the previous behaviour for me?
>
> Sorry, I am not clever with fontconfig and the fonts-noto-core package
> includes only a small configuration related to older name Droid:
> /etc/fonts/conf.avail/30-droid-noto.conf
>
> I notice that package fonts-noto-cjk ships a more extensive
> configuration seemingly related to identifying as "monospace":
> /usr/share/fontconfig/conf.avail/70-fonts-noto-cjk.conf
>
> Perhaps it helps to edit that CJK configuration to add binding="strong"
> also to the monospace sections?
>
> Please to report back if that helps

I tried the suggestion.  It did not affect the fallback selection.

I tried blacklisting "Noto Sans Mongolian" and that did work, as can be
expected but it's a very heavy-handed approach.  Unfortunately, I
do not yet grok FC configuration to a level that would allow me to
state what I really want:

  For code points belonging to the charset of the 'ja' language, always
  prioritize a font designed for that language.

> and whether or not you think this
> is universally a preferred setup or we should perhaps introduce some
> flexibility in these packages - i.e. a mechanism to let Mongolians
> prioritize their glyphs and let Japansese prioritize theirs.

Please see above.

Thanks.