[Pkg-fonts-devel] Bug#974537: Bug#974537: fonts-noto-core: Fallback font selection changed and incorrect glyph displayed

[Jonas Smedegaard]

Quoting astian (2020-11-12 16:58:00)
> Jonas Smedegaard:
> > Quoting astian (2020-11-11 21:31:00)
> >> With version 20200323-1, when attempting to render code points such 
> >> as 0x3001 and 0x3002, fontconfig would choose "Noto Sans CJK JP" 
> >> [0] as fallback for "Monospace".  This was expected behaviour, I 
> >> want to see Japanese punctuation glyphs.
> >>   0: /usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
> >>
> >> Binary packages for version 20200323-1 seem to be gone from the 
> >> archive but version 20181227-1 also shows the wanted behaviour.
> >
> > Some are here: https://snapshot.debian.org/binary/fonts-noto-core/
> 
> I forgot about that site.  Unfortunately the APT package lists in 
> stable/testing/unstable no longer have a record of that version and 
> consequently I no longer have a trusted path to those files.
> 
> >> After updating to version 20201027-3 and later also 20201109-1, 
> >> fontconfig chooses "Noto Sans Mongolian" [1].  This results in 
> >> unintended glyphs.
> >>   1: /usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf
> >
> > I understand that this changed.  But is it a bug?  I mean, is it 
> > universally preferred to use Japanese over Mongolian for those 
> > characters?
> 
> Universally?  How could I say?  I know nothing about Mongolian, but 
> taking U+3001 as example, it's described as "ideographic comma" and 
> fontconfig (FC) recognises it as part of the charset of various CJK 
> languages (see for example "fc-lang/{ja,ko,zh_tw,zh_hk}.orth" in FC 
> sources).  Interestingly, FC does not think it is part of mn_cn (see 
> "fc-lang/mn_cn.orth").

How you could say?  Well, you just did. :-)


> It seems an odd change to me.  Here are some cases I can imagine:
> 
>   - No bug in fonts-noto-core, no technical bug in FC, no policy bug 
>     in FC: I should explicitly configure FC to prioritise a Japanese 
>     fallback font.  I should still want an explanation for why is 
>     Mongolian preferred.
> 
>   - No bug in fonts-noto-core, no technical bug in FC, but policy bug 
>     in FC: Lacking explicit configuration, a Japanese (or some other 
>     CJK language) font should be generally preferred over Mongolian.
> 
>   - No bug in fonts-noto-core, but technical bug in FC: The changes in 
>     fonts-noto-core have exposed a bug that causes FC to incorrectly 
>     change the fallback selection in favour of this Mongolian font.
> 
>   - Bug in fonts-noto-core: Either the glyphs don't belong or there is 
>     some incorrect metadata or configuration, or bad interaction with 
>     other fonts, or any combination thereof, which causes FC to 
>     correctly (in technical terms) change the fallback selection in 
>     favour of this Mongolian font.
> 
>   - Bug in fonts-noto-cjk: As above.
> 
> I don't currently have the expertise to decide myself and at present 
> don't have the time to investigate in depth.  Maybe a FC expert's 
> opinion could help.

Thanks for lining up those possibilities.

Yeah, gathering some fontconfig experts could help clarify this.

Or the more lazy option of simply wait some time, grab another snapshot 
of Noto, and see if the problem goes away... :-P


> >> Expected behaviour:
> >>
> >>   a) The pango-view window shows the Japanese comma glyph (see for
> >>      example "Noto Sans CJK JP" in fontforge).
> >>
> >>   b) A Japanese font is preferred:
> >>        Noto Sans CJK JP:style=Regular:file=/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
> >>        Noto Sans Mongolian:style=Regular:file=/usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf
> >>
> >> Actual behaviour:
> >>
> >>   a) The pango-view window shows a different glyph (from "Noto Sans
> >>      Mongolian").
> >>
> >>   b) A Mongolian font is preferred:
> >>        Noto Sans Mongolian:style=Regular:file=/usr/share/fonts/truetype/noto/NotoSansMongolian-Regular.ttf
> >>        Noto Sans CJK JP:style=Regular:file=/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc
> >
> > When I try the above with packages in unstable as of today, I get 
> > what looks to me as the comma glyph, even though fc-cache indeed 
> > shows Mongolian as prioritized.
> 
> Here's a Japanese comma (base64, PNG):
> 
>   iVBORw0KGgoAAAANSUhEUgAAACQAAAAsCAIAAACCMY2iAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA
>   i0lEQVRYw2P8//8/A70AEwMdwahlo5aNWjZq2ahlo5aNWjZq2ahlo5ZRw7IfNxfEOoRNu/SVHpax
>   CcrI/NyT5+zVdOQ95e30/wTBv4+nu13FWARtmk98+vefEsBAlKp/H080WvGzaVec/EF7y/7////3
>   xZmjt75R5LH/jKNdplHLRi0btWzUsoG1DADyRkmF4+MjugAAAABJRU5ErkJggg==
> 
> Here's what I get instead (base64, PNG):
> 
>   iVBORw0KGgoAAAANSUhEUgAAABsAAAAxCAIAAACkgUVSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA
>   d0lEQVRIx2P8//8/A1UBEwO1waiJoyaOmjhq4qiJoyaOmjjQJv79+fMPFU3893R9mrFR+sYX//Cp
>   +k80+Hau1Ypf0DRv/f2f+JQRbeLf2/328q49Zz4TUshIfEvq58cP//gFOAkpYxxtm42aOGriqImj
>   Jo6aCAcAgcywLcj1CDYAAAAASUVORK5CYII=
> 
> Indeed it looks like a comma, but not like a typical Japanese one.
> 
> To compare the fonts you can also try fontforge or gucharmap (tick 
> "View->Show only glyphs from this font").

Thanks.  Not convenient for me to extract those images at the moment - 
may I will later...


> >> This looks like a regression and it is one for me, but I guess it 
> >> could also be a configuration issue involving fontconfig.  I have 
> >> no custom fontconfig configuration, though, so if somehow this is 
> >> not considered a regression, perhaps you could recommend a 
> >> configuration that would restore the previous behaviour for me?
> >
> > Sorry, I am not clever with fontconfig and the fonts-noto-core 
> > package includes only a small configuration related to older name 
> > Droid: /etc/fonts/conf.avail/30-droid-noto.conf
> >
> > I notice that package fonts-noto-cjk ships a more extensive 
> > configuration seemingly related to identifying as "monospace": 
> > /usr/share/fontconfig/conf.avail/70-fonts-noto-cjk.conf
> >
> > Perhaps it helps to edit that CJK configuration to add 
> > binding="strong" also to the monospace sections?
> >
> > Please to report back if that helps
> 
> I tried the suggestion.  It did not affect the fallback selection.
> 
> I tried blacklisting "Noto Sans Mongolian" and that did work, as can 
> be expected but it's a very heavy-handed approach.  Unfortunately, I 
> do not yet grok FC configuration to a level that would allow me to 
> state what I really want:
> 
>   For code points belonging to the charset of the 'ja' language, 
>   always prioritize a font designed for that language.

Let's leave this bugreport open, then.  Maybe someone comes by who knows 
more elegant fontconfig tricks, or one of us gains more knowledge later.

Also, I might split the fonts-noto package into smaller parts where it 
would be possible to simply omit the parts you don't want.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-fonts-devel/attachments/20201113/111c3d6e/attachment-0001.sig>