[luatex] Luaotfload still cannot embed several new CJK fonts in Mac OS X 10.11 El Capitan

[Kazuki Maeda]

Hi all.

Mac OS X 10.11 El Capitan provides many new CJK fonts as OpenType Collection (but extension is .ttc).
LuaTeX 0.81.0 can handle these new fonts (see rev 5330).
However, luaotfload still cannot embed several fonts, e.g. HiraginoSans-W1, with "Invalid TTC index number" error.

The cause of the problem is that this font does not have subfonts whose PostScript name is same as "fontname" of the font file.
For HiraginoSans-W1, luatex fontloader returns:

...
["familyname"]="Hiragino Sans W1",
["fontname"]="HiraKakuProN-W1",
["fontstyle_id"]=0,
["fullname"]="HiraKakuProN-W1",
...

This font file has two subfonts. Their PostScript name are:

...
["family"]=".Hiragino Kaku Gothic Interface",
["fullname"]=".Hiragino Kaku Gothic Interface W1",
["manufacturer"]="SCREEN Graphic and Precision Solutions Co., Ltd.",
["postscriptname"]=".HiraKakuInterface-W1",
["preffamilyname"]=".Hiragino Kaku Gothic Interface",
...
["family"]="Hiragino Sans W1",
["fullname"]="Hiragino Sans W1",
["manufacturer"]="SCREEN Graphic and Precision Solutions Co., Ltd.",
["postscriptname"]="HiraginoSans-W1",
["preffamilyname"]="Hiragino Sans",
...

There is no subfont whose postscriptname is "HiraKakuProN-W1".
But luaotfload sets fontname "HiraKakuProN-W1" to psname, and causes "Invalid TTC index number" error.


I attach a patch to luaotfload (or font-otf.lua).
This code reads postscriptname, writes it to font cache as metadata.psname, and sets it to psname.
After applying this patch, you will be able to embed all the new CJK fonts.


Best regards
Kazuki Maeda

[Élie Roux]

Dear Kazuki Maeda,

Thank you very much for your bug report and patch. I've reported it on
luaotfload tracker: https://github.com/lualatex/luaotfload/issues/289
and will ask Hans about it (he's the author of the upstream ConTeXt
code). Is one of these fonts publicly available? If not, could you send
it to me privately?

Best Regards,

[Yusuke Terada]

Dear Elie,

I am Yusuke Terada, a friend of Kazuki's.
I found the bug and asked him to report it to this mailing list.
Please let me add some more information about it.

Here are the paths of fonts which cause the bug:

/System/Library/Fonts/ヒラギノ角ゴシック W0.ttc
/System/Library/Fonts/ヒラギノ角ゴシック W1.ttc
/System/Library/Fonts/ヒラギノ角ゴシック W2.ttc
/System/Library/Fonts/ヒラギノ角ゴシック W4.ttc
/System/Library/Fonts/ヒラギノ角ゴシック W5.ttc
/System/Library/Fonts/ヒラギノ角ゴシック W7.ttc
/System/Library/Fonts/ヒラギノ角ゴシック W9.ttc
/Library/Fonts/Klee.ttc
/Library/Fonts/TsukushiAMaruGothic.ttc
/Library/Fonts/TsukushiBMaruGothic.ttc

Post by Élie Roux
Is one of these fonts publicly available? If not, could you send
it to me privately?

I'm afraid they are not freely available and cannot be sent, because they are commercial fonts and sending them violates Apple's EULA.
If you or someone else has OS X 10.11 El Capitan, please try this source:

\documentclass{article}
\usepackage{fontspec}
\font\hiragino=HiraginoSans-W1 % PostScript name of a subfont of "ヒラギノ角ゴシック W1.ttc"
\begin{document}
\hiragino あ
\end{document}

If you compile this by LuaTeX 0.81.0, you will get the "Invalid TTC index number" error.

Sincerely,

Yusuke Terada

Post by Élie Roux
Dear Kazuki Maeda,
Thank you very much for your bug report and patch. I've reported it on
https://github.com/lualatex/luaotfload/issues/289
and will ask Hans about it (he's the author of the upstream ConTeXt
code). Is one of these fonts publicly available? If not, could you send
it to me privately?
Best Regards,
--
Elie

[Élie Roux]

Dear Yusuke Terada,

Post by Yusuke Terada
I'm afraid they are not freely available and cannot be sent, because
they are commercial fonts and sending them violates Apple's EULA.

I know that, but none of the people who can work on this have OSX, so
the EULA will have to be broken one way or another if the bug has to be
fixed.

Does someone on the list have access to these fonts?

Thank you,

[Hans Hagen]

Post by Élie Roux
Dear Yusuke Terada,

Post by Yusuke Terada
I'm afraid they are not freely available and cannot be sent, because
they are commercial fonts and sending them violates Apple's EULA.

So we cannot support apple/osx then.

Post by Élie Roux
I know that, but none of the people who can work on this have OSX, so
the EULA will have to be broken one way or another if the bug has to be
fixed.
Does someone on the list have access to these fonts?

Indeed. No font, no fix ... (my mac is way too old, cannot be updated to
a newer osx, and i'm not going to buy a new one for testing).

Hans


-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: [phone] | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------

[Wolfgang Schuster]

8. Oktober 2015 um 22:40
Dear Yusuke Terada,
I know that, but none of the people who can work on this have OSX, so
the EULA will have to be broken one way or another if the bug has to be
fixed.
Does someone on the list have access to these fonts?

You can use the Source Han Sans font for test because it is also
available in the OTC format.

https://github.com/adobe-fonts/source-han-sans/releases

Wolfgang

[Hans Hagen]

Post by Wolfgang Schuster

8. Oktober 2015 um 22:40
Dear Yusuke Terada,
I know that, but none of the people who can work on this have OSX, so
the EULA will have to be broken one way or another if the bug has to be
fixed.
Does someone on the list have access to these fonts?

You can use the Source Han Sans font for test because it is also
available in the OTC format.
https://github.com/adobe-fonts/source-han-sans/releases

those fonts work ok (at my end), the problem probably relates to
filenames as well as fontnames in some non-latin encoding

Hans

-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: [phone] | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------

[Wolfgang Schuster]

9. Oktober 2015 um 00:03
those fonts work ok (at my end), the problem probably relates to
filenames as well as fontnames in some non-latin encoding

The normal ttc-files (with separate files for each style) work but the
file in the folder SuperOTC doesn’t.

The following example

\starttext
\definedfont[file:SourceHanSans(SourceHanSans-Bold)]Source Han Sans Bold
\stoptext

results in this error:

!LuaTeX error (file /Users/wolf/Library/Fonts/SourceHanSans.ttc): sfnt:
table not found...

Wolfgang

[Hans Hagen]

Post by Wolfgang Schuster

9. Oktober 2015 um 00:03
those fonts work ok (at my end), the problem probably relates to
filenames as well as fontnames in some non-latin encoding

The normal ttc-files (with separate files for each style) work but the
file in the folder SuperOTC doesn’t.
The following example
\starttext
\definedfont[file:SourceHanSans(SourceHanSans-Bold)]Source Han Sans Bold
\stoptext
table not found...

that is ok in luatex 0.81

Hans


-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: [phone] | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------

[Kazuki Maeda]

On Fri, 9 Oct 2015 00:03:29 +0200

Post by Hans Hagen

Post by Wolfgang Schuster
You can use the Source Han Sans font for test because it is also
available in the OTC format.
https://github.com/adobe-fonts/source-han-sans/releases

those fonts work ok (at my end), the problem probably relates to
filenames as well as fontnames in some non-latin encoding

SourceHanSans-Medium.ttc has the following metadata:

...
["fontname"]="SourceHanSans-Medium",
...

and four subfonts:

(0):
["postscriptname"]="SourceHanSans-Medium",
(1):
["postscriptname"]="SourceHanSansK-Medium",
(2):
["postscriptname"]="SourceHanSansSC-Medium",
(3):
["postscriptname"]="SourceHanSansTC-Medium",

In this case, ("postscriptname" of the subfont 0) == ("fontname" of SourceHanSans-Medium.ttc).
Therefore, "Invalid TTC index number" error does not occur.
If you delete subfont 0 from the ttc, you may get the error.


Best regards
Kazuki Maeda

[Yusuke Terada]

I've made a font for testing on Kazuki's advice using AFDKO.

You can download it from here:
https://dl.dropboxusercontent.com/u/5807100/SourceHanSans-Medium-Reduced.ttc

If you compile the attached luaotfload_test.tex with plain-LuaTeX 0.81.0, you will get this error:
<./SourceHanSans-Medium-Reduced.ttc(SourceHanSans-Medium:-1)Invalid TTC index number

After applying the patch Kazuki submitted yesterday to fontloader, this source can be compiled correctly.

Sincerely,

Yusuke Terada

[Hironobu Yamashita]

Hi all,

I think Kazuki says:

It is true that **most of** the collection fonts available in the world have a subfont
whose postscript name is exactly same as its fontname, however, it is NOT true
that **all of** the collection fonts do.

And he is right: some of the examples are Apple's new CJK fonts and what Yusuke
has sent to us. And the patch Kazuki has sent works fine, I think.

-----
Hironobu Yamashita

Post by Yusuke Terada
I've made a font for testing on Kazuki's advice using AFDKO.
https://dl.dropboxusercontent.com/u/5807100/SourceHanSans-Medium-Reduced.ttc
<./SourceHanSans-Medium-Reduced.ttc(SourceHanSans-Medium:-1)Invalid TTC index number
After applying the patch Kazuki submitted yesterday to fontloader, this source can be compiled correctly.
Sincerely,
Yusuke Terada
<luaotfload_test.tex>

Post by Kazuki Maeda
...
["fontname"]="SourceHanSans-Medium",
...
["postscriptname"]="SourceHanSans-Medium",
["postscriptname"]="SourceHanSansK-Medium",
["postscriptname"]="SourceHanSansSC-Medium",
["postscriptname"]="SourceHanSansTC-Medium",
In this case, ("postscriptname" of the subfont 0) == ("fontname" of SourceHanSans-Medium.ttc).
Therefore, "Invalid TTC index number" error does not occur.
If you delete subfont 0 from the ttc, you may get the error.
Best regards
Kazuki Maeda

[Hans Hagen]

Post by Hironobu Yamashita
It is true that **most of** the collection fonts available in the world have a subfont
whose postscript name is exactly same as its fontname, however, it is NOT true
that **all of** the collection fonts do.

I've uploaded a new context beta that now looks at the postscriptname
(but with some extra checking and in the same spot where fontname and
fullname are checked).

The ps names are normally used in the backend for unique font
identifiers btu are also used by the backend for ttc lookups. So, if
different fonts have the same ps fontnames the result might be
unexpected. Also, in an upcoming luatex patch we will fallback on index
0 + warning which is in most cases quite ok (at least some output).

Wolfgang: the new context font code already had that ps name resolving
so now both methods are more or less similar in that respect, although
we can have subtle differences in font/fullnames.

Hans



-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: [phone] | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------

[E'lie Roux]

Dear Hans,

Post by Hans Hagen
I've uploaded a new context beta that now looks at the
postscriptname (but with some extra checking and in the same spot
where fontname and fullname are checked).

Thanks a lot, I'll try to update luaotfload with the new beta.

Thank you,

[Kazuki Maeda]

Dear Hans

Post by Hans Hagen
I've uploaded a new context beta that now looks at the postscriptname
(but with some extra checking and in the same spot where fontname and
fullname are checked).

Thank you very much! I will try the updated version as soon as possible.

Best regards
Kazuki Maeda

[Hironobu YAMASHITA]

​​Dear Hans,

Post by Hans Hagen
I've uploaded a new context beta that now looks at the postscriptname
(but with some extra checking and in the same spot where fontname and
fullname are checked).

Thank you very much for new context beta.
I've tested it with following source, using SourceHanSans-Medium.ttc.
I removed all the other private font files from my TEXMF tree, and deleted
all cache files which had already existed.

==========
\definefont[xxx][name:SourceHanSansSCMedium]
\definefont[yyy][name:SourceHanSansTCMedium]
\definefont[zzz][name:SourceHanSansMedium]
\definefont[www][name:SourceHanSansKMedium]
\starttext
{\zzz Using SourceHanSans-Medium.ttc (OTC: OpenType Collection)}

\xxx 骚\yyy 骚\zzz 骚\www 骚

\xxx 曜\yyy 曜\zzz 曜\www 曜
\stoptext
==========

In version 2015.10.07 12:03, four different characters were printed
correctly. However, in version 2015.10.09 10:59, all four characters
were printed in "SourceHanSansMedium" form.
Sample source, pdfs, and logs are available at:
https://dl.dropboxusercontent.com/s/xrflnjyovslxjbo/contextbeta-test.zip?dl=0

I checked the cache files, and all of them have the same definitions:
["familyname"]="Source Han Sans",
["fontname"]="SourceHanSans-Medium",
["fullname"]="Source Han Sans Medium",
This seems to be an unexpected result...

Regards,
Hironobu

[Yusuke Terada]

Dear Hans,

mtx-context | current version: 2015.10.09 21:28

I've confirmed that Hironobu's test source posted at http://tug.org/pipermail/luatex/2015-October/005396.html can be compiled correctly.

Moreover, most of the new CJK fonts of El Capitan can be loaded.
Here is the test source:

%%%%%%%%%%%
\definefont[xxxa][name:HiraginoSans-W0]
\definefont[xxxb][name:HiraginoSans-W1]
\definefont[xxxc][name:HiraginoSans-W2]
\definefont[xxxd][name:HiraginoSans-W3]
\definefont[xxxe][name:HiraginoSans-W4]
\definefont[xxxf][name:HiraginoSans-W5]
\definefont[xxxg][name:HiraginoSans-W6]
\definefont[xxxh][name:HiraginoSans-W7]
\definefont[xxxi][name:HiraginoSans-W8]
\definefont[xxxj][name:HiraginoSans-W9]
\definefont[yyya][name:Klee-Medium]
\definefont[yyyb][name:Klee-Demibold]
%\definefont[zzza][name:TsukuARdGothic-Regular]
%\definefont[zzzb][name:TsukuARdGothic-Bold]
%\definefont[zzzc][name:TsukuBRdGothic-Regular]
%\definefont[zzzd][name:TsukuBRdGothic-Bold]

\starttext

\xxxa ヒラギノ角ゴシック Ｗ０\par
\xxxb ヒラギノ角ゴシック Ｗ１\par
\xxxc ヒラギノ角ゴシック Ｗ２\par
\xxxd ヒラギノ角ゴシック Ｗ３\par
\xxxe ヒラギノ角ゴシック Ｗ４\par
\xxxf ヒラギノ角ゴシック Ｗ５\par
\xxxg ヒラギノ角ゴシック Ｗ６\par
\xxxh ヒラギノ角ゴシック Ｗ７\par
\xxxi ヒラギノ角ゴシック Ｗ８\par
\xxxj ヒラギノ角ゴシック Ｗ９\par
\yyya クレー ミディアム\par
\yyyb クレー デミボールド\par
%\zzza 筑紫Ａ丸ゴシック レギュラー\par
%\zzzb 筑紫Ａ丸ゴシック ボールド\par
%\zzzc 筑紫Ｂ丸ゴシック レギュラー\par
%\zzzd 筑紫Ｂ丸ゴシック ボールド\par

\stoptext
%%%%%%%%%%%

Invalid TTC index number -1 (0..1), using index 0 for font TsukuARdGothic-Regular

These fonts (TsukushiAMaruGothic.ttc and TsukushiBMaruGothic.ttc) are, however, not so essential for Japanese typesetting that fixing this issue is an urgent matter.
I am very happy to see most of the new CJK fonts handled correctly with ConTeXt.
Thank you very much.

Sincerely,

Yusuke Terada

[Hironobu YAMASHITA]

​Dear Hans,

Sorry for my late response,

Post by Hironobu YAMASHITA
In version 2015.10.07 12:03, four different characters were printed
correctly. However, in version 2015.10.09 10:59, all four characters
were printed in "SourceHanSansMedium" form.

the problem was gone. I could get what I expected.

Though Yusuke has reported "Invalid TTC index number -1" error in some
fonts (TsukushiAMaruGothic.ttc and TsukushiBMaruGothic.ttc), I think
this is rather a trivial issue.

Thanks a lot!
Hironobu​

[Hans Hagen]

​Dear Hans,
Sorry for my late response,

Post by Hironobu YAMASHITA
In version 2015.10.07 12:03, four different characters were printed
correctly. However, in version 2015.10.09 10:59, all four characters
were printed in "SourceHanSansMedium" form.

the problem was gone. I could get what I expected.
Though Yusuke has reported "Invalid TTC index number -1" error in some
fonts (TsukushiAMaruGothic.ttc and TsukushiBMaruGothic.ttc), I think
this is rather a trivial issue.

when a subfont can't be found by name (it's on my list to see if we can
just pass the subfont index directly) luatex will now fallback to index
0 (so no -1 abort but a warning)

Hans

-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: [phone] | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------

[Hans Hagen]

Post by Yusuke Terada
I've made a font for testing on Kazuki's advice using AFDKO.
https://dl.dropboxusercontent.com/u/5807100/SourceHanSans-Medium-Reduced.ttc
<./SourceHanSans-Medium-Reduced.ttc(SourceHanSans-Medium:-1)Invalid TTC index number
After applying the patch Kazuki submitted yesterday to fontloader, this source can be compiled correctly.

that patch is not ok (it can result in empty ps names for other fonts
and is also applied in the wrong place in the code) ... keep in mind
that what works for one font can fail for another (and anything you can
imagine that goes wrong with names/string, will go wrong with fonts)

Hans

-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: [phone] | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------

[Kazuki Maeda]

Dear Hans

Post by Hans Hagen
and is also applied in the wrong place in the code

Ahhh, I see. I have written the code in very strange place (between report_otf() and containers.write())...
In addition, such a process should be executed by enhancers?
Where should I write the code in?

Post by Hans Hagen
it can result in empty ps names for other fonts

I agree with you. But, in fact, several new commercial fonts cause the problem as I wrote.
I would be happy if we could find a good solution to the problem.


Best regards
Kazuki Maeda