kc3-lang/freetype/src/gxvalid/README

Branch - branch -v2.13

• Show log

  Commit

• Hash : 65f85237
  Author :
  Date : 2023-01-17T09:18:25
  

  Update all copyright notices.

Download

• src/gxvalid/README

• gxvalid: TrueType GX validator
  ==============================
  
  
  1. What is this
  ---------------
  
    `gxvalid' is a module to  validate TrueType GX tables: a collection of
    additional tables  in TrueType  font which are  used by  `QuickDraw GX
    Text',  Apple Advanced  Typography  (AAT).  In  addition, gxvalid  can
    validates `kern'  tables which have  been extended for AAT.   Like the
    otvalid  module,   gxvalid  uses   FreeType  2's  validator  framework
    (ftvalid).
  
    You can link gxvalid with your program; before running your own layout
    engine, gxvalid validates a font  file.  As the result, you can remove
    error-checking code  from the layout  engine.  It is also  possible to
    use  gxvalid  as a  stand-alone  font  validator;  the `ftvalid'  test
    program  included  in the  ft2demo  bundle  calls gxvalid  internally.
    A stand-alone font validator may be useful for font developers.
  
    This documents documents the following issues.
  
    - supported TrueType GX tables
    - fundamental validation limitations
    - permissive error handling of broken GX tables
    - `kern' table issue.
  
  
  2. Supported tables
  -------------------
  
    The following GX tables are currently supported.
  
      bsln
      feat
      just
      kern(*)
      lcar
      mort
      morx
      opbd
      prop
      trak
  
    The following GX tables are currently unsupported.
  
      cvar
      fdsc
      fmtx
      fvar
      gvar
      Zapf
  
    The following GX tables won't be supported.
  
      acnt(**)
      hsty(***)
  
    The following undocumented tables in TrueType fonts designed for Apple
    platform aren't handled either.
  
      addg
      CVTM
      TPNM
      umif
  
  
    *)   The `kern'  validator handles both  the classic and the  new kern
         formats;  the former  is supported  on both  Microsoft  and Apple
         platforms, while the latter is supported on Apple platforms.
  
    **)  `acnt' tables are not supported by currently available Apple font
         tools.
  
    ***) There  is  one more  Apple  extension,  `hsty',  but  it  is  for
         Newton-OS, not GX  (Newton-OS is a platform by  Apple, but it can
         use  sfnt- housed bitmap  fonts only).   Therefore, it  should be
         excluded  from  `Apple  platform'  in the  context  of  TrueType.
         gxvalid ignores it as Apple font tools do so.
  
  
    We have  checked 183  fonts bundled with  MacOS 9.1, MacOS  9.2, MacOS
    10.0, MacOS X 10.1, MSIE  for MacOS, and AppleWorks 6.0.  In addition,
    we have  checked 67 Dynalab fonts  (designed for MacOS)  and 189 Ricoh
    fonts (designed for Windows and  MacOS dual platforms).  The number of
    fonts including TrueType GX tables are as follows.
  
      bsln:  76
      feat: 191
      just:  84
      kern:  59
      lcar:   4
      mort: 326
      morx:  19
      opbd:   4
      prop: 114
      trak:  16
  
    Dynalab  and Ricoh fonts  don't have  GX tables  except of  `feat' and
    `mort'.
  
  
  3. Fundamental validation limitations
  -------------------------------------
  
    TrueType  GX  provides  layout   information  to  libraries  for  font
    rasterizers  and text layout.   gxvalid can  check whether  the layout
    data in  a font is conformant  to the TrueType GX  format specified by
    Apple.  But gxvalid cannot check  a how QuickDraw GX/AAT renderer uses
    the stored information.
  
    3-1. Validation of State Machine activity
    -----------------------------------------
  
      QuickDraw GX/AAT uses a `State Machine' to provide `stateful' layout
      features,  and TrueType GX  stores the  state transition  diagram of
      this `State  Machine' in a  `StateTable' data structure.   While the
      State  Machine receives  a series  of glyph  IDs, the  State Machine
      starts with `start  of text' state, walks around  various states and
      generates various  layout information  to the  renderer, and finally
      reaches the `end of text' state.
  
      gxvalid can check essential errors like:
  
        - possibility of state transitions to undefined states
        - existence of glyph  IDs that the State Machine  doesn't know how
          to handle
        - the  State Machine  cannot compute  the layout  information from
          given diagram
  
      These errors  can be  checked within finite  steps, and  without the
      State Machine itself, because these are `expression' errors of state
      transition diagram.
  
      There  is no  limitation  about  how long  the  State Machine  walks
      around,  so validation  of  the algorithm  in  the state  transition
      diagram requires infinite  steps, even if we had  a State Machine in
      gxvalid.   Therefore, the  following errors  and problems  cannot be
      checked.
  
        - existence of states which the State Machine never transits to
        - the  possibility that the  State Machine  never reaches  `end of
          text'
        - the possibility of stack underflow/overflow in the State Machine
          (in  ligature  and  contextual  glyph substitutions,  the  State
          Machine can store 16 glyphs onto its stack)
  
      In addition, gxvalid doesn't check `temporary glyph IDs' used in the
      chained State Machines  (in `mort' and `morx' tables).   If a layout
      feature  is  implemented by  a  single  State  Machine, a  glyph  ID
      converted by the State Machine is passed to the glyph renderer, thus
      it  should not  point to  an undefined  glyph ID.   But if  a layout
      feature is implemented by  chained State Machines, a component State
      Machine  (if it  is  not the  final  one) is  permitted to  generate
      undefined glyph IDs for temporary use, because it is handled by next
      component State Machine and not  by the glyph renderer.  To validate
      such temporary glyph IDs, gxvalid must stack all undefined glyph IDs
      which  can occur in  the output  of the  previous State  Machine and
      search  them in  the  `ClassTable' structure  of  the current  State
      Machine.  It is too complex to  list all possible glyph IDs from the
      StateTable, especially from a ligature substitution table.
  
    3-2. Validation of relationship between multiple layout features
    ----------------------------------------------------------------
  
      gxvalid does  not validate the relationship  between multiple layout
      features at all.
  
      If  multiple layout  features  are defined  in  TrueType GX  tables,
      possible  interactions,  overrides,  and  conflicts  between  layout
      features are implicitly  given in the font too.   For example, there
      are several predefined spacing control features:
  
        - Text Spacing          (Proportional/Monospace/Half-width/Normal)
        - Number Spacing        (Monospaced-numbers/Proportional-numbers)
        - Kana Spacing          (Full-width/Proportional)
        - Ideographic Spacing   (Full-width/Proportional)
        - CJK Roman Spacing     (Half-width/Proportional/Default-roman
                                 /Full-width-roman/Proportional)
  
      If all  layout features are  independently managed, we  can activate
      inconsistent  typographic rules  like  `Text Spacing=Monospace'  and
      `Ideographic Spacing=Proportional' at the same time.
  
      The combinations  of layout features  is managed by a  32bit integer
      (one bit each for selector  setting), so we can define relationships
      between  up  to 32  features,  theoretically.   But  if one  feature
      setting  affects  another   feature  setting,  we  need  typographic
      priority  rules to  validate the  relationship.   Unfortunately, the
      TrueType GX format specification does not give such information even
      for predefined features.
  
  
  4. Permissive error handling of broken GX tables
  ------------------------------------------------
  
    When  Apple's font  rendering system  finds an  inconsistency,  like a
    specification  violation or  an  unspecified value  in  a TrueType  GX
    table, it does not always  return error.  In most cases, the rendering
    engine silently  ignores such wrong  values or even whole  tables.  In
    fact, MacOS is shipped with  fonts including broken GX/AAT tables, but
    no harmful  effects due to  `officially broken' fonts are  observed by
    end-users.
  
    gxvalid  is designed  to continue  the validation  process as  long as
    possible.  When gxvalid find wrong  values, gxvalid warns it at least,
    and takes  a fallback procedure  if possible.  The  fallback procedure
    depends on the debug level.
  
    We used the following three tools to investigate Apple's error handling.
  
      - FontValidator  (for MacOS 8.5 - 9.2)  resource fork font
      - ftxvalidator   (for MacOS X 10.1 -)   dfont or naked-sfnt
      - ftxdumperfuser (for MacOS X 10.1 -)   dfont or naked-sfnt
  
    However, all tests were done on a PowerPC based Macintosh; at present,
    we have not checked those tools on a m68k-based Macintosh.
  
    In total, we checked 183 fonts  bundled to MacOS 9.1, MacOS 9.2, MacOS
    10.0, MacOS X  10.1, MSIE for MacOS, and  AppleWorks 6.0.  These fonts
    are distributed  officially, but many broken GX/AAT  tables were found
    by Apple's font tools.  In the following, we list typical violation of
    the GX specification, in fonts officially distributed with those Apple
    systems.
  
    4-1. broken BinSrchHeader (19/183)
    ----------------------------------
  
      `BinSrchHeader' is  a header of a  data array for  m68k platforms to
      access memory efficiently.  Although  there are only two independent
      parameters  for real  (`unitSize' and  `nUnits'),  BinSrchHeader has
      three additional parameters which  can be calculated from `unitSize'
      and  `nUnits',  for  fast  setup.   Apple  font  tools  ignore  them
      silently, so gxvalid warns if it finds and inconsistency, and always
      continues  validation.    The  additional  parameters   are  ignored
      regardless of the consistency.
  
        19  fonts include  such  inconsistencies; all  breaks  are in  the
        BinSrchHeader structure of the `kern' table.
  
    4-2. too-short LookupTable (5/183)
    ----------------------------------
  
      LookupTable format 0  is a simple array to get a  value from a given
      GID (glyph  ID); the index of  this array is a  GID too.  Therefore,
      the length  of the array is expected  to be same as  the maximum GID
      value defined  in the `maxp' table,  but there are  some fonts whose
      LookupTable format 0 is too  short to cover all GIDs.  FontValidator
      ignores  this error silently,  ftxvalidator and  ftxdumperfuser both
      warn and continue.  Similar problems are found in format 3 subtables
      of `kern'.  gxvalid  warns always and abort if  the validation level
      is set to FT_VALIDATE_PARANOID.
  
        5 fonts include too-short kern format 0 subtables.
        1 font includes too-short kern format 3 subtable.
  
    4-3. broken LookupTable format 2 (1/183)
    ----------------------------------------
  
      LookupTable  format  2,  subformat  4  covers the  GID  space  by  a
      collection  of  segments which  are  specified  by `firstGlyph'  and
      `lastGlyph'.   Some  fonts  store  `firstGlyph' and  `lastGlyph'  in
      reverse order,  so the segment specification is  broken.  Apple font
      tools ignore this error silently;  a broken segment is ignored as if
      it  did not  exist.   gxvalid  warns and  normalize  the segment  at
      FT_VALIDATE_DEFAULT, or ignore  the segment at FT_VALIDATE_TIGHT, or
      abort at FT_VALIDATE_PARANOID.
  
        1 font includes broken LookupTable format 2, in the `just' table.
  
      *) It seems  that all fonts manufactured by  ITC for AppleWorks have
         this error.
  
    4-4. bad bracketing in glyph property (14/183)
    ----------------------------------------------
  
      GX/AAT defines a  `bracketing' property of the glyphs  in the `prop'
      table,  to control layout  features of  strings enclosed  inside and
      outside  of   brackets.   Some  fonts   give  inappropriate  bracket
      properties  to glyphs.   Apple  font tools  warn  about this  error;
      gxvalid warns too and aborts at FT_VALIDATE_PARANOID.
  
        14 fonts include wrong bracket properties.
  
  
    4-5. invalid feature number (117/183)
    -------------------------------------
  
      The GX/AAT  extension can  include 255 different  layout features,
      but    popular    layout     features    are    predefined    (see
      https://developer.apple.com/fonts/TrueType-Reference-Manual/RM09/AppendixF.html).
      Some fonts include feature numbers which are incompatible with the
      predefined feature registry.
  
      In our survey, there are 140 fonts including `feat' table.
  
      a) 67 fonts use a feature number which should not be used.
      b) 117 fonts set the wrong feature range (nSetting).  This is mostly
         found in the `mort' and `morx' tables.
  
      Apple  font tools give  no warning,  although they  cannot recognize
      what  the feature  is.   At FT_VALIDATE_DEFAULT,  gxvalid warns  but
      continues in both cases (a, b).  At FT_VALIDATE_TIGHT, gxvalid warns
      and aborts for (a), but continues for (b).  At FT_VALIDATE_PARANOID,
      gxvalid warns and aborts in both cases (a, b).
  
    4-6. invalid prop version (10/183)
    ----------------------------------
  
      As most TrueType GX tables, the `prop' table must start with a 32bit
      version identifier: 0x00010000,  0x00020000 or 0x00030000.  But some
      fonts  store nonsense binary  data instead.   When Apple  font tools
      find them, they abort the processing immediately, and the data which
      follows is unhandled.  gxvalid does the same.
  
        10 fonts include broken `prop' version.
  
      All  of these  fonts are  classic  TrueType fonts  for the  Japanese
      script, manufactured by Apple.
  
    4-7. unknown resource name (2/183)
    ------------------------------------
  
      NOTE: THIS IS NOT A TRUETYPE GX ERROR.
  
      If  a TrueType  font is  stored  in the  resource fork  or in  dfont
      format, the data must be tagged as `sfnt' in the resource fork index
      to invoke TrueType font handler for the data.  But the TrueType font
      data  in   `Keyboard.dfont'  is  tagged   as  `kbd',  and   that  in
      `LastResort.dfont' is tagged as  `lst'.  Apple font tools can detect
      that the data is in  TrueType format and successfully validate them.
      Maybe  this is possible  because they  are known  to be  dfont.  The
      current  implementation  of the  resource  fork  driver of  FreeType
      cannot do that, thus gxvalid cannot validate them.
  
        2 fonts use an unknown tag for the TrueType font resource.
  
  5. `kern' table issues
  ----------------------
  
    In common terminology of TrueType, `kern' is classified as a basic and
    platform-independent table.  But there are Apple extensions of `kern',
    and  there is  an  extension which  requires  a GX  state machine  for
    contextual kerning.   Therefore, gxvalid includes  a special validator
    for  `kern' tables.   Unfortunately, there  is no  exact  algorithm to
    check Apple's extension, so  gxvalid includes a heuristic algorithm to
    find  the proper validation  routines for  all possible  data formats,
    including    the   data    format   for    Microsoft.     By   calling
    classic_kern_validate() instead of gxv_validate(), you can specify the
    `kern' format  explicitly.  However, current  FreeType2 uses Microsoft
    `kern' format  only, others  are ignored (and  should be handled  in a
    library one level higher than FreeType).
  
    5-1. History
    ------------
  
      The original  16bit version of `kern'  was designed by  Apple in the
      pre-GX  era, and  it was  also approved  by  Microsoft.  Afterwards,
      Apple designed a  new 32bit version of the  `kern' table.  According
      to  the documentation, the  difference between  the 16bit  and 32bit
      version is only the size of  variables in the `kern' header.  In the
      following,  we call  the original  16bit version  as  `classic', and
      32bit version as `new'.
  
    5-2. Versions and dialects which should be differentiated
    ---------------------------------------------------------
  
      The `kern' table  consists of a table header  and several subtables.
      The version number  which identifies a `classic' or  a `new' version
      is  explicitly   written  in  the   table  header,  but   there  are
      undocumented  differences between  Microsoft's and  Apple's formats.
      It is  called a `dialect' in  the following.  There  are three cases
      which  should  be  handled:   the  new  Apple-dialect,  the  classic
      Apple-dialect,  and the classic  Microsoft-dialect.  An  analysis of
      the formats and the auto detection algorithm of gxvalid is described
      in the following.
  
      5-2-1. Version detection: classic and new kern
      ----------------------------------------------
  
        According  to Apple  TrueType  specification, there  are only  two
        differences between the classic and the new:
  
          - The `kern' table header starts with the version number.
            The classic version starts with 0x0000 (16bit),
            the new version starts with 0x00010000 (32bit).
  
          - In the  `kern' table header,  the number of  subtables follows
            the version number.
            In the classic version, it is stored as a 16bit value.
            In the new version, it is stored as a 32bit value.
  
        From Apple font tool's output (DumpKERN is also tested in addition
        to  the  three  Apple  font  tools in  above),  there  is  another
        undocumented difference.  In the  new version, the subtable header
        includes a 16bit variable  named `tupleIndex' which does not exist
        in the classic version.
  
        The new version  can store all subtable formats (0,  1, 2, and 3),
        but the Apple TrueType specification does not mention the subtable
        formats available in the classic version.
  
      5-2-2. Available subtable formats in classic version
      ----------------------------------------------------
  
        Although the  Apple TrueType  specification recommends to  use the
        classic version in  the case if the font is  designed for both the
        Apple and Microsoft platforms,  it does not document the available
        subtable formats in the classic version.
  
        According  to the Microsoft  TrueType specification,  the subtable
        format  assured for  Windows  and OS/2  support  is only  subtable
        format  0.  The  Microsoft TrueType  specification  also describes
        subtable format  2, but does  not mention which  platforms support
        it.  Subtable formats 1, 3,  and higher are documented as reserved
        for future use.  Therefore, the classic version can store subtable
        formats 0 and 2, at least.  `ttfdump.exe', a font tool provided by
        Microsoft,  ignores the  subtable format  written in  the subtable
        header, and parses the table as if all subtables are in format 0.
  
        `kern'  subtable format  1  uses  a StateTable,  so  it cannot  be
        utilized without a GX  State Machine.  Therefore, it is reasonable
        to assume  that format 1 (and  3) were introduced  after Apple had
        introduced GX and moved to the new 32bit version.
  
      5-2-3. Apple and Microsoft dialects
      -----------------------------------
  
        The  `kern' subtable  has  a 16bit  `coverage'  field to  describe
        kerning attributes, but bit interpretations by Apple and Microsoft
        are different:  For example, Apple  uses bits 0-7 to  identify the
        subtable, while Microsoft uses bits 8-15.
  
        In  addition, due  to the  output of  DumpKERN  and FontValidator,
        Apple's bit interpretations of coverage in classic and new version
        are  incompatible also.   In  summary, there  are three  dialects:
        classic Apple  dialect, classic  Microsoft dialect, and  new Apple
        dialect.  The classic Microsoft  dialect and the new Apple dialect
        are documented  by each vendors' TrueType  font specification, but
        the documentation for classic Apple dialect is not available.
  
        For example,  in the  new Apple dialect,  bit 15 is  documented as
        `set to  1 if  the kerning  is vertical'.  On  the other  hand, in
        classic Microsoft dialect, bit 1 is documented as `set to 1 if the
        kerning  is  horizontal'.   From   the  outputs  of  DumpKERN  and
        FontValidator, classic  Apple dialect recognizes  15 as `set  to 1
        when  the kerning  is horizontal'.   From the  results  of similar
        experiments, classic Apple dialect  seems to be the Endian reverse
        of the classic Microsoft dialect.
  
        As a  conclusion it must be  noted that no font  tool can identify
        classic Apple dialect or classic Microsoft dialect automatically.
  
      5-2-4. gxvalid auto dialect detection algorithm
      -----------------------------------------------
  
        The first 16  bits of the `kern' table are  enough to identify the
        version:
  
          - if  the first  16  bits are  0x0000,  the `kern'  table is  in
            classic Apple dialect or classic Microsoft dialect
          - if the first 16 bits are  0x0001, and next 16 bits are 0x0000,
            the kern table is in new Apple dialect.
  
        If the `kern'  table is a classic one,  the 16bit `coverage' field
        is checked next.   Firstly, the coverage bits are  decoded for the
        classic Apple dialect using the following bit masks (this is based
        on DumpKERN output):
  
          0x8000: 1=horizontal, 0=vertical
          0x4000: not used
          0x2000: 1=cross-stream, 0=normal
          0x1FF0: reserved
          0x000F: subtable format
  
        If  any  of  reserved  bits  are  set  or  the  subtable  bits  is
        interpreted as format 1 or 3, we take it as `impossible in classic
        Apple dialect' and retry, using the classic Microsoft dialect.
  
          The most popular coverage in new Apple-dialect:         0x8000,
          The most popular coverage in classic Apple-dialect:     0x0000,
          The most popular coverage in classic Microsoft dialect: 0x0001.
  
    5-3. Tested fonts
    -----------------
  
      We checked  59 fonts  bundled with MacOS  and 38 fonts  bundled with
      Windows, where all font include a `kern' table.
  
        - fonts bundled with MacOS
          * new Apple dialect
            format 0: 18
            format 2:  1
            format 3:  1
          * classic Apple dialect
            format 0: 14
          * classic Microsoft dialect
            format 0: 15
  
        - fonts bundled with Windows
          * classic Microsoft dialect
            format 0: 38
  
      It looks strange that classic Microsoft-dialect fonts are bundled to
      MacOS: they come from MSIE for MacOS, except of MarkerFelt.dfont.
  
  
    ACKNOWLEDGEMENT
    ---------------
  
    Some parts of gxvalid are  derived from both the `gxlayout' module and
    the `otvalid'  module.  Development of  gxlayout was supported  by the
    Information-technology Promotion Agency(IPA), Japan.
  
    The detailed analysis of undefined  glyph ID utilization in `mort' and
    `morx' tables is provided by George Williams.
  
  ------------------------------------------------------------------------
  
  Copyright (C) 2004-2023 by
  suzuki toshiya, Masatake YAMATO, Red hat K.K.,
  David Turner, Robert Wilhelm, and Werner Lemberg.
  
  This  file is  part  of the  FreeType  project, and  may  only be  used,
  modified,  and  distributed under  the  terms  of  the FreeType  project
  license, LICENSE.TXT.  By continuing  to use, modify, or distribute this
  file  you indicate that  you have  read the  license and  understand and
  accept it fully.
  
  
  --- end of README ---
  

Download