Edit
kc3-lang/harfbuzz/docs/usermanual-what-is-harfbuzz.xml
David Corbett
- docs/usermanual-what-is-harfbuzz.xml
-
<?xml version="1.0"?> <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [ <!ENTITY % local.common.attrib "xmlns:xi CDATA #FIXED 'http://www.w3.org/2003/XInclude'"> <!ENTITY version SYSTEM "version.xml"> ]> <chapter id="what-is-harfbuzz"> <title>What is HarfBuzz?</title> <para> HarfBuzz is a <emphasis>text-shaping engine</emphasis>. If you give HarfBuzz a font and a string containing a sequence of Unicode codepoints, HarfBuzz selects and positions the corresponding glyphs from the font, applying all of the necessary layout rules and font features. HarfBuzz then returns the string to you in the form that is correctly arranged for the language and writing system. </para> <para> HarfBuzz can properly shape all of the world's major writing systems. It runs on all major operating systems and software platforms and it supports the major font formats in use today. </para> <section id="what-is-text-shaping"> <title>What is text shaping?</title> <para> Text shaping is the process of translating a string of character codes (such as Unicode codepoints) into a properly arranged sequence of glyphs that can be rendered onto a screen or into final output form for inclusion in a document. </para> <para> The shaping process is dependent on the input string, the active font, the script (or writing system) that the string is in, and the language that the string is in. </para> <para> Modern software systems generally only deal with strings in the Unicode encoding scheme (although legacy systems and documents may involve other encodings). </para> <para> There are several font formats that a program might encounter, each of which has a set of standard text-shaping rules. </para> <para>The dominant format is <ulink url="http://www.microsoft.com/typography/otspec/">OpenType</ulink>. The OpenType specification defines a series of <ulink url="https://github.com/n8willis/opentype-shaping-documents">shaping models</ulink> for various scripts from around the world. These shaping models depend on the font incorporating certain features as <emphasis>lookups</emphasis> in its <literal>GSUB</literal> and <literal>GPOS</literal> tables. </para> <para> Alternatively, OpenType fonts can include shaping features for the <ulink url="https://graphite.sil.org/">Graphite</ulink> shaping model. </para> <para> TrueType fonts can also include OpenType shaping features. Alternatively, TrueType fonts can also include <ulink url="https://developer.apple.com/fonts/TrueType-Reference-Manual/RM09/AppendixF.html">Apple Advanced Typography</ulink> (AAT) tables to implement shaping support. AAT fonts are generally only found on macOS and iOS systems. </para> <para> Text strings will usually be tagged with a script and language tag that provide the context needed to perform text shaping correctly. The necessary <ulink url="https://docs.microsoft.com/en-us/typography/opentype/spec/scripttags">script</ulink> and <ulink url="https://docs.microsoft.com/en-us/typography/opentype/spec/languagetags">language</ulink> tags are defined by OpenType. </para> </section> <section id="why-do-i-need-a-shaping-engine"> <title>Why do I need a shaping engine?</title> <para> Text shaping is an integral part of preparing text for display. Before a Unicode sequence can be rendered, the codepoints in the sequence must be mapped to the corresponding glyphs provided in the font, and those glyphs must be positioned correctly relative to each other. For many of the scripts supported in Unicode, these steps involve script-specific layout rules, including complex joining, reordering, and positioning behavior. Implementing these rules is the job of the shaping engine. </para> <para> Text shaping is a fairly low-level operation. HarfBuzz is used directly by text-handling libraries like <ulink url="https://www.pango.org/">Pango</ulink>, as well as by the layout engines in Firefox, LibreOffice, and Chromium. Unless you are <emphasis>writing</emphasis> one of these layout engines yourself, you will probably not need to use HarfBuzz: normally, a layout engine, toolkit, or other library will turn text into glyphs for you. </para> <para> However, if you <emphasis>are</emphasis> writing a layout engine or graphics library yourself, then you will need to perform text shaping, and this is where HarfBuzz can help you. </para> <para> Here are some specific scenarios where a text-shaping engine like HarfBuzz helps you: </para> <itemizedlist> <listitem> <para> OpenType fonts contain a set of glyphs (that is, shapes to represent the letters, numbers, punctuation marks, and all other symbols), which are indexed by a <literal>glyph ID</literal>. </para> <para> A particular glyph ID within the font does not necessarily correlate to a predictable Unicode codepoint. For instance, some fonts have the letter "a" as glyph ID 1, but many others do not. In order to retrieve the right glyph from the font to display "a", you need to consult the table inside the font (the <literal>cmap</literal> table) that maps Unicode codepoints to glyph IDs. In other words, <emphasis>text shaping turns codepoints into glyph IDs</emphasis>. </para> </listitem> <listitem> <para> Many OpenType fonts contain ligatures: combinations of characters that are rendered as a single unit. For instance, it is common for the "f, i" letter sequence to appear in print as the single ligature glyph "fi". </para> <para> Whether you should render an "f, i" sequence as <literal>fi</literal> or as "fi" does not depend on the input text. Instead, it depends on the whether or not the font includes an "fi" glyph and on the level of ligature application you wish to perform. The font and the amount of ligature application used are under your control. In other words, <emphasis>text shaping involves querying the font's ligature tables and determining what substitutions should be made</emphasis>. </para> </listitem> <listitem> <para> While ligatures like "fi" are optional typographic refinements, some languages <emphasis>require</emphasis> certain substitutions to be made in order to display text correctly. </para> <para> For example, in Tamil, when the letter "TTA" (ட) letter is followed by the vowel sign "U" (ு), the pair must be replaced by the single glyph "டு". The sequence of Unicode characters "ட,ு" needs to be substituted with a single "டு" glyph from the font. </para> <para> But "டு" does not have a Unicode codepoint. To find this glyph, you need to consult the table inside the font (the <literal>GSUB</literal> table) that contains substitution information. In other words, <emphasis>text shaping chooses the correct glyph for a sequence of characters provided</emphasis>. </para> </listitem> <listitem> <para> Similarly, each Arabic character has four different variants corresponding to the different positions it might appear in within a sequence. Inside a font, there will be separate glyphs for the initial, medial, final, and isolated forms of each letter, each at a different glyph ID. </para> <para> Unicode only assigns one codepoint per character, so a Unicode string will not tell you which glyph variant to use for each character. To decide, you need to analyze the whole string and determine the appropriate glyph for each character based on its position. In other words, <emphasis>text shaping chooses the correct form of the letter by its position and returns the correct glyph from the font</emphasis>. </para> </listitem> <listitem> <para> Other languages involve marks and accents that need to be rendered in specific positions relative a base character. For instance, the Moldovan language includes the Cyrillic letter "zhe" (ж) with a breve accent, like so: "ӂ". </para> <para> Some fonts will provide this character as a single zhe-with-breve glyph, but other fonts will not and, instead, will expect the rendering engine to form the character by superimposing the separate "ж" and "˘" glyphs. </para> <para> But exactly where you should draw the breve depends on the height and width of the preceding zhe glyph. To find the right position, you need to consult the table inside the font (the <literal>GPOS</literal> table) that contains positioning information. In other words, <emphasis>text shaping tells you whether you have a precomposed glyph within your font or if you need to compose a glyph yourself out of combining marks—and, if so, where to position those marks.</emphasis> </para> </listitem> </itemizedlist> <para> If tasks like these are something that you need to do, then you need a text shaping engine. You could use Uniscribe if you are writing Windows software; you could use CoreText on macOS; or you could use HarfBuzz. </para> <note> <para> In the rest of this manual, the text will assume that the reader is that implementor of a text-layout engine. </para> </note> </section> <section id="what-does-harfbuzz-do"> <title>What does HarfBuzz do?</title> <para> HarfBuzz provides text shaping through a cross-platform C API that accepts sequences of Unicode codepoints as input. Currently, the following OpenType shaping models are supported: </para> <itemizedlist> <listitem> <para> Indic (covering Devanagari, Bengali, Gujarati, Gurmukhi, Kannada, Malayalam, Oriya, Tamil, and Telugu) </para> </listitem> <listitem> <para> Arabic (covering Arabic, N'Ko, Syriac, and Mongolian) </para> </listitem> <listitem> <para> Thai and Lao </para> </listitem> <listitem> <para> Khmer </para> </listitem> <listitem> <para> Myanmar </para> </listitem> <listitem> <para> Tibetan </para> </listitem> <listitem> <para> Hangul </para> </listitem> <listitem> <para> Hebrew </para> </listitem> <listitem> <para> The Universal Shaping Engine or <emphasis>USE</emphasis> (covering complex scripts not covered by the above shaping models) </para> </listitem> <listitem> <para> A default shaping model for non-complex scripts (covering Latin, Cyrillic, Greek, Armenian, Georgian, Tifinagh, and many others) </para> </listitem> <listitem> <para> Emoji (including emoji modifier sequences, flag sequences, and ZWJ sequences) </para> </listitem> </itemizedlist> <para> In addition to OpenType shaping, HarfBuzz supports the latest version of Graphite shaping (the "Graphite 2" model) and AAT shaping. </para> <para> HarfBuzz can read and understand TrueType fonts (.ttf), TrueType collections (.ttc), and OpenType fonts (.otf, including those fonts that contain TrueType-style outlines and those that contain PostScript CFF or CFF2 outlines). </para> <para> HarfBuzz is designed and tested to run on top of the FreeType font renderer. It can run on Linux, Android, Windows, macOS, and iOS systems. </para> <para> In addition to its core shaping functionality, HarfBuzz provides functions for accessing other font features, including optional GSUB and GPOS OpenType features, as well as all color-font formats (<literal>CBDT</literal>, <literal>sbix</literal>, <literal>COLR/CPAL</literal>, and <literal>SVG-OT</literal>) and OpenType variable fonts. HarfBuzz also includes a font-subsetting feature. HarfBuzz can perform some low-level math-shaping operations, although it does not currently perform full shaping for mathematical typesetting. </para> <para> A suite of command-line utilities is also provided in the source-code tree, designed to help users test and debug HarfBuzz's features on real-world fonts and input. </para> </section> <section id="what-harfbuzz-doesnt-do"> <title>What HarfBuzz doesn't do</title> <para> HarfBuzz will take a Unicode string, shape it, and give you the information required to lay it out correctly on a single horizontal (or vertical) line using the font provided. That is the extent of HarfBuzz's responsibility. </para> <para> It is important to note that if you are implementing a complete text-layout engine you may have other responsibilities that HarfBuzz will <emphasis>not</emphasis> help you with. For example: </para> <itemizedlist> <listitem> <para> HarfBuzz won't help you with bidirectionality. If you want to lay out text that includes a mix of Hebrew and English, you will need to ensure that each buffer provided to HarfBuzz has all of its characters in the same order and that the directionality of the buffer is set correctly. This may mean segmenting the text before it is placed into HarfBuzz buffers. In other words, the user will hit the keys in the following sequence: </para> <programlisting> A B C [space] ג ב א [space] D E F </programlisting> <para> but will expect to see in the output: </para> <programlisting> ABC אבג DEF </programlisting> <para> This reordering is called <emphasis>bidi processing</emphasis> ("bidi" is short for bidirectional), and there's an algorithm as an annex to the Unicode Standard which tells you how to process a string of mixed directionality. Before sending your string to HarfBuzz, you may need to apply the bidi algorithm to it. Libraries such as <ulink url="http://icu-project.org/">ICU</ulink> and <ulink url="http://fribidi.org/">fribidi</ulink> can do this for you. </para> </listitem> <listitem> <para> HarfBuzz won't help you with text that contains different font properties. For instance, if you have the string "a <emphasis>huge</emphasis> breakfast", and you expect "huge" to be italic, then you will need to send three strings to HarfBuzz: <literal>a</literal>, in your Roman font; <literal>huge</literal> using your italic font; and <literal>breakfast</literal> using your Roman font again. </para> <para> Similarly, if you change the font, font size, script, language, or direction within your string, then you will need to shape each run independently and output them independently. HarfBuzz expects to shape a run of characters that all share the same properties. </para> </listitem> <listitem> <para> HarfBuzz won't help you with line breaking, hyphenation, or justification. As mentioned above, HarfBuzz lays out the string along a <emphasis>single line</emphasis> of, notionally, infinite length. If you want to find out where the potential word, sentence and line break points are in your text, you could use the ICU library's break iterator functions. </para> <para> HarfBuzz can tell you how wide a shaped piece of text is, which is useful input to a justification algorithm, but it knows nothing about paragraphs, lines or line lengths. Nor will it adjust the space between words to fit them proportionally into a line. </para> </listitem> </itemizedlist> <para> As a layout-engine implementor, HarfBuzz will help you with the interface between your text and your font, and that's something that you'll need—what you then do with the glyphs that your font returns is up to you. </para> </section> <section id="why-is-it-called-harfbuzz"> <title>Why is it called HarfBuzz?</title> <para> HarfBuzz began its life as text-shaping code within the FreeType project (and you will see references to the FreeType authors within the source code copyright declarations), but was then extracted out to its own project. This project is maintained by Behdad Esfahbod, who named it HarfBuzz. Originally, it was a shaping engine for OpenType fonts—"HarfBuzz" is the Persian for "open type". </para> </section> </chapter>