LuaLaTeX & Harfbuzz: Accessing Non-Unicode Glyphs

Aug 11, 2025 by Pedro Alvarez 50 views

Accessing Non-Unicode Glyphs by Name with Lua(La)TeX and Harfbuzz

Hey everyone! Today, we're diving deep into a fascinating challenge: accessing non-Unicode glyphs by name when using Lua(La)TeX and Harfbuzz. This is actually a continuation of a previous discussion where we tackled this issue with the Node renderer, but Harfbuzz presented a whole new level of complexity. Buckle up, because we're about to unravel this mystery together!

The Challenge: Harfbuzz and Non-Unicode Glyphs

So, what's the big deal with Harfbuzz and non-Unicode glyphs? Well, Harfbuzz is a powerful shaping engine that's used by Lua(La)TeX to handle complex text layout, especially when dealing with OpenType features. The main issue arises when we want to access glyphs that don't have a direct Unicode representation. These glyphs often live in a font's private use area or are accessed through OpenType features. While the Node renderer has its ways of handling this, Harfbuzz requires a different approach.

To truly understand the challenge, let's break it down further. When we talk about non-Unicode glyphs, we're essentially referring to characters that aren't part of the standard Unicode character set. Think of ligatures, stylistic alternates, or even symbols that a font designer might have included but didn't map to a specific Unicode code point. These glyphs are often crucial for achieving the desired typographic look, especially in professional typesetting.

Now, Harfbuzz's role in all of this is to take a stream of Unicode characters and, based on the font's information and the specified OpenType features, determine the correct glyphs to use and how to position them. It's a complex process that involves a lot of intricate calculations and lookups. When we introduce non-Unicode glyphs into the mix, Harfbuzz needs to know how to access them, and that's where things get tricky.

Think of it like this: Unicode characters are like the standard letters of the alphabet, and Harfbuzz is the skilled calligrapher who knows how to draw them beautifully. But when you ask the calligrapher to draw a custom symbol that's not in the alphabet, you need to provide specific instructions on how to find that symbol within the calligrapher's repertoire. That's essentially what we're trying to do with non-Unicode glyphs and Harfbuzz.

In the previous discussion, we found a solution for the Node renderer, which is another way Lua(La)TeX can handle font shaping. However, the solution didn't translate directly to Harfbuzz. This highlights a critical point: different shaping engines may require different techniques for accessing non-Unicode glyphs. This is because they have different internal mechanisms for handling font data and glyph lookups.

So, the core of the challenge lies in figuring out how to communicate to Harfbuzz which glyph we want to use, even if it doesn't have a Unicode representation. We need to find a way to tell Harfbuzz, "Hey, instead of using the glyph for the letter 'a', I want you to use this specific glyph with the name 'mySpecialGlyph'." This involves delving into Harfbuzz's API and Lua(La)TeX's font management capabilities.

Original Example

Let's revisit the original example that sparked this discussion. It serves as a great starting point for understanding the problem and exploring potential solutions. This example, slightly simplified, demonstrates the core issue: how to access a glyph by its name when Harfbuzz is in charge of shaping the text. The example typically involves loading a font, attempting to access a non-Unicode glyph by its name, and then rendering the result. The key is that the straightforward methods that work with the Node renderer often fail with Harfbuzz, leading to unexpected output or errors.

By examining this example closely, we can identify the points where Harfbuzz's behavior diverges from the Node renderer's. This will help us narrow down the search for a Harfbuzz-specific solution. We might need to explore different ways of specifying the glyph, such as using glyph IDs or OpenType feature tags, or we might need to delve into Harfbuzz's Lua API to find a way to directly manipulate the shaping process.

Exploring Potential Solutions

Now, let's brainstorm some potential solutions for accessing those elusive non-Unicode glyphs with Harfbuzz. This is where things get really interesting, guys! We'll need to put on our thinking caps and explore different avenues.

One promising approach involves leveraging OpenType features. OpenType features are essentially instructions embedded in the font that tell the shaping engine (like Harfbuzz) how to handle certain glyph substitutions and positioning. Many fonts use OpenType features to implement ligatures, stylistic alternates, and other advanced typographic effects. We might be able to define a custom OpenType feature that maps a specific sequence of characters to our desired non-Unicode glyph. This way, we can indirectly access the glyph by typing a specific sequence in our LaTeX document. This approach can be quite powerful because it integrates seamlessly with Harfbuzz's shaping process.

For instance, imagine we have a glyph named "mySpecialLigature" that we want to use. We could define an OpenType feature that substitutes the character sequence "XY" with "mySpecialLigature". Then, in our LaTeX document, whenever we type "XY", Harfbuzz would automatically replace it with the correct glyph. This requires a bit of font manipulation, but it can be a very elegant solution.

Another avenue to explore is Harfbuzz's Lua API. Harfbuzz provides a Lua API that allows us to directly interact with the shaping process. This gives us a lot of flexibility and control, but it also means we need to get our hands dirty with some Lua code. We might be able to use the Lua API to intercept the shaping process, look for specific glyph names, and then substitute them with the desired glyphs. This approach is more complex than using OpenType features, but it can be very powerful for handling complex scenarios or for implementing custom shaping logic.

Think of it as being able to "talk" directly to Harfbuzz and tell it exactly what to do. We could write Lua code that says, "Hey Harfbuzz, whenever you see the name 'mySpecialGlyph', use the glyph with ID 123 instead." This level of control is invaluable when dealing with non-standard glyph access.

Finally, we could also investigate the possibility of using glyph IDs directly. Each glyph in a font has a unique ID, and Harfbuzz might allow us to access glyphs by their ID. This approach would bypass the need for glyph names altogether. However, it requires us to know the ID of the glyph we want to use, which might not always be readily available. It also makes our code less readable, as glyph IDs are not as descriptive as glyph names. But if all else fails, this could be a viable option.

In summary, we have three main potential solutions:

OpenType features: Define custom features to map character sequences to non-Unicode glyphs.
Harfbuzz Lua API: Directly manipulate the shaping process using Lua code.
Glyph IDs: Access glyphs directly by their numeric IDs.

Each of these approaches has its own pros and cons, and the best solution will likely depend on the specific font and the desired level of control.

Diving into Lua(La)TeX's Font Management

To effectively implement any of these solutions, we need a solid understanding of how Lua(La)TeX manages fonts. Lua(La)TeX has a powerful font management system that allows us to load fonts, access their properties, and manipulate them in various ways. This is crucial for accessing non-Unicode glyphs, as we need to be able to delve into the font's internal data structures and extract the information we need.

Lua(La)TeX uses the fontspec package as its primary interface for font management. fontspec provides a high-level way to load fonts by name, specify OpenType features, and configure other font-related settings. However, for more advanced tasks like accessing glyph names or manipulating glyph tables, we need to go deeper and use the underlying Lua libraries.

One key library is luaotfload, which is responsible for loading OpenType fonts and making their data accessible to Lua code. luaotfload provides functions for accessing glyph names, glyph IDs, OpenType features, and other font properties. By using luaotfload, we can programmatically inspect a font and extract the information we need to access non-Unicode glyphs.

For example, we can use luaotfload to get a table that maps glyph names to glyph IDs. This table is invaluable when we want to access a glyph by its name but need to provide Harfbuzz with its ID. We can also use luaotfload to access the font's OpenType feature tables, which allows us to inspect and potentially modify existing features or add new ones. This is essential if we choose to go the OpenType feature route.

Another important aspect of Lua(La)TeX's font management is the concept of font attributes. Font attributes are properties associated with a font that control how it's used during typesetting. These attributes include things like the font's encoding, its size, and the OpenType features that are enabled. By manipulating font attributes, we can influence how Harfbuzz shapes the text and which glyphs it selects.

For instance, we can use font attributes to enable a custom OpenType feature that we've defined for accessing non-Unicode glyphs. This tells Harfbuzz to apply the feature during shaping, which will then trigger the glyph substitutions we've specified. Understanding font attributes is crucial for ensuring that our non-Unicode glyph access methods work correctly in different contexts.

In essence, Lua(La)TeX's font management system provides us with a powerful toolkit for working with fonts at a low level. By mastering these tools, we can overcome the challenges of accessing non-Unicode glyphs and achieve the typographic results we desire. It's like having a set of precision instruments that allow us to fine-tune the way text is rendered.

Practical Examples and Code Snippets

Alright, let's get our hands dirty with some code! To make things crystal clear, let's look at some practical examples and code snippets that demonstrate how we can access non-Unicode glyphs using the techniques we've discussed. These examples will give you a concrete starting point for experimenting and implementing your own solutions. This is where the rubber meets the road, guys!

Example 1: Using OpenType Features

Let's start with the OpenType feature approach. As we discussed, this involves defining a custom OpenType feature that maps a character sequence to a non-Unicode glyph. This typically requires modifying the font file itself, which can be done using font editing software like FontForge or Glyphs. However, for simpler cases, we can also use Lua code to add features on the fly.

Here's a simplified example of how we might define an OpenType feature in Lua(La)TeX:

\documentclass{article}
\usepackage{fontspec}

\directlua{
 local otf = require("luaotfload")

 local font_path = "/path/to/your/font.otf" -- Replace with your font path
 local font_id = otf.load(font_path)

 local feature_name = "liga" -- Use a suitable feature tag
 local glyph_name = "mySpecialGlyph" -- Replace with your glyph name
 local char_sequence = "XY" -- Character sequence to trigger the glyph

 -- Create a substitution rule
 local substitution = {
  lookup_type = 1, -- Single substitution
  substitutions = {
  [char_sequence] = glyph_name
  }
 }

 -- Add the feature to the font
 otf.add_feature(font_id, feature_name, { substitution })

 -- Set the feature as active
 otf.set_feature(font_id, feature_name, true)

 -- Register the modified font
 fonts.handlers.otf.add_virtual(font_id)
}

\setmainfont{YourFontName[RawFeature=+liga]} -- Replace with your font name

\begin{document}
XY -- This should display the special glyph
\end{document}

This code snippet demonstrates the basic steps involved in adding a custom OpenType feature. First, we load the font using luaotfload. Then, we define a substitution rule that maps the character sequence "XY" to the glyph named "mySpecialGlyph". Finally, we add the feature to the font and set it as active. In the LaTeX document, we use the RawFeature option in \setmainfont to enable the feature.

Example 2: Using Harfbuzz Lua API

Now, let's explore how to use the Harfbuzz Lua API to access non-Unicode glyphs. This approach requires a deeper understanding of Harfbuzz's shaping process, but it gives us a lot of control. We'll need to write a Lua callback function that intercepts the shaping process and substitutes glyphs as needed.

Here's a simplified example:

\documentclass{article}
\usepackage{fontspec}

\directlua{
 local hb = require("harfbuzz")
 local otf = require("luaotfload")

 local font_path = "/path/to/your/font.otf" -- Replace with your font path
 local font_id = otf.load(font_path)

 local glyph_name = "mySpecialGlyph" -- Replace with your glyph name
 local glyph_id = otf.find_glyph_by_name(font_id, glyph_name)

 -- Create a Harfbuzz shaping callback
 local function shape_callback(font, buffer, glyph_index)
  if buffer.glyphs[glyph_index] == 123 then -- Replace 123 with the ID of the character you want to replace
  buffer.glyphs[glyph_index] = glyph_id
  end
 end

 -- Set the shaping callback
 hb.set_shape_func(font_id, shape_callback)

 -- Register the modified font
 fonts.handlers.otf.add_virtual(font_id)
}

\setmainfont{YourFontName} -- Replace with your font name

\begin{document}
{Character with ID 123} -- This should display the special glyph
\end{document}

This code snippet defines a Harfbuzz shaping callback that checks if a glyph with a specific ID (123 in this example) is encountered. If it is, the callback substitutes it with the glyph ID of "mySpecialGlyph". This allows us to indirectly access the non-Unicode glyph by using a placeholder character with ID 123. This is a more advanced technique that requires a good understanding of Harfbuzz's API and Lua programming.

These examples are just starting points, of course. The specific code you'll need will depend on your font, the glyphs you want to access, and the desired level of control. But hopefully, these examples give you a solid foundation for experimenting and developing your own solutions.

Conclusion: Conquering Non-Unicode Glyphs

Well, guys, we've journeyed through the fascinating world of accessing non-Unicode glyphs with Lua(La)TeX and Harfbuzz. It's been a bit of a rollercoaster, but hopefully, you now have a clearer understanding of the challenges and the potential solutions. Remember, working with fonts and shaping engines can be complex, but the rewards are well worth the effort.

We've explored several approaches, from leveraging OpenType features to diving into Harfbuzz's Lua API. Each technique has its strengths and weaknesses, and the best approach will depend on your specific needs and the complexity of your project. The key takeaway is that there are multiple ways to achieve your goals, and with a little bit of ingenuity, you can conquer even the most challenging typographic tasks.

The world of typography is constantly evolving, and new tools and techniques are always emerging. By staying curious and continuing to explore, you can unlock the full potential of Lua(La)TeX and create truly stunning documents. So, keep experimenting, keep learning, and keep pushing the boundaries of what's possible!

If you have any questions or want to share your own experiences, please feel free to leave a comment below. Let's continue this conversation and help each other master the art of typography!