The uax-15 Reference Manual

Table of Contents

The uax-15 Reference Manual

This is the uax-15 Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 15:23:07 2020 GMT+0.

1 Introduction

uax-15

This package provides a common lisp unicode normalization function using nfc, nfd, nfkc and nfkd as per Unicode Standard Annex #15 found at http://www.unicode.org/reports/tr15/tr15-22.html.

This is a fork of a subset of work done by Takeru Ohta in 2010. Future work is intended to provide support for https://tools.ietf.org/html/rfc8264 and https://tools.ietf.org/html/rfc7564.

Implementation Notes

This has been successfully tested on sbcl, ccl, ecl, abcl, allegro and cmucl against the unicode test file found at http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt

Clisp still has some issues. It has not been tested against lispworks or other common lisp implementations.

Usage

It has one major exported function:

• (normalize (str unicode-normalization-method))

The currently supported normalization methods are :nfc :nfkc :nfd :nfkd

Normalization example with reference to relevant xkcd https://www.xkcd.com/936/

    (normalize "正しい馬バッテリーステープル" :nfkc)
    "正しい馬バッテリーステープル"

    (normalize "الحصان الصحيح البطارية التيلة" :nfkc)
    "الحصان الصحيح البطارية التيلة"

    (normalize "اstáplacha ceart ceallraí capall" :nfkc)
    "اstáplacha ceart ceallraí capall"

To Do list

• Implement and validate against https://tools.ietf.org/html/rfc8264 and https://tools.ietf.org/html/rfc7564
• Optimization?

More relevant xkcd https://xkcd.com/1726/, https://xkcd.com/1953/, https://www.xkcd.com/1209/, https://xkcd.com/1137/

Data Files

• UnicodeData.txt was downloaded from http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
• The file CompositionExclusions.txt was downloaded from http://www.unicode.org/Public/UNIDATA/CompositionExclusions.txt
• The file DerivedNormalizationProps.txt was downloaded from http://www.unicode.org/Public/UNIDATA/DerivedNormalizationProps.txt
• The test file NormalizationTest.txt was downloaded from http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt

Other References

• https://tools.ietf.org/html/rfc5802
• http://www.unicode.org/reports/tr15/#References
• https://www.unicode.org/reports/tr41/tr41-24.html
• https://www.unicode.org/charts/normalization/
• https://towardsdatascience.com/difference-between-nfd-nfc-nfkd-and-nfkc-explained-with-python-code-e2631f96ae6c
• https://en.wikipedia.org/wiki/Unicode_equivalence
• http://www.unicode.org/faq/normalization.html
• https://github.com/edicl/cl-unicode/blob/master/specials.lisp
• https://perldoc.perl.org/Unicode/Normalize.html
• [https://www.mkssoftware.com/docs/perl/lib/Unicode/Normalize.asp](https://www.mkssoftware.com/docs/perl/lib/

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 uax-15

Author

Takeru Ohta, Sabra Crolleton <sabra.crolleton@gmail.com>

License

MIT

Description

Common lisp implementation of Unicode normalization functions :nfc, :nfd, :nfkc and :nfkd (Uax-15)

Dependencies

• split-sequence
• cl-ppcre
• uiop

Source

uax-15.asd (file)

Component

src (module)

3 Modules

Modules are listed depth-first from the system components tree.

3.1 uax-15/src

Parent

uax-15 (system)

Location

src/

Components

• package.lisp (file)
• utilities.lisp (file)
• trivial-utf-16.lisp (file)
• precomputed-tables.lisp (file)
• normalize-backend.lisp (file)
• uax-15.lisp (file)

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.

4.1 Lisp

4.1.1 uax-15.asd

Location

uax-15.asd

Systems

uax-15 (system)

Packages

uax-15-system

Internal Definitions

*string-file* (special variable)

4.1.2 uax-15/src/package.lisp

Parent

src (module)

Location

src/package.lisp

Packages

uax-15

4.1.3 uax-15/src/utilities.lisp

Dependency

package.lisp (file)

Parent

src (module)

Location

src/utilities.lisp

Internal Definitions

• bad-char-error (function)
• bad-char-error (condition)
• bad-char-error-message (method)
• (setf bad-char-error-message) (method)
• bad-char-error-normalization-form (method)
• (setf bad-char-error-normalization-form) (method)
• bad-char-error-value (method)
• (setf bad-char-error-value) (method)
• int-to-hex-string (function)
• nconcf (macro)
• parse-hex-list-to-string (function)
• parse-hex-string-to-char (function)
• parse-hex-string-to-int (function)
• parse-hex-string-to-string (function)

4.1.4 uax-15/src/trivial-utf-16.lisp

Dependency

package.lisp (file)

Parent

src (module)

Location

src/trivial-utf-16.lisp

Exported Definitions

• codepoint-as-utf-16 (function)
• from-unicode-string (function)
• surrogates-to-codepoint (function)
• to-unicode-string (function)
• unicode-string (type)

Internal Definitions

• decode-utf-16 (function)
• encode-utf-16 (function)
• high-surrogate (type)
• low-surrogate (type)
• unicode-point (type)

4.1.5 uax-15/src/precomputed-tables.lisp

Dependencies

• package.lisp (file)
• utilities.lisp (file)

Parent

src (module)

Location

src/precomputed-tables.lisp

Internal Definitions

• *canonical-combining-class* (special variable)
• *canonical-comp-map* (special variable)
• *canonical-decomp-map* (special variable)
• *compatible-decomp-map* (special variable)
• *composition-exclusions-data* (special variable)
• *data-directory* (special variable)
• *unicode-data* (special variable)

4.1.6 uax-15/src/normalize-backend.lisp

Dependencies

• package.lisp (file)
• utilities.lisp (file)
• precomputed-tables.lisp (file)
• trivial-utf-16.lisp (file)

Parent

src (module)

Location

src/normalize-backend.lisp

Internal Definitions

• canonical-ordering (function)
• compose (function)
• compose-hangul (function)
• decompose (function)
• decompose-char (function)
• decompose-hangul-char (function)
• get-canonical-combining-class (function)
• nfc (function)
• nfd (function)
• nfkc (function)
• nfkd (function)

4.1.7 uax-15/src/uax-15.lisp

Dependencies

• package.lisp (file)
• utilities.lisp (file)
• normalize-backend.lisp (file)
• trivial-utf-16.lisp (file)

Parent

src (module)

Location

src/uax-15.lisp

Exported Definitions

• get-canonical-combining-class-map (function)
• get-illegal-char-list (function)
• get-mapping (function)
• normalize (function)

Internal Definitions

• *derived-normalization-props-data* (special variable)
• *derived-normalization-props-data-file* (special variable)
• normalize-char (function)

5 Packages

Packages are listed by definition order.

5.1 uax-15-system

Source

uax-15.asd

Use List

• asdf/interface
• common-lisp

Internal Definitions

*string-file* (special variable)

5.2 uax-15

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions

• codepoint-as-utf-16 (function)
• from-unicode-string (function)
• get-canonical-combining-class-map (function)
• get-illegal-char-list (function)
• get-mapping (function)
• normalize (function)
• surrogates-to-codepoint (function)
• to-unicode-string (function)
• unicode-string (type)

Internal Definitions

• *canonical-combining-class* (special variable)
• *canonical-comp-map* (special variable)
• *canonical-decomp-map* (special variable)
• *compatible-decomp-map* (special variable)
• *composition-exclusions-data* (special variable)
• *data-directory* (special variable)
• *derived-normalization-props-data* (special variable)
• *derived-normalization-props-data-file* (special variable)
• *unicode-data* (special variable)
• bad-char-error (function)
• bad-char-error (condition)
• bad-char-error-message (generic function)
• bad-char-error-message (method)
• (setf bad-char-error-message) (method)
• (setf bad-char-error-message) (generic function)
• bad-char-error-normalization-form (generic function)
• bad-char-error-normalization-form (method)
• (setf bad-char-error-normalization-form) (method)
• (setf bad-char-error-normalization-form) (generic function)
• bad-char-error-value (generic function)
• bad-char-error-value (method)
• (setf bad-char-error-value) (method)
• (setf bad-char-error-value) (generic function)
• canonical-ordering (function)
• compose (function)
• compose-hangul (function)
• decode-utf-16 (function)
• decompose (function)
• decompose-char (function)
• decompose-hangul-char (function)
• encode-utf-16 (function)
• get-canonical-combining-class (function)
• high-surrogate (type)
• int-to-hex-string (function)
• low-surrogate (type)
• nconcf (macro)
• nfc (function)
• nfd (function)
• nfkc (function)
• nfkd (function)
• normalize-char (function)
• parse-hex-list-to-string (function)
• parse-hex-string-to-char (function)
• parse-hex-string-to-int (function)
• parse-hex-string-to-string (function)
• unicode-point (type)

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

6.1 Exported definitions

6.1.1 Functions

Function: codepoint-as-utf-16 CODEPOINT

Translate a Unicode code point to its UTF-16 representation. Returns a list of one or two codepoints. Passes surrogate code points straight through.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Function: from-unicode-string UNICODE-STRING

Take a vector of Unicode code points and turn it into a Lisp string.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Function: get-canonical-combining-class-map ()

Package

uax-15

Source

uax-15.lisp (file)

Function: get-illegal-char-list NORMALIZATION-FORM

Takes a normalization form, e.g. :nfkc and returns a list of lists of form (#NO-BREAK_SPACE NIL) where the first item is the character name and the second item has the value N or M or nil indicating whether the character may require renormalization.

Package

uax-15

Source

uax-15.lisp (file)

Function: get-mapping NORMALIZATION-FORM &aux MAPPING

Note no mapping for :nfkc

Package

uax-15

Source

uax-15.lisp (file)

Function: normalize STR NORMALIZATION-FORM &key RFC

Base external function which calls the appropriate normalization for the normalization form. The default normaliation form is :nfkc, but :nfd, :nfkd and :nfc are also available.

Package

uax-15

Source

uax-15.lisp (file)

Function: surrogates-to-codepoint HIGH-SURROGATE LOW-SURROGATE

Translate a pair of surrogate codepoints to a non-BMP codepoint. Returns the codepoint as an integer.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Function: to-unicode-string LISP-STRING

Take a Lisp string and turn it into a vector of Unicode code points.

Package

uax-15

Source

trivial-utf-16.lisp (file)

6.1.2 Types

Type: unicode-string ()

A vector of Unicode code points.

Package

uax-15

Source

trivial-utf-16.lisp (file)

6.2 Internal definitions

6.2.1 Special variables

Special Variable: *canonical-combining-class*

Package

uax-15

Source

precomputed-tables.lisp (file)

Special Variable: *canonical-comp-map*

Package

uax-15

Source

precomputed-tables.lisp (file)

Special Variable: *canonical-decomp-map*

Package

uax-15

Source

precomputed-tables.lisp (file)

Special Variable: *compatible-decomp-map*

Package

uax-15

Source

precomputed-tables.lisp (file)

Special Variable: *composition-exclusions-data*

Package

uax-15

Source

precomputed-tables.lisp (file)

Special Variable: *data-directory*

Package

uax-15

Source

precomputed-tables.lisp (file)

Special Variable: *derived-normalization-props-data*

Package

uax-15

Source

uax-15.lisp (file)

Special Variable: *derived-normalization-props-data-file*

Package

uax-15

Source

uax-15.lisp (file)

Special Variable: *string-file*

Package

uax-15-system

Source

uax-15.asd

Special Variable: *unicode-data*

Package

uax-15

Source

precomputed-tables.lisp (file)

6.2.2 Macros

Macro: nconcf LIST1 LIST2

Package

uax-15

Source

utilities.lisp (file)

6.2.3 Functions

Function: bad-char-error MESSAGE &key VALUE NORMALIZATION-FORM

Package

uax-15

Source

utilities.lisp (file)

Function: canonical-ordering DECOMPOSED-STRING &aux S

Package

uax-15

Source

normalize-backend.lisp (file)

Function: compose DECOMPOSED-STRING

Package

uax-15

Source

normalize-backend.lisp (file)

Function: compose-hangul STR &aux LEN

Package

uax-15

Source

normalize-backend.lisp (file)

Function: decode-utf-16 UTF-16-STRING

Turn a vector of UTF-16 code units into a vector of Unicode code points. Passes unpaired surrogate codepoints straight through.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Function: decompose S TYPE

Package

uax-15

Source

normalize-backend.lisp (file)

Function: decompose-char CHAR &optional TYPE

Package

uax-15

Source

normalize-backend.lisp (file)

Function: decompose-hangul-char CH

Package

uax-15

Source

normalize-backend.lisp (file)

Function: encode-utf-16 UNICODE-STRING

Turn a vector of Unicode code points into a vector of UTF-16 code units. Indifferent to unpaired surrogates.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Function: get-canonical-combining-class CH

Package

uax-15

Source

normalize-backend.lisp (file)

Function: int-to-hex-string INT

Package

uax-15

Source

utilities.lisp (file)

Function: nfc S

Package

uax-15

Source

normalize-backend.lisp (file)

Function: nfd S

Package

uax-15

Source

normalize-backend.lisp (file)

Function: nfkc S

Package

uax-15

Source

normalize-backend.lisp (file)

Function: nfkd S

Package

uax-15

Source

normalize-backend.lisp (file)

Function: normalize-char CHR NORMALIZATION-FORM

Runs normalize on a single character input and returns a single character string. You must provide the normalization form (:nfd, :nfkd, :nfc, or :nfkc)

Package

uax-15

Source

uax-15.lisp (file)

Function: parse-hex-list-to-string LST

Takes a list of numbers and returns a string of characters

Package

uax-15

Source

utilities.lisp (file)

Function: parse-hex-string-to-char STR

Parse a hex string which is a single character into a character using code-char.

Package

uax-15

Source

utilities.lisp (file)

Function: parse-hex-string-to-int STR

Parse a string which is a single character in hex to a decimal.

Package

uax-15

Source

utilities.lisp (file)

Function: parse-hex-string-to-string STR

Takes a string which may be one or more hex numbers e.g. ’0044 0307’, builds an array of characters, coerces to string and returns the string. Mostly used for testing.

Package

uax-15

Source

utilities.lisp (file)

6.2.4 Generic functions

Generic Function: bad-char-error-message CONDITION

Generic Function: (setf bad-char-error-message) NEW-VALUE CONDITION

Package

uax-15

Methods

Method: bad-char-error-message (CONDITION bad-char-error)

Method: (setf bad-char-error-message) NEW-VALUE (CONDITION bad-char-error)

Source

utilities.lisp (file)

Generic Function: bad-char-error-normalization-form CONDITION

Generic Function: (setf bad-char-error-normalization-form) NEW-VALUE CONDITION

Package

uax-15

Methods

Method: bad-char-error-normalization-form (CONDITION bad-char-error)

Method: (setf bad-char-error-normalization-form) NEW-VALUE (CONDITION bad-char-error)

Source

utilities.lisp (file)

Generic Function: bad-char-error-value CONDITION

Generic Function: (setf bad-char-error-value) NEW-VALUE CONDITION

Package

uax-15

Methods

Method: bad-char-error-value (CONDITION bad-char-error)

Method: (setf bad-char-error-value) NEW-VALUE (CONDITION bad-char-error)

Source

utilities.lisp (file)

6.2.5 Conditions

Condition: bad-char-error ()

Package

uax-15

Source

utilities.lisp (file)

Direct superclasses

error (condition)

Direct methods

• print-object (method)
• bad-char-error-normalization-form (method)
• bad-char-error-normalization-form (method)
• bad-char-error-value (method)
• bad-char-error-value (method)
• bad-char-error-message (method)
• bad-char-error-message (method)

Direct slots

Slot: message

Text message indicating what went wrong with the validation.

Initargs

:message

Initform

(quote nil)

Readers

bad-char-error-message (generic function)

Writers

(setf bad-char-error-message) (generic function)

Slot: value

The value of the field for which the error is signalled.

Initargs

:value

Initform

(quote nil)

Readers

bad-char-error-value (generic function)

Writers

(setf bad-char-error-value) (generic function)

Slot: normalization-form

The normalization form for the error was signalled.

Initargs

:normalization-form

Initform

(quote nil)

Readers

bad-char-error-normalization-form (generic function)

Writers

(setf bad-char-error-normalization-form) (generic function)

6.2.6 Types

Type: high-surrogate ()

A Unicode High Surrogate.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Type: low-surrogate ()

A Unicode Low Surrogate.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Type: unicode-point ()

A Unicode code point.

Package

uax-15

Source

trivial-utf-16.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types