Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the uax-14 Reference Manual, version 1.0.0, generated automatically by Declt version 3.0 "Montgomery Scott" on Sun May 15 06:22:22 2022 GMT+0.
| • Introduction | What uax-14 is all about | |
| • Systems | The systems documentation | |
| • Files | The files documentation | |
| • Packages | The packages documentation | |
| • Definitions | The symbols documentation | |
| • Indexes | Concepts, functions, variables and data types |
## About UAX-14
This is an implementation of the "Unicode Standards Annex #14"(http://www.unicode.org/reports/tr14/)'s line breaking algorithm. It provides a fast and convenient way to determine line breaking opportunities in text.
Note that this algorithm does not support break opportunities that require morphological analysis. In order to handle such cases, please consult a system that provides this kind of capability, such as a hyphenation algorithm.
Also note that this system is completely unaware of layouting decisions. Any kind of layouting decisions, such as which breaks to pick, how to space between words, how to handle bidirectionality, and what to do in emergency situations when there are no breaks on an overfull line are left up to the user.
The system passes all tests offered by the Unicode standard.
## How To
The system will compile binary database files on first load. Should anything go wrong during this process, a note is produced on load. If you would like to prevent this automated loading, push ``uax-14-no-load`` to ``*features*`` before loading. You can then manually load the database files when convenient through ``load-databases``.
Once loaded, you can produce a list of line breaks for a string with ``list-breaks`` or break a string at every opportunity with ``break-string``. Typically however you will want to scan for the next break as you move along the string during layouting. To do so, create a breaker with ``make-breaker``, and call ``next-break`` whenever the next line break opportunity is required.
In pseudo-code, that could look something like this. We assume the local nickname ``uax-14`` for ``org.shirakumo.alloy.uax-14`` here.
::common lisp
(loop with breaker = (uax-14:make-breaker string)
with start = 0 and last = 0
do (multiple-value-bind (pos mandatory) (uax-14:next-break breaker)
(cond (mandatory
(insert-break pos)
(setf start pos))
((beyond-extents-p start pos)
(if (< last start) ; Force a break if we are overfull.
(loop while (beyond-extents-p start pos)
do (let ((next (find-last-fitting-cluster start)))
(insert-break next)
(setf start next))
finally (setf pos start))
(insert-break last))))
(setf last pos)))
::
## External Files
The following files are from their corresponding external sources, last accessed on 2019.09.03:
- ``LineBreak.txt`` https://www.unicode.org/Public/UCD/latest/ucd/LineBreak.txt
- ``LineBreakTest.txt`` https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/LineBreakTest.txt
At the time, Unicode 12.1 was considered the latest version.
## Acknowledgements
The code in this project is largely based on the "linebreak"(https://github.com/foliojs/linebreak) project by Devon Govett et al.
Next: Files, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
| • The uax-14 system |
Nicolas Hafner <shinmera@tymoon.eu>
Nicolas Hafner <shinmera@tymoon.eu>
(:git "https://github.com/shinmera/uax-14.git")
zlib
Implementation of the Unicode Standards Annex #14’s line breaking algorithm
1.0.0
documentation-utils
uax-14.asd (file)
Files are sorted by type and then listed depth-first from the systems components trees.
| • Lisp files |
| • The uax-14.asd file | ||
| • The uax-14/package.lisp file | ||
| • The uax-14/database.lisp file | ||
| • The uax-14/uax-14.lisp file | ||
| • The uax-14/documentation.lisp file |
Next: The uax-14/package․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
uax-14.asd
uax-14 (system)
Next: The uax-14/database․lisp file, Previous: The uax-14․asd file, Up: Lisp files [Contents][Index]
uax-14 (system)
package.lisp
Next: The uax-14/uax-14․lisp file, Previous: The uax-14/package․lisp file, Up: Lisp files [Contents][Index]
package.lisp (file)
uax-14 (system)
database.lisp
Next: The uax-14/documentation․lisp file, Previous: The uax-14/database․lisp file, Up: Lisp files [Contents][Index]
database.lisp (file)
uax-14 (system)
uax-14.lisp
Previous: The uax-14/uax-14․lisp file, Up: Lisp files [Contents][Index]
uax-14.lisp (file)
uax-14 (system)
documentation.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
| • The org.shirakumo.alloy.uax-14 package |
package.lisp (file)
common-lisp
Definitions are sorted by export status, category, package, and then by lexicographic order.
| • Exported definitions | ||
| • Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
| • Exported special variables | ||
| • Exported functions | ||
| • Exported conditions | ||
| • Exported structures |
Next: Exported functions, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
Variable containing the absolute path of the line break database file.
See LOAD-DATABASES
See COMPILE-DATABASES
database.lisp (file)
Variable containing the absolute path of the pair table file.
See LOAD-DATABASES
See COMPILE-DATABASES
database.lisp (file)
Next: Exported conditions, Previous: Exported special variables, Up: Exported definitions [Contents][Index]
Returns a list of all the pieces of the string, broken.
If MANDATORY-ONLY is T, the string is only split at mandatory line break opportunities, otherwise it is split at every opportunity.
See MAKE-BREAKER
See NEXT-BREAK
uax-14.lisp (file)
Compiles the database files from their sources.
This will load an optional part of the system and compile the database
files to an efficient byte representation. If the compilation is
successful, LOAD-DATABASES is called automatically.
See *LINE-BREAK-DATABASE-FILE*
See *PAIR-TABLE-FILE*
See LOAD-DATABASES
database.lisp (file)
Returns a list of all line break opportunities in the string.
The list has the following form:
LIST ::= ENTRY+
ENTRY ::= (position mandatory)
This is equivalent to constructing a breaker and collecting the values
of NEXT-BREAK in a loop.
See MAKE-BREAKER
See NEXT-BREAK
uax-14.lisp (file)
Loads the databases from their files into memory.
If one of the files is missing, a warning of type NO-DATABASE-FILES is
signalled. If the loading succeeds, T is returned.
See *LINE-BREAK-DATABASE-FILE*
See *PAIR-TABLE-FILE*
See NO-DATABASE-FILES
database.lisp (file)
Returns a breaker that can find line break opportunities in the given string.
If the optional breaker argument is supplied, the supplied breaker is
modified and reset to work with the new string instead. This allows
you to re-use a breaker.
Note that while you may pass a non-simple string, modifying this
string without resetting any breaker using it will result in undefined
behaviour.
See BREAKER
uax-14.lisp (file)
Returns the next line breaking opportunity of the breaker, if any.
Returns two values:
POSITION — The character index in the string at which the break
is located, or NIL if no further breaks are possible.
MANDATORY — Whether the break must be made at this location.
Note that there is always in the very least one break opportunity,
namely at the end of the string. However, after consuming this break
opportunity, NEXT-BREAK will return NIL.
Note that you may have to insert additional line breaks as required by
the layout constraints.
See BREAKER
uax-14.lisp (file)
Next: Exported structures, Previous: Exported functions, Up: Exported definitions [Contents][Index]
Warning signalled when LOAD-DATABASES is called and the files are not present.
Two restarts must be active when this condition is signalled:
COMPILE — Call COMPILE-DATABASES
ABORT — Abort loading the databases, leaving them at their
previous state.
See LOAD-DATABASES
database.lisp (file)
warning (condition)
Previous: Exported conditions, Up: Exported definitions [Contents][Index]
Contains line breaking state.
An instance of this is only useful for passing to MAKE-BREAKER and
NEXT-BREAK. It contains internal state that manages the line breaking
algorithm.
See MAKE-BREAKER
See NEXT-BREAK
uax-14.lisp (file)
structure-object (structure)
print-object (method)
string
breaker-string (function)
(setf breaker-string) (function)
org.shirakumo.alloy.uax-14::idx
0
breaker-pos (function)
(setf breaker-pos) (function)
org.shirakumo.alloy.uax-14::idx
0
breaker-last-pos (function)
(setf breaker-last-pos) (function)
(unsigned-byte 8)
0
breaker-cur-class (function)
(setf breaker-cur-class) (function)
(unsigned-byte 8)
0
breaker-next-class (function)
(setf breaker-next-class) (function)
boolean
breaker-lb8a (function)
(setf breaker-lb8a) (function)
boolean
breaker-lb21a (function)
(setf breaker-lb21a) (function)
org.shirakumo.alloy.uax-14::idx
0
breaker-lb30a (function)
(setf breaker-lb30a) (function)
Previous: Exported definitions, Up: Definitions [Contents][Index]
| • Internal special variables | ||
| • Internal macros | ||
| • Internal compiler macros | ||
| • Internal functions | ||
| • Internal types |
Next: Internal macros, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
database.lisp (file)
Next: Internal compiler macros, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
database.lisp (file)
Next: Internal functions, Previous: Internal macros, Up: Internal definitions [Contents][Index]
database.lisp (file)
database.lisp (file)
Next: Internal types, Previous: Internal compiler macros, Up: Internal definitions [Contents][Index]
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
database.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
database.lisp (file)
database.lisp (file)
database.lisp (file)
uax-14.lisp (file)
uax-14.lisp (file)
database.lisp (file)
database.lisp (file)
database.lisp (file)
database.lisp (file)
Previous: Internal functions, Up: Internal definitions [Contents][Index]
database.lisp (file)
database.lisp (file)
Previous: Definitions, Up: Top [Contents][Index]
| • Concept index | ||
| • Function index | ||
| • Variable index | ||
| • Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
| Jump to: | F L U |
|---|
| Jump to: | F L U |
|---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
| Jump to: | %
(
B C D F H L M N P T |
|---|
| Jump to: | %
(
B C D F H L M N P T |
|---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
| Jump to: | *
C L N P S |
|---|
| Jump to: | *
C L N P S |
|---|
Previous: Variable index, Up: Indexes [Contents][Index]
| Jump to: | B C I N O P S T U |
|---|
| Jump to: | B C I N O P S T U |
|---|