The lowlight Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The lowlight Reference Manual

This is the lowlight Reference Manual, version 0.99.0, generated automatically by Declt version 2.3 "Robert April" on Wed Mar 14 04:16:07 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

lowlight

A simple syntax highlighter in Common Lisp based on a lexer and a parser.

Version 1.0 approaching!

Features

Supported languages

I will probably support more languages in the future, but you can also define your own language styles. Also, patches to get your language style included are welcome.

Installation

Lowlight is already available in Quicklisp, but at the time of writing this you will find an old version there. If you want the latest version lowlight, please download the source and load it via ASDF (or as a local Quicklisp project), or make sure. You know, that the Quicklisp version has been updated to the current version, if it defines a package lowlight.1.

Usage

All lowlight functions and macros live in the package lowlight:

(in-package #:lowlight)

To highlight a string use light:

(light :common-lisp ":bla") ;=> "<span class=\"keyword\">:bla</span>"

If you want to highlight a whole file, use light-file:

(light-file :common-lisp "~/lowlight.lisp" :css "github-colors.css")

Known Bugs an Limitations

Lexing problems

As lowlight uses graylex for lexing, it currently suffers from a bug related do the buffer size graylex uses. If you experience Problems with the lexer not correctly recognizing tokens that cross the 1024 or 2048 character border, try to increase the buffer size lowlight:*lexer-buffer-size*.

Parsing problems

The grammars that are created by define-cfg-style are ambiguous by design, so conflict warnings will be ignored by default. If you run into problems, please consider using define-simple-style or tell me, how to create unambiguous grammars in define-cfg-style. Maybe I will provide a third macro to explicitly specify a complete cfg, so you can make sure it's unambiguous yourself (if anyone actually needs this).


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 lowlight

Author

Christoph Finkensiep <chfin@freenet.de>

License

MIT/X11

Description

A simple and flexible syntax highlighter

Version

0.99.0

Dependencies
Source

lowlight.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 lowlight.asd

Location

lowlight.asd

Systems

lowlight (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 lowlight/package.lisp

Parent

lowlight (system)

Location

package.lisp

Packages

lowlight.1


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.3 lowlight/common.lisp

Dependency

package.lisp (file)

Parent

lowlight (system)

Location

common.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.4 lowlight/simple.lisp

Dependency

common.lisp (file)

Parent

lowlight (system)

Location

simple.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.5 lowlight/grammar.lisp

Dependency

simple.lisp (file)

Parent

lowlight (system)

Location

grammar.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.6 lowlight/lowlight.lisp

Dependency

grammar.lisp (file)

Parent

lowlight (system)

Location

lowlight.lisp

Exported Definitions
Internal Definitions

Previous: , Up: Lisp files   [Contents][Index]

3.1.7 lowlight/style-lisp.lisp

Dependency

lowlight.lisp (file)

Parent

lowlight (system)

Location

style-lisp.lisp

Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 lowlight.1

Source

package.lisp (file)

Nicknames
Use List

common-lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Special variables

Special Variable: *debug*

controls debug output (‘t‘ or ‘nil‘)

Package

lowlight.1

Source

common.lisp (file)

Special Variable: *lexer-buffer-size*

controlling knob for the graylex buffer size

Package

lowlight.1

Source

common.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Macros

Macro: define-cfg-style NAME RULES PRODUCTIONS

Defines a cfg style with token rules ‘rules‘ and CFG productions ‘productions‘.
The resulting ‘cfg-style‘ object is registered under ‘name‘ which should be a keyword.

‘rules‘ is a list of token rules, each of the form ‘(regex class)‘,
where ‘regex‘ is either a string describing a regex or a list with such a string in ‘car‘. In the latter case the string is taken as a control string for ‘format‘
and the rest of the list is given to it as arguments.
The result is then used as the regex.
‘class‘ can be a keyword or a function. If it is a function, it is called with the token detected by ‘regex‘ and should return a keyword.
Rules may contain forms like ‘(:other-tokens ...)‘, each containing a list of keywords that do not appear as a token keyword in other rules (e.g. ‘(:other-tokens :a :b :c)‘). Those are added to the list of terminals for the production rules.
You should use this for token keywords that are generated by functions,
as described above.

‘productions‘ is a list of productions in cl-yacc syntax.
The productions may contain *one* form ‘(:toplevel ...)‘ containing a list of toplevel non-terminals (e.g. ‘(:toplevel a b c)‘).
If no such form is given, all nontermials on the left-hand side of the given productions will become toplevel.
Productions may also contain *one* form ‘(:ignore ...)‘ containing a list of ignorable tokens. Those may occur at any positition (like whitespace and comments)
and are preserved, if all tokens are printed with ‘print-token*‘.
If no ‘:ignore‘ form is given, it defaults to ‘’(:default)‘.

Package

lowlight.1

Source

grammar.lisp (file)

Macro: define-simple-style NAME &body RULES

Defines a simple style with the token rules ‘rules‘.

The resulting ‘simple-style‘ object is registered under ‘name‘ which should be a keyword. ‘rules‘ is a list of token rules, each of the form ‘(regex class)‘,
where ‘regex‘ is either a string describing a regex or a list with such a string in ‘car‘. In the latter case the string is taken as a control string for ‘format‘
and the rest of the list is given to it as arguments.
The result is then used as the regex.
‘class‘ can be a keyword or a function. If it is a function, it is called with the token detected by ‘regex‘ and should return a keyword.

Package

lowlight.1

Source

simple.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.3 Functions

Function: light STYLE INPUT &optional OUTPUT

=> a highlighted string or ‘t‘
Highlights ‘input‘ which can be either a stream or a string. The used style is given by ‘style‘.
If ‘output‘ (a stream) is given, the result is printed on it, otherwise the result is returned as a string.

Package

lowlight.1

Source

lowlight.lisp (file)

Function: light-blocks STYLE INPUT &key OUTPUT BLOCKS

=> a highlighted string or ‘t‘
Highlights all markdown code blocks in ‘input‘ with ‘style‘
and writes the result to ‘output‘ (or returns the result as a string,
if ‘output‘ is not given).

‘light-blocks‘ only hightlights blocks starting with a language name from ‘blocks‘ which defaults to the name of ‘style‘. So if you are using the style ‘:common-lisp‘ and want to highlight blocks starting with ‘ “‘common-lisp‘, then you may leave out ‘blocks‘. If you want blocks starting with ‘ “‘cl‘ to be highlighted as well, use ‘’(:common-lisp :cl)‘ for ‘blocks‘.

Package

lowlight.1

Source

lowlight.lisp (file)

Function: light-file STYLE IN &key OUT CSS TITLE RAW

=> ‘t‘
Highlights the file given by ‘in‘.
The used style is denoted by ‘style‘.
If given, the result is written to the file ‘out‘,
otherwise ‘in‘ is used, with the file ending replaced by html.
If given, ‘css‘ is used as the href parameter to a css relation
and ‘title‘ is used as the page title.
If ‘raw‘ is ‘t‘ the highlighted code is *not* wrapped into a html skeleton.

Package

lowlight.1

Source

lowlight.lisp (file)

Function: light-file-blocks STYLE IN &key OUT BLOCKS

=> ‘t‘
Highlights a file with ‘light-blocks‘.
‘in‘ denotes the input file, if ‘out‘ is given, it denotes the output file. If not, ‘in‘ is used, with the file extension replaced by ‘.html‘,
so ‘foo.lisp‘ becomes ‘foo.html‘.
Be careful with files already ending on ‘.html‘ as they will not change.

Package

lowlight.1

Source

lowlight.lisp (file)

Function: make-cfg-style NAME RULES PRODUCTIONS

=> a ‘cfg-style‘ object
Creates a new ‘cfg-style‘ with the name ‘name‘, the token rules ‘rules‘ and the CFG productions ‘productions‘.
For the structure of ‘rules‘ and ‘productions‘ see ‘define-cfg-style‘.

Package

lowlight.1

Source

grammar.lisp (file)

Function: make-simple-style NAME RULES

=> a ‘simple-style‘ object
Creates a new ‘simple-style‘ with the name ‘name‘.
‘rules‘ is a list of token rules, each of the form described at ‘define-simple-style‘.

Package

lowlight.1

Source

simple.lisp (file)


Previous: , Up: Exported definitions   [Contents][Index]

5.1.4 Generic functions

Generic Function: style-classes STYLE

=> the set of token classes generated by ‘style‘

Package

lowlight.1

Source

common.lisp (file)

Methods
Method: style-classes (STYLE simple-style)
Source

simple.lisp (file)

Method: style-classes STYLE

Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Special variables

Special Variable: *parser*
Package

lowlight.1

Source

style-lisp.lisp (file)

Special Variable: *styles*

An alist mapping from style keywords to style definitions.

Package

lowlight.1

Source

common.lisp (file)

Special Variable: *token-printer*
Package

lowlight.1

Source

grammar.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.2 Macros

Macro: make-cfg-style% NAME RULES PRODUCTIONS
Package

lowlight.1

Source

grammar.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.3 Functions

Function: classify-symbol STR
Package

lowlight.1

Source

style-lisp.lisp (file)

Function: inject-ignorables TERMINALS IGNORES PRODUCTIONS

=> a list of productions
Takes a list of terminals, a list of ignorables, and a list of productions in cl-yacc syntax. Transforms the productions to allow an arbitrary number of ignorables after each terminal.

Package

lowlight.1

Source

grammar.lisp (file)

Function: light-blocks% STYLE INPUT OUTPUT BLOCKS

highlight all blocks in input

Package

lowlight.1

Source

lowlight.lisp (file)

Function: light-blocks%* STYLE INPUT OUTPUT BLOCKS
Package

lowlight.1

Source

lowlight.lisp (file)

Function: make-blocks-regex BLOCKS
Package

lowlight.1

Source

lowlight.lisp (file)

Function: maprec FN TREE &optional DEPTH
Package

lowlight.1

Source

grammar.lisp (file)

Function: print-define DEF NAME
Package

lowlight.1

Source

style-lisp.lisp (file)

Function: print-if CND OBJ &optional STREAM
Package

lowlight.1

Source

grammar.lisp (file)

Function: print-setf-define DEF SETF NAME
Package

lowlight.1

Source

style-lisp.lisp (file)

Function: print-token CLASS TOKEN OUTPUT
Package

lowlight.1

Source

simple.lisp (file)

Function: print-token* CLASS TOKEN
Package

lowlight.1

Source

grammar.lisp (file)

Function: print-typedef DEF NAME
Package

lowlight.1

Source

style-lisp.lisp (file)

Function: print-vardef DEF NAME
Package

lowlight.1

Source

style-lisp.lisp (file)

Function: read-into-string IN-STREAM &key BUFFER-SIZE

=> a string containing the whole stream data from ‘in-stream‘ Taken from Alexandria’s ‘read-file-into-string.‘

Package

lowlight.1

Source

lowlight.lisp (file)

Function: rules-to-grammar RULES PRODUCTIONS

=> an ignorable sensitive cl-yacc grammar
Takes style rules and productions and creates a cl-yacc grammar.

Rules may contain forms like ‘(:other-tokens ...)‘, each containing a list of keywords that do not appear as a token keyword in other rules (e.g. ‘(:other-tokens :a :b :c)‘). Those are added to the list of terminals for the production rules.

Productions may contain one form ‘(:toplevel ...)‘ containing a list of toplevel non-terminals (e.g. ‘(:toplevel a b c)‘).
If no such form is given, all nontermials on the left-hand side of the given productions will become toplevel.
Productions may also contain one form ‘(:ignore ...)‘ containing a list of ignorable tokens. Those may occur at any positition and are preserved, if all tokens are printed with ‘print-token*‘.
If no ‘:ignore‘ form is given, it defaults to ‘’(:default)‘.

Package

lowlight.1

Source

grammar.lisp (file)

Function: tokenize LEXER OUTPUT
Package

lowlight.1

Source

simple.lisp (file)

Function: tree-sub TABLE TREE &optional DEPTH
Package

lowlight.1

Source

grammar.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.4 Generic functions

Generic Function: get-style STYLE

=> a style denoted by ‘style‘ being either a key or an actual style

Package

lowlight.1

Source

common.lisp (file)

Methods
Method: get-style (STYLE simple-style)
Source

simple.lisp (file)

Method: get-style STYLE
Generic Function: process-style STYLE INPUT OUTPUT
Package

lowlight.1

Source

common.lisp (file)

Methods
Method: process-style (STYLE cfg-style) INPUT OUTPUT

Process a stream with a ‘style‘.

Source

grammar.lisp (file)

Method: process-style (STYLE simple-style) INPUT OUTPUT

Process a stream with a ‘simple-style‘.

Source

simple.lisp (file)

Generic Function: style-name STYLE

=> the name of ‘style‘

Package

lowlight.1

Source

common.lisp (file)

Methods
Method: style-name (SIMPLE-STYLE simple-style)

automatically generated reader method

Source

simple.lisp (file)

Method: style-name STYLE
Generic Function: style-parser OBJECT
Package

lowlight.1

Methods
Method: style-parser (CFG-STYLE cfg-style)

automatically generated reader method

Source

grammar.lisp (file)

Generic Function: style-rules OBJECT
Package

lowlight.1

Methods
Method: style-rules (SIMPLE-STYLE simple-style)

automatically generated reader method

Source

simple.lisp (file)


Previous: , Up: Internal definitions   [Contents][Index]

5.2.5 Classes

Class: cfg-style ()
Package

lowlight.1

Source

grammar.lisp (file)

Direct superclasses

simple-style (class)

Direct methods
Direct slots
Slot: parser
Initargs

:parser

Readers

style-parser (generic function)

Class: simple-style ()
Package

lowlight.1

Source

simple.lisp (file)

Direct superclasses

standard-object (class)

Direct subclasses

cfg-style (class)

Direct methods
Direct slots
Slot: rules
Initargs

:rules

Readers

style-rules (generic function)

Slot: name
Initargs

:name

Readers

style-name (generic function)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L  
Index Entry  Section

F
File, Lisp, lowlight.asd: The lowlight<dot>asd file
File, Lisp, lowlight/common.lisp: The lowlight/common<dot>lisp file
File, Lisp, lowlight/grammar.lisp: The lowlight/grammar<dot>lisp file
File, Lisp, lowlight/lowlight.lisp: The lowlight/lowlight<dot>lisp file
File, Lisp, lowlight/package.lisp: The lowlight/package<dot>lisp file
File, Lisp, lowlight/simple.lisp: The lowlight/simple<dot>lisp file
File, Lisp, lowlight/style-lisp.lisp: The lowlight/style-lisp<dot>lisp file

L
Lisp File, lowlight.asd: The lowlight<dot>asd file
Lisp File, lowlight/common.lisp: The lowlight/common<dot>lisp file
Lisp File, lowlight/grammar.lisp: The lowlight/grammar<dot>lisp file
Lisp File, lowlight/lowlight.lisp: The lowlight/lowlight<dot>lisp file
Lisp File, lowlight/package.lisp: The lowlight/package<dot>lisp file
Lisp File, lowlight/simple.lisp: The lowlight/simple<dot>lisp file
Lisp File, lowlight/style-lisp.lisp: The lowlight/style-lisp<dot>lisp file
lowlight.asd: The lowlight<dot>asd file
lowlight/common.lisp: The lowlight/common<dot>lisp file
lowlight/grammar.lisp: The lowlight/grammar<dot>lisp file
lowlight/lowlight.lisp: The lowlight/lowlight<dot>lisp file
lowlight/package.lisp: The lowlight/package<dot>lisp file
lowlight/simple.lisp: The lowlight/simple<dot>lisp file
lowlight/style-lisp.lisp: The lowlight/style-lisp<dot>lisp file

Jump to:   F   L  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   C   D   F   G   I   L   M   P   R   S   T  
Index Entry  Section

C
classify-symbol: Internal functions

D
define-cfg-style: Exported macros
define-simple-style: Exported macros

F
Function, classify-symbol: Internal functions
Function, inject-ignorables: Internal functions
Function, light: Exported functions
Function, light-blocks: Exported functions
Function, light-blocks%: Internal functions
Function, light-blocks%*: Internal functions
Function, light-file: Exported functions
Function, light-file-blocks: Exported functions
Function, make-blocks-regex: Internal functions
Function, make-cfg-style: Exported functions
Function, make-simple-style: Exported functions
Function, maprec: Internal functions
Function, print-define: Internal functions
Function, print-if: Internal functions
Function, print-setf-define: Internal functions
Function, print-token: Internal functions
Function, print-token*: Internal functions
Function, print-typedef: Internal functions
Function, print-vardef: Internal functions
Function, read-into-string: Internal functions
Function, rules-to-grammar: Internal functions
Function, tokenize: Internal functions
Function, tree-sub: Internal functions

G
Generic Function, get-style: Internal generic functions
Generic Function, process-style: Internal generic functions
Generic Function, style-classes: Exported generic functions
Generic Function, style-name: Internal generic functions
Generic Function, style-parser: Internal generic functions
Generic Function, style-rules: Internal generic functions
get-style: Internal generic functions
get-style: Internal generic functions
get-style: Internal generic functions

I
inject-ignorables: Internal functions

L
light: Exported functions
light-blocks: Exported functions
light-blocks%: Internal functions
light-blocks%*: Internal functions
light-file: Exported functions
light-file-blocks: Exported functions

M
Macro, define-cfg-style: Exported macros
Macro, define-simple-style: Exported macros
Macro, make-cfg-style%: Internal macros
make-blocks-regex: Internal functions
make-cfg-style: Exported functions
make-cfg-style%: Internal macros
make-simple-style: Exported functions
maprec: Internal functions
Method, get-style: Internal generic functions
Method, get-style: Internal generic functions
Method, process-style: Internal generic functions
Method, process-style: Internal generic functions
Method, style-classes: Exported generic functions
Method, style-classes: Exported generic functions
Method, style-name: Internal generic functions
Method, style-name: Internal generic functions
Method, style-parser: Internal generic functions
Method, style-rules: Internal generic functions

P
print-define: Internal functions
print-if: Internal functions
print-setf-define: Internal functions
print-token: Internal functions
print-token*: Internal functions
print-typedef: Internal functions
print-vardef: Internal functions
process-style: Internal generic functions
process-style: Internal generic functions
process-style: Internal generic functions

R
read-into-string: Internal functions
rules-to-grammar: Internal functions

S
style-classes: Exported generic functions
style-classes: Exported generic functions
style-classes: Exported generic functions
style-name: Internal generic functions
style-name: Internal generic functions
style-name: Internal generic functions
style-parser: Internal generic functions
style-parser: Internal generic functions
style-rules: Internal generic functions
style-rules: Internal generic functions

T
tokenize: Internal functions
tree-sub: Internal functions

Jump to:   C   D   F   G   I   L   M   P   R   S   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   *  
N   P   R   S  
Index Entry  Section

*
*debug*: Exported special variables
*lexer-buffer-size*: Exported special variables
*parser*: Internal special variables
*styles*: Internal special variables
*token-printer*: Internal special variables

N
name: Internal classes

P
parser: Internal classes

R
rules: Internal classes

S
Slot, name: Internal classes
Slot, parser: Internal classes
Slot, rules: Internal classes
Special Variable, *debug*: Exported special variables
Special Variable, *lexer-buffer-size*: Exported special variables
Special Variable, *parser*: Internal special variables
Special Variable, *styles*: Internal special variables
Special Variable, *token-printer*: Internal special variables

Jump to:   *  
N   P   R   S  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   C   L   P   S  
Index Entry  Section

C
cfg-style: Internal classes
Class, cfg-style: Internal classes
Class, simple-style: Internal classes

L
lowlight: The lowlight system
lowlight.1: The lowlight<dot>1 package

P
Package, lowlight.1: The lowlight<dot>1 package

S
simple-style: Internal classes
System, lowlight: The lowlight system

Jump to:   C   L   P   S