The cl-lexer Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The cl-lexer Reference Manual

This is the cl-lexer Reference Manual, version 1.4, generated automatically by Declt version 2.3 "Robert April" on Tue Feb 20 08:08:42 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

LEXER package

The LEXER package implements a lexical-analyzer-generator called DEFLEXER,
which is built on top of both REGEX and CLAWK. Many of the optimizations in the
recent rewrite of the regex engine went into optimizing the sorts of patterns
generated by DEFLEX.

The default lexer doesn't implement full greediness. If you have a rule for
ints followed by a rule for floats, the int rule will match on the part before
the decimal before the float rule gets a chance to look at it. You can fix this
by specifying :flex-compatible as the first rule. This gives all patterns a
chance to examine the text and takes the one that matches the longest string
(first pattern wins in case of a tie). The down side of this option is that it
slows down the analyser. If you can solve the issue by reordering your rules
that's the way to do it.

I'm currently writing an AWK->CLAWK translator using this as the lexer, and
it's working fine. As far as I can tell, the DEFLEXER-generated lexing
functions should be fast enough for production use.

Currently, the LEX/FLEX/BISON feature of switching productions on and off using
state variables is not supported, but it's a pretty simple feature to add. If
you're using LEXER and discover you need this feature, let me know.

It also doesn't yet support prefix and postfix context patterns. This isn't
quite so trivial to add, but it's planned for a future release of regex, so
LEXER will be getting it someday.

Anyway, Here's a simple DEFLEXER example:

  (deflexer test-lexer
    ("[0-9]+([.][0-9]+([Ee][0-9]+)?)"
      (return (values 'flt (num %0))))
    ("[0-9]+"
      (return (values 'int (int %0))))
    ("[:alpha:][:alnum:]*"
      (return (values 'name %0)))
    ("[:space:]+") )

  > (setq *lex* (test-lexer "1.0 12 fred 10.23e45"))
  
 
  > (funcall *lex*)
  FLT
  1.0
 
  > (funcall *lex*)
  INT
  12
 
  > (funcall *lex*)
  NAME
  "fred"

  > (funcall *lex*)
  FLT
  1.0229999999999997E46

  > (funcall *lex*)
  NIL
  NIL

You can also write this lexer using the :flex-compatible option, in which case
you can write the int and flt rules in any order.

(deflexer test-lexer
  :flex-compatible
  ("[0-9]+"
    (return (values 'int (int %0))))
  ("[0-9]+([.][0-9]+([Ee][0-9]+)?)"
    (return (values 'flt (num %0))))
  ("[:space:]+")
 )



Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 lexer

Author

Kenneth Michael Parker

License

BSD-new

Description

cl-lexer: a lexical analyzer generator

Version

1.4

Dependency

regex

Source

lexer.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 lexer.asd

Location

lexer.asd

Systems

lexer (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 lexer/packages.lisp

Parent

lexer (system)

Location

packages.lisp

Packages

lexer


Previous: , Up: Lisp files   [Contents][Index]

3.1.3 lexer/lexer.lisp

Parent

lexer (system)

Location

lexer.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 lexer

Source

packages.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Macros

Macro: deflexer NAME &rest RULES

Create a lexical analyser. This analyser function takes a string :position, :end, :end-token and :end-value keyword parameters, and returns a function of no arguments that returns the next token and value each time it is called, or (values end-token end-value) when the input string is exhausted.
By default, position = 0, end = length of str, and end-token and end-value = nil.

Package

lexer

Source

lexer.lisp (file)

Macro: tokenize STR POS-VAR &rest RULES
Package

lexer

Source

lexer.lisp (file)


Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Generic functions

Generic Function: int X
Package

lexer

Source

lexer.lisp (file)

Methods
Method: int (X string)
Method: int (X (eql nil))
Method: int (X number)
Method: int (X integer)
Generic Function: num X
Package

lexer

Source

lexer.lisp (file)

Methods
Method: num (X string)
Method: num (X (eql nil))
Method: num (X number)

Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Functions

Function: combine-patterns PATS
Package

lexer

Source

lexer.lisp (file)

Function: expand-tokenize STR POS RULES &aux STRVAR MATCHEDP
Package

lexer

Source

lexer.lisp (file)

Function: expand-tokenize-rules STR POS MATCHEDP RULES
Package

lexer

Source

lexer.lisp (file)

Function: extract-patterns-and-actions RULES &aux PATTERNS ACTIONS
Package

lexer

Source

lexer.lisp (file)

Function: make-lexer-actions ACTIONS
Package

lexer

Source

lexer.lisp (file)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L  
Index Entry  Section

F
File, Lisp, lexer.asd: The lexer<dot>asd file
File, Lisp, lexer/lexer.lisp: The lexer/lexer<dot>lisp file
File, Lisp, lexer/packages.lisp: The lexer/packages<dot>lisp file

L
lexer.asd: The lexer<dot>asd file
lexer/lexer.lisp: The lexer/lexer<dot>lisp file
lexer/packages.lisp: The lexer/packages<dot>lisp file
Lisp File, lexer.asd: The lexer<dot>asd file
Lisp File, lexer/lexer.lisp: The lexer/lexer<dot>lisp file
Lisp File, lexer/packages.lisp: The lexer/packages<dot>lisp file

Jump to:   F   L  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   C   D   E   F   G   I   M   N   T  
Index Entry  Section

C
combine-patterns: Internal functions

D
deflexer: Exported macros

E
expand-tokenize: Internal functions
expand-tokenize-rules: Internal functions
extract-patterns-and-actions: Internal functions

F
Function, combine-patterns: Internal functions
Function, expand-tokenize: Internal functions
Function, expand-tokenize-rules: Internal functions
Function, extract-patterns-and-actions: Internal functions
Function, make-lexer-actions: Internal functions

G
Generic Function, int: Exported generic functions
Generic Function, num: Exported generic functions

I
int: Exported generic functions
int: Exported generic functions
int: Exported generic functions
int: Exported generic functions
int: Exported generic functions

M
Macro, deflexer: Exported macros
Macro, tokenize: Exported macros
make-lexer-actions: Internal functions
Method, int: Exported generic functions
Method, int: Exported generic functions
Method, int: Exported generic functions
Method, int: Exported generic functions
Method, num: Exported generic functions
Method, num: Exported generic functions
Method, num: Exported generic functions

N
num: Exported generic functions
num: Exported generic functions
num: Exported generic functions
num: Exported generic functions

T
tokenize: Exported macros

Jump to:   C   D   E   F   G   I   M   N   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables


Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   L   P   S  
Index Entry  Section

L
lexer: The lexer system
lexer: The lexer package

P
Package, lexer: The lexer package

S
System, lexer: The lexer system

Jump to:   L   P   S