Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the cl-lexer Reference Manual, version 1.4, generated automatically by Declt version 3.0 "Montgomery Scott" on Sun May 15 03:55:18 2022 GMT+0.
• Introduction | What cl-lexer is all about | |
• Systems | The systems documentation | |
• Files | The files documentation | |
• Packages | The packages documentation | |
• Definitions | The symbols documentation | |
• Indexes | Concepts, functions, variables and data types |
CL-LEXER package The CL-LEXER package implements a lexical-analyzer-generator called DEFLEXER, which is built on top of both REGEX and CLAWK. Many of the optimizations in the recent rewrite of the regex engine went into optimizing the sorts of patterns generated by DEFLEX. The default lexer doesn't implement full greediness. If you have a rule for ints followed by a rule for floats, the int rule will match on the part before the decimal before the float rule gets a chance to look at it. You can fix this by specifying :flex-compatible as the first rule. This gives all patterns a chance to examine the text and takes the one that matches the longest string (first pattern wins in case of a tie). The down side of this option is that it slows down the analyser. If you can solve the issue by reordering your rules that's the way to do it. I'm currently writing an AWK->CLAWK translator using this as the lexer, and it's working fine. As far as I can tell, the DEFLEXER-generated lexing functions should be fast enough for production use. Currently, the LEX/FLEX/BISON feature of switching productions on and off using state variables is not supported, but it's a pretty simple feature to add. If you're using CL-LEXER and discover you need this feature, let me know. It also doesn't yet support prefix and postfix context patterns. This isn't quite so trivial to add, but it's planned for a future release of regex, so CL-LEXER will be getting it someday. Anyway, Here's a simple DEFLEXER example: (deflexer test-lexer ("[0-9]+([.][0-9]+([Ee][0-9]+)?)" (return (values 'flt (num %0)))) ("[0-9]+" (return (values 'int (int %0)))) ("[:alpha:][:alnum:]*" (return (values 'name %0))) ("[:space:]+") ) > (setq *lex* (test-lexer "1.0 12 fred 10.23e45"))> (funcall *lex*) FLT 1.0 > (funcall *lex*) INT 12 > (funcall *lex*) NAME "fred" > (funcall *lex*) FLT 1.0229999999999997E46 > (funcall *lex*) NIL NIL You can also write this lexer using the :flex-compatible option, in which case you can write the int and flt rules in any order. (deflexer test-lexer :flex-compatible ("[0-9]+" (return (values 'int (int %0)))) ("[0-9]+([.][0-9]+([Ee][0-9]+)?)" (return (values 'flt (num %0)))) ("[:space:]+") )
Next: Files, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
• The cl-lexer system |
Kenneth Michael Parker
BSD-new
cl-lexer: a lexical analyzer generator
1.4
regex
cl-lexer.asd (file)
Files are sorted by type and then listed depth-first from the systems components trees.
• Lisp files |
• The cl-lexer.asd file | ||
• The cl-lexer/packages.lisp file | ||
• The cl-lexer/lexer.lisp file |
Next: The cl-lexer/packages․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
cl-lexer.asd
cl-lexer (system)
Next: The cl-lexer/lexer․lisp file, Previous: The cl-lexer․asd file, Up: Lisp files [Contents][Index]
Previous: The cl-lexer/packages․lisp file, Up: Lisp files [Contents][Index]
cl-lexer (system)
lexer.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
• The cl-lexer package |
packages.lisp (file)
Definitions are sorted by export status, category, package, and then by lexicographic order.
• Exported definitions | ||
• Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
• Exported macros | ||
• Exported generic functions |
Next: Exported generic functions, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
Create a lexical analyser. This analyser function takes a string :position,
:end, :end-token and :end-value keyword parameters, and returns a function of
no arguments that returns the next token and value each time it is called,
or (values end-token end-value) when the input string is exhausted.
By default, position = 0, end = length of str, and end-token and end-value = nil.
lexer.lisp (file)
lexer.lisp (file)
Previous: Exported macros, Up: Exported definitions [Contents][Index]
lexer.lisp (file)
lexer.lisp (file)
Previous: Exported definitions, Up: Definitions [Contents][Index]
• Internal functions |
Previous: Internal definitions, Up: Internal definitions [Contents][Index]
lexer.lisp (file)
lexer.lisp (file)
lexer.lisp (file)
lexer.lisp (file)
lexer.lisp (file)
Previous: Definitions, Up: Top [Contents][Index]
• Concept index | ||
• Function index | ||
• Variable index | ||
• Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
Jump to: | C F L |
---|
Jump to: | C F L |
---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
Jump to: | C D E F G I M N T |
---|
Jump to: | C D E F G I M N T |
---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
Previous: Variable index, Up: Indexes [Contents][Index]
Jump to: | C P S |
---|
Index Entry | Section | ||
---|---|---|---|
| |||
C | |||
cl-lexer : | The cl-lexer system | ||
cl-lexer : | The cl-lexer package | ||
| |||
P | |||
Package, cl-lexer : | The cl-lexer package | ||
| |||
S | |||
System, cl-lexer : | The cl-lexer system | ||
|
Jump to: | C P S |
---|