The graylex Reference Manual

This is the graylex Reference Manual, version 0.0.1, generated automatically by Declt version 4.0 beta 2 "William Riker" on Mon Feb 26 16:37:36 2024 GMT+0.

Table of Contents


1 Introduction


2 Systems

The main system appears first, followed by any subsystem dependency.


2.1 graylex

Gray lexer input streams

Author

Alexander Kahl <>

License

GPLv3+

Version

0.0.1

Dependencies
  • cl-ppcre (system).
  • trivial-gray-streams (system).
  • alexandria (system).
Source

graylex.asd.

Child Component

src (module).


3 Modules

Modules are listed depth-first from the system components tree.


3.1 graylex/src

Source

graylex.asd.

Parent Component

graylex (system).

Child Components

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


4.1 Lisp


4.1.1 graylex/graylex.asd

Source

graylex.asd.

Parent Component

graylex (system).

ASDF Systems

graylex.

Packages

graylex-system.


4.1.2 graylex/src/package.lisp

Source

graylex.asd.

Parent Component

src (module).

Packages

graylex.


4.1.3 graylex/src/buffered-input-stream.lisp

Dependency

package.lisp (file).

Source

graylex.asd.

Parent Component

src (module).

Public Interface
Internals

4.1.4 graylex/src/lexer-input-stream.lisp

Dependency

buffered-input-stream.lisp (file).

Source

graylex.asd.

Parent Component

src (module).

Public Interface
Internals

5 Packages

Packages are listed by definition order.


5.1 graylex

Source

package.lisp.

Use List
  • common-lisp.
  • trivial-gray-streams.
Public Interface
Internals

5.2 graylex-system

Source

graylex.asd.

Use List
  • asdf/interface.
  • common-lisp.

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


6.1 Public Interface


6.1.1 Generic functions

Generic Reader: buffered-input-buffer (object)
Generic Writer: (setf buffered-input-buffer) (object)
Package

graylex.

Methods
Reader Method: buffered-input-buffer ((buffered-input-stream buffered-input-stream))
Writer Method: (setf buffered-input-buffer) ((buffered-input-stream buffered-input-stream))

Wrapped input buffer

Source

buffered-input-stream.lisp.

Target Slot

buffer.

Generic Reader: buffered-input-position (object)
Generic Writer: (setf buffered-input-position) (object)
Package

graylex.

Methods
Reader Method: buffered-input-position ((buffered-input-stream buffered-input-stream))
Writer Method: (setf buffered-input-position) ((buffered-input-stream buffered-input-stream))

Reading position of the wrapped buffer

Source

buffered-input-stream.lisp.

Target Slot

buffer-position.

Generic Reader: buffered-input-size (object)
Generic Writer: (setf buffered-input-size) (object)
Package

graylex.

Methods
Reader Method: buffered-input-size ((buffered-input-stream buffered-input-stream))
Writer Method: (setf buffered-input-size) ((buffered-input-stream buffered-input-stream))

Size of the input buffer

Source

buffered-input-stream.lisp.

Target Slot

buffer-size.

Generic Reader: buffered-stream (object)
Package

graylex.

Methods
Reader Method: buffered-stream ((buffered-input-stream buffered-input-stream))

Wrapped input stream that is to be buffered

Source

buffered-input-stream.lisp.

Target Slot

stream.

Generic Function: fill-buffer (buffered-input-stream)

Fill the input buffer

Package

graylex.

Source

buffered-input-stream.lisp.

Methods
Method: fill-buffer ((stream buffered-input-stream))

fill-buffer stream => position

Fill the input buffer by reading from the wrapped stream. Also reset the reading position to zero.

Generic Function: flush-buffer (buffered-input-stream)

Flush the input buffer

Package

graylex.

Source

buffered-input-stream.lisp.

Methods
Method: flush-buffer ((stream lexer-input-stream))

flush-buffer stream => string

Return unread rest of the wrapped main buffer but also append it to the double buffer.

Source

lexer-input-stream.lisp.

Method: flush-buffer ((stream buffered-input-stream))

flush-buffer stream => string

Return unread rest of the wrapped buffer and replenish it.

Generic Reader: lexer-column (object)
Generic Writer: (setf lexer-column) (object)
Package

graylex.

Methods
Reader Method: lexer-column ((lexer-input-stream lexer-input-stream))
Writer Method: (setf lexer-column) ((lexer-input-stream lexer-input-stream))

Current column in lexer stream

Source

lexer-input-stream.lisp.

Target Slot

column.

Generic Reader: lexer-double-buffer (object)
Generic Writer: (setf lexer-double-buffer) (object)
Package

graylex.

Methods
Reader Method: lexer-double-buffer ((lexer-input-stream lexer-input-stream))
Writer Method: (setf lexer-double-buffer) ((lexer-input-stream lexer-input-stream))

Double buffer

Source

lexer-input-stream.lisp.

Target Slot

double-buffer.

Generic Reader: lexer-non-stream-position (object)
Generic Writer: (setf lexer-non-stream-position) (object)
Package

graylex.

Methods
Reader Method: lexer-non-stream-position ((lexer-input-stream lexer-input-stream))
Writer Method: (setf lexer-non-stream-position) ((lexer-input-stream lexer-input-stream))

Position in unread sequence

Source

lexer-input-stream.lisp.

Target Slot

non-stream-position.

Generic Reader: lexer-row (object)
Generic Writer: (setf lexer-row) (object)
Package

graylex.

Methods
Reader Method: lexer-row ((lexer-input-stream lexer-input-stream))
Writer Method: (setf lexer-row) ((lexer-input-stream lexer-input-stream))

Current row in lexer stream

Source

lexer-input-stream.lisp.

Target Slot

row.

Generic Reader: lexer-rules (object)
Generic Writer: (setf lexer-rules) (object)
Package

graylex.

Methods
Reader Method: lexer-rules ((lexer-input-stream lexer-input-stream))
Writer Method: (setf lexer-rules) ((lexer-input-stream lexer-input-stream))

List of regexp/keyword conses

Source

lexer-input-stream.lisp.

Target Slot

rules.

Generic Function: lexer-unread-sequence (lexer-input-stream seq)

Unread a sequence by feeding it into the double buffer

Package

graylex.

Source

lexer-input-stream.lisp.

Methods
Method: lexer-unread-sequence ((stream lexer-input-stream) seq)

lexer-unread-sequence stream seq => position

Prepend sequence SEQ to the internal double buffer and increase the non-stream position.

Generic Function: stream-read-token (lexer-input-stream &optional peek)

Read lexical tokens from the input stream

Package

graylex.

Source

lexer-input-stream.lisp.

Methods
Method: stream-read-token :before ((stream lexer-input-stream) &optional peek)

stream-read-token :before stream &optional peek => string

If the internal double buffer is empty, flush the main buffer first in order to replenish it.

Method: stream-read-token :around ((stream lexer-input-stream) &optional peek)

stream-read-token :around stream &optional peek => (class image)

Scan the result from calling the next method if PEEK is NIL:
Discard the matched part from the beginning of the double buffer and either just decrease the non-stream position or record the column and row progress.

Method: stream-read-token ((stream lexer-input-stream) &optional peek)

stream-read-token stream &optional peek => (class image)

Scan the lexer’s double buffer successively with all its rules. Rules are
expected to be conses of PCRE-compatible regular expressions and class name
keywords. Heads-up: Every rule get prepended with an implicit start
anchor ("^") to match the beginning of the buffer!

If the double buffer is empty, simply return NIL; is no rule matches, signal an UNMATCHED-LEXING-SEQUENCE with further details and provide the following
restarts:
- flush-buffer: Call the method of the same name and try to scan again
- skip-characters count: Skip COUNT characters of the reported sequence and try
to scan again

Be sure to *not* use any of these unconditionally, you’ll end up with an
infinite loop! Instead, apply UNMATCHED-SEQUENCE to the condition in your
handler to investigate and act accordingly; e.g. a hypothetical lexer rule could
require at least five characters to match but the unmatched sequence has only
three so reasonable handling code could look like this:
> (handler-bind ((unmatched-lexing-sequence #’(lambda (condition)
(if (< (length (unmatched-sequence condition)) 5) (invoke-restart ’flush-buffer)
(error condition))))) (function-that-invokes-stream-read-token))

Generic Reader: unmatched-sequence (condition)
Package

graylex.

Methods
Reader Method: unmatched-sequence ((condition unmatched-lexing-sequence))
Source

lexer-input-stream.lisp.

Target Slot

sequence.

Generic Reader: unmatched-sequence-column (condition)
Package

graylex.

Methods
Reader Method: unmatched-sequence-column ((condition unmatched-lexing-sequence))
Source

lexer-input-stream.lisp.

Target Slot

column.

Generic Reader: unmatched-sequence-row (condition)
Package

graylex.

Methods
Reader Method: unmatched-sequence-row ((condition unmatched-lexing-sequence))
Source

lexer-input-stream.lisp.

Target Slot

row.


6.1.2 Standalone methods

Method: close ((stream buffered-input-stream) &key abort)
Source

buffered-input-stream.lisp.

Method: initialize-instance :after ((stream lexer-input-stream) &rest initargs)

initialize-instance :after stream &rest initargs => position

Create the double buffer after initialization.

Source

lexer-input-stream.lisp.

Method: initialize-instance :after ((stream buffered-input-stream) &rest initargs)

initialize-instance :after stream &rest initargs => position

Create the input buffer after initialization and fill it for the first time. Also, (re)set EOF-REACHED to NIL.

Source

buffered-input-stream.lisp.

Method: stream-read-char ((stream lexer-input-stream))

stream-read-char stream => char or :eof

Also save read characters into the double buffer.

Package

sb-gray.

Source

lexer-input-stream.lisp.

Method: stream-read-char ((stream buffered-input-stream))

stream-read-char stream => char or :eof

Return next character from wrapped input buffer or :EOF if the end of the input stream is reached.

Package

sb-gray.

Source

buffered-input-stream.lisp.

Method: stream-read-char :before ((stream buffered-input-stream))

stream-read-char :before stream => position

If reading beyond the internal buffer, replenish it.

Package

sb-gray.

Source

buffered-input-stream.lisp.

Method: stream-read-sequence ((stream buffered-input-stream) seq start end &key)

stream-read-sequence stream seq start end &key => position

Apply READ-SEQUENCE to given arguments and the wrapped input stream. All mechanisms of BUFFERED-INPUT-STREAM apply.
Please see READ-SEQUENCE’s documentation.

Package

trivial-gray-streams.

Source

buffered-input-stream.lisp.

Method: stream-unread-char ((stream buffered-input-stream) char)

stream-unread-char stream char => error

Unreading chars is not supported for BUFFERED-INPUT-STREAMS.

Package

sb-gray.

Source

buffered-input-stream.lisp.


6.1.3 Conditions

Condition: unmatched-lexing-sequence

Condition signaling that no lexer rule matches.

Package

graylex.

Source

lexer-input-stream.lisp.

Direct superclasses

error.

Direct methods
Direct slots
Slot: sequence

Copy of the unmatchable sequence

Package

common-lisp.

Initargs

:sequence

Readers

unmatched-sequence.

Writers

This slot is read-only.

Slot: row

Row part of unmatching position

Initargs

:row

Readers

unmatched-sequence-row.

Writers

This slot is read-only.

Slot: column

Column part of unmatching position

Initargs

:column

Readers

unmatched-sequence-column.

Writers

This slot is read-only.


6.1.4 Classes

Class: buffered-input-stream

Base class for buffered input streams. The default buffer size is 1024 bytes.

Package

graylex.

Source

buffered-input-stream.lisp.

Direct superclasses
  • fundamental-character-input-stream.
  • trivial-gray-stream-mixin.
Direct subclasses

lexer-input-stream.

Direct methods
Direct slots
Slot: stream

Wrapped input stream that is to be buffered

Package

common-lisp.

Initform

(alexandria:required-argument :stream)

Initargs

:stream

Readers

buffered-stream.

Writers

This slot is read-only.

Slot: buffer-size

Size of the input buffer

Initform

1024

Initargs

:buffer-size

Readers

buffered-input-size.

Writers

(setf buffered-input-size).

Slot: buffer-position

Reading position of the wrapped buffer

Initform

0

Readers

buffered-input-position.

Writers

(setf buffered-input-position).

Slot: buffer

Wrapped input buffer

Readers

buffered-input-buffer.

Writers

(setf buffered-input-buffer).

Slot: eof-reached

Whether EOF in stream has been reached

Readers

buffered-eof-p.

Writers

(setf buffered-eof-p).

Class: lexer-input-stream

Lexer input streams provide lexical analysis, tracking of
input row and column and a dynamic second buffer for input tokens longer than the primary BUFFERED-INPUT-STREAM buffer size.

Package

graylex.

Source

lexer-input-stream.lisp.

Direct superclasses

buffered-input-stream.

Direct methods
Direct slots
Slot: rules

List of regexp/keyword conses

Initargs

:rules

Readers

lexer-rules.

Writers

(setf lexer-rules).

Slot: row

Current row in lexer stream

Initform

1

Readers

lexer-row.

Writers

(setf lexer-row).

Slot: column

Current column in lexer stream

Initform

0

Readers

lexer-column.

Writers

(setf lexer-column).

Slot: non-stream-position

Position in unread sequence

Initform

0

Readers

lexer-non-stream-position.

Writers

(setf lexer-non-stream-position).

Slot: double-buffer

Double buffer

Readers

lexer-double-buffer.

Writers

(setf lexer-double-buffer).


6.2 Internals


6.2.1 Ordinary functions

Function: eof-watchdog-filter (string stream)

eof-watchdog-filter string stream => lambda position => position

To be used as a CL-PPCRE filter; evaluates to function that
takes one argument POSITION and signals LEXING-BUFFER-EOF if end of STRING to scan is reached and no more input is to come from STREAM.

Package

graylex.

Source

lexer-input-stream.lisp.


6.2.2 Generic functions

Generic Reader: buffered-eof-p (object)
Generic Writer: (setf buffered-eof-p) (object)
Package

graylex.

Methods
Reader Method: buffered-eof-p ((buffered-input-stream buffered-input-stream))
Writer Method: (setf buffered-eof-p) ((buffered-input-stream buffered-input-stream))

Whether EOF in stream has been reached

Source

buffered-input-stream.lisp.

Target Slot

eof-reached.


6.2.3 Conditions

Condition: lexing-buffer-eof

Signal that a regex tries to scan beyond the internal buffer.

Package

graylex.

Source

lexer-input-stream.lisp.

Direct superclasses

error.


Appendix A Indexes


A.1 Concepts


A.2 Functions

Jump to:   (  
B   C   E   F   G   I   L   M   S   U  
Index Entry  Section

(
(setf buffered-eof-p): Private generic functions
(setf buffered-eof-p): Private generic functions
(setf buffered-input-buffer): Public generic functions
(setf buffered-input-buffer): Public generic functions
(setf buffered-input-position): Public generic functions
(setf buffered-input-position): Public generic functions
(setf buffered-input-size): Public generic functions
(setf buffered-input-size): Public generic functions
(setf lexer-column): Public generic functions
(setf lexer-column): Public generic functions
(setf lexer-double-buffer): Public generic functions
(setf lexer-double-buffer): Public generic functions
(setf lexer-non-stream-position): Public generic functions
(setf lexer-non-stream-position): Public generic functions
(setf lexer-row): Public generic functions
(setf lexer-row): Public generic functions
(setf lexer-rules): Public generic functions
(setf lexer-rules): Public generic functions

B
buffered-eof-p: Private generic functions
buffered-eof-p: Private generic functions
buffered-input-buffer: Public generic functions
buffered-input-buffer: Public generic functions
buffered-input-position: Public generic functions
buffered-input-position: Public generic functions
buffered-input-size: Public generic functions
buffered-input-size: Public generic functions
buffered-stream: Public generic functions
buffered-stream: Public generic functions

C
close: Public standalone methods

E
eof-watchdog-filter: Private ordinary functions

F
fill-buffer: Public generic functions
fill-buffer: Public generic functions
flush-buffer: Public generic functions
flush-buffer: Public generic functions
flush-buffer: Public generic functions
Function, eof-watchdog-filter: Private ordinary functions

G
Generic Function, (setf buffered-eof-p): Private generic functions
Generic Function, (setf buffered-input-buffer): Public generic functions
Generic Function, (setf buffered-input-position): Public generic functions
Generic Function, (setf buffered-input-size): Public generic functions
Generic Function, (setf lexer-column): Public generic functions
Generic Function, (setf lexer-double-buffer): Public generic functions
Generic Function, (setf lexer-non-stream-position): Public generic functions
Generic Function, (setf lexer-row): Public generic functions
Generic Function, (setf lexer-rules): Public generic functions
Generic Function, buffered-eof-p: Private generic functions
Generic Function, buffered-input-buffer: Public generic functions
Generic Function, buffered-input-position: Public generic functions
Generic Function, buffered-input-size: Public generic functions
Generic Function, buffered-stream: Public generic functions
Generic Function, fill-buffer: Public generic functions
Generic Function, flush-buffer: Public generic functions
Generic Function, lexer-column: Public generic functions
Generic Function, lexer-double-buffer: Public generic functions
Generic Function, lexer-non-stream-position: Public generic functions
Generic Function, lexer-row: Public generic functions
Generic Function, lexer-rules: Public generic functions
Generic Function, lexer-unread-sequence: Public generic functions
Generic Function, stream-read-token: Public generic functions
Generic Function, unmatched-sequence: Public generic functions
Generic Function, unmatched-sequence-column: Public generic functions
Generic Function, unmatched-sequence-row: Public generic functions

I
initialize-instance: Public standalone methods
initialize-instance: Public standalone methods

L
lexer-column: Public generic functions
lexer-column: Public generic functions
lexer-double-buffer: Public generic functions
lexer-double-buffer: Public generic functions
lexer-non-stream-position: Public generic functions
lexer-non-stream-position: Public generic functions
lexer-row: Public generic functions
lexer-row: Public generic functions
lexer-rules: Public generic functions
lexer-rules: Public generic functions
lexer-unread-sequence: Public generic functions
lexer-unread-sequence: Public generic functions

M
Method, (setf buffered-eof-p): Private generic functions
Method, (setf buffered-input-buffer): Public generic functions
Method, (setf buffered-input-position): Public generic functions
Method, (setf buffered-input-size): Public generic functions
Method, (setf lexer-column): Public generic functions
Method, (setf lexer-double-buffer): Public generic functions
Method, (setf lexer-non-stream-position): Public generic functions
Method, (setf lexer-row): Public generic functions
Method, (setf lexer-rules): Public generic functions
Method, buffered-eof-p: Private generic functions
Method, buffered-input-buffer: Public generic functions
Method, buffered-input-position: Public generic functions
Method, buffered-input-size: Public generic functions
Method, buffered-stream: Public generic functions
Method, close: Public standalone methods
Method, fill-buffer: Public generic functions
Method, flush-buffer: Public generic functions
Method, flush-buffer: Public generic functions
Method, initialize-instance: Public standalone methods
Method, initialize-instance: Public standalone methods
Method, lexer-column: Public generic functions
Method, lexer-double-buffer: Public generic functions
Method, lexer-non-stream-position: Public generic functions
Method, lexer-row: Public generic functions
Method, lexer-rules: Public generic functions
Method, lexer-unread-sequence: Public generic functions
Method, stream-read-char: Public standalone methods
Method, stream-read-char: Public standalone methods
Method, stream-read-char: Public standalone methods
Method, stream-read-sequence: Public standalone methods
Method, stream-read-token: Public generic functions
Method, stream-read-token: Public generic functions
Method, stream-read-token: Public generic functions
Method, stream-unread-char: Public standalone methods
Method, unmatched-sequence: Public generic functions
Method, unmatched-sequence-column: Public generic functions
Method, unmatched-sequence-row: Public generic functions

S
stream-read-char: Public standalone methods
stream-read-char: Public standalone methods
stream-read-char: Public standalone methods
stream-read-sequence: Public standalone methods
stream-read-token: Public generic functions
stream-read-token: Public generic functions
stream-read-token: Public generic functions
stream-read-token: Public generic functions
stream-unread-char: Public standalone methods

U
unmatched-sequence: Public generic functions
unmatched-sequence: Public generic functions
unmatched-sequence-column: Public generic functions
unmatched-sequence-column: Public generic functions
unmatched-sequence-row: Public generic functions
unmatched-sequence-row: Public generic functions