The langutils Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The langutils Reference Manual

This is the langutils Reference Manual, version 1.0, generated automatically by Declt version 2.4 "Will Decker" on Wed Jun 20 11:12:25 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

LANGUTILS LIBRARY

This file contains a simple guide to the main functions and files of the langutils library. The code is reasonably documented with doc strings and inline comments. Write to the author if there are any questions. Also read docs/LISP2005-langutils.pdf which is a more involved exposition of the implementation and performance issues in the toolkit.

The library provides a heirarchy of major functions and auxiliary functions related to the structured analysis and processing of open text. The major functions working from raw text up are:

We also provide auxiliary functions that operate on strings, tokens or vector-documents. The lisp functions implementing the functionality can be found under the appropriately labled section in the reference below.

Strings

Tokens

Vector-Documents:

Miscellaneous:

INTERFACE REFERENCE

This documents the important functions of the langutils toolkit. Documentation entries are of the form:


function( args )

Input: arg1 - description arg2 - description

Output: description

Notes: discussion of use cases, etc.

Functions are explicitely referenced by putting () around them; variables or parameters have the form of .

TOKENS and TOKENIZATION


tokenize-stream (stream &key (by-sentence nil) (fragment ""))

Input: stream - A standard lisp stream containing the characters to analyze, the stream can be of any length by-sentence - Stop the tokenization process after each processed sentence meaning each validly parsed period, exclamation or question mark. fragment - Provide a fragment from a prior call to tokenize stream at the beginning of the parse stream.

Output: (multiple-values) 1 - parsing success (t) or failure (nil) 2 - the current index into the stream, starts from 0 on every call 3 - a string containing the tokenized data parsed up to 'index' 4 - if parsing was a success, provides a fragment of any unparsed data (primarily in by-sentence mode)

Notes: This function is intended to be called all at once or in batches. For large strings or files it should be called in by-sentence mode in a loop that captures any fragments and passes them to the next call. The function operates by grabbing one character at a time from the stream and writing it into a temporary array. When it reaches a punctuation character, it inserts a whitespace then backs up to the beginning of the current token and checks whether the token should have included the punctuation and fixes up the temporary array. Upon completion of the current parse (end of stream or end of sentence) it


tokenize-string (string)

Input:

Output: (string)

Returns a string which is the result of calling (tokenize-stream) on the stream version of the input string.


tokenize-file (source target &key (if-exists :supersede))

Input:


id-for-token ( token )

Input:

Output: A fixnum providing a unique id for the provided string token.

Notes: Tokens are case sensitive so several 'The', 'the' and 'THE' all map to different tokens but should map to the same entry in the lexicon. The root form of a lexicon word is the lower case representation.


token-for-id ( id )

Input:

Output: The original token string.


tokens-for-id ( ids )

Input:

Output: A list of string representations of the each id


save-token-map ( filename )

Input:

Output: t on success or nil otherwise

Notes: This procedure will default to the filename in default-token-map-file-int which can be set via the asdf-config parameter 'token-map'


load-token-map ( filename )

Input:

Output: t on success or nil otherwise

Notes: This procedure will default to the filename in default-token-map-file-int which can be set via the asdf-config parameter 'token-map'


suspicious-word? ( word )

Input:

Output: A boolean representing whether this word has been labelled as fishy


suspicious-string? ( string )

Input:

Output: A boolean representing whether the word is fishy as determined by parameters set in tokens.lisp (max numbers, total length and other characters in the token). This is used inside id-for-token to keep the hash for suspicious-word? up to date.

POS TAGGING AND OPERATIONS ON TOKENS


tag ( string )

Input:

Output: A tagged string using the format / where the tags are symbols taken from the Penn Treebank 2 tagset. Actual slash characters will show up as '///' meaning a slash word and slash token slash-separated!

Note: This procedure calls the tokenizer to ensure that the input string is properly tokenized in advance.


tag-tokenized ( string )

Input:

Output: A tagged string as in 'tag' above.


vector-tag ( string )

Input:

Output: A CLOS object of type vector-document with the token array initialized to fixnum representations of the word tokens and the tag array initialized with symbols represented the selected tags.


vector-tag-tokenized ( string &key end-tokens )

Input:

Output: A vector-document as in vector-tag

Note: As in tag and tag-tokenized, this interface does not tokenize the input string.


get-lexicon-entry ( word )

Input:

Output: A lexicon-entry structure related to the lexical characteristics of the token

Notes: The lexical-entry can be manipulated with a set of accessor functions: lexicon-entry-tag, lexicon-entry-tags, lexical-entry-id, lexical-entry-roots, lexical-entry-surface-forms, lexical-entry-case-forms, get-lexicon-default-pos. These functions are not all exported from the library package, however.


initial-tag ( token )

Input:

Output: A keyword symbol of the initially guessed tag (:PP :NN, etc)

Notes: Provides an initial guess based purely on lexical features and lexicon information of the provided string token.


read-file-as-tagged-document ( file )

Input:

Output: A vector-document representing the tagged contents of file

Notes: Loads the file into a string then calls vector-tag


read-and-tag-file ( file )

Input:

Output: A string with tag annotations of the contents of file

Notes: Uses tag on the string contents of file


get-lemma ( word &key pos (noun t) porter )

Input:

Output: A string representing the lemma of the word, if found


get-lemma-for-id ( id &key pos (noun t) porter )

Input:

Output: The lemma id


lemmatize ((sequence list/array) &key strip-det pos (noun t) porter last-only )

Input:

Output: Return the lemmatized list of tokens

Notes: The main method for performing lemmatization. Valid on lists and arrays of fixnum values only. Useful for getting the lemmatization of short phrases.


morph-surface-forms ( root &optional pos-class )

Input:

Output: A list of suface ids


morph-surface-forms-text ( root &optional pos-class )

String to string form of the above function


stopword? ( id )

Input:

Output: boolean


concise-stopword? ( id )

Input:

Output: boolean


contains-is? ( ids )

Input:

Output: boolean

Notes: A sometimes useful utility. Searches the list for the token for 'is'


string-stopword?, string-concise-stopword?, string-contains-is? ( string )

The three above functions but accepting string or list of string arguments

CHUNKING


chunk ( text )

Input:

Output: A list of phrases referencing a document created from the text

Note: Runs the tokenizer on the text prior to POS tagging


chunk-tokenized ( text )

Input:

Output: A list of phrases referencing a document created from the text

Note: Does not run the tokenizer on text prior to POS tagging


get-all-chunks ( doc )

Input:

Output: A list of chunks of all the primitive types (verb, adverb, preps and nouns)

Related functions:

Notes:

These two functions could search over sequences of phrases, but usually those are done alone and not on top of a more primitive verb, noun, adverb decomposition. Also note that common preposition idioms (by way of, in front of, etc) are not typically captured properly and would need to be special cased (ie would be VP-sNP-P-NP where sNP is a special type of NP instead of the usual VP-P-NP verb-arg formulation)

CONCEPTS

Concepts are a CLOS abstraction over token sequences that establishes identity over lemmatized phrases. This supports special applications (ConceptNet, LifeNet) at the MIT Media Lab but might be more generally useful.


concept

The 'concept' is a clos object with the following methods


lookup-canonical-concept-instance ( ta )

Input:

Output: A concept instance

EXAMPLE USES

See the file example.lisp. This shows basic use of the tagger, tokenizer, lemmatizer and chunker interfaces.

More examples of use can be generated if enough mail is sent to the author to invoke a guilt-driven re-release of the library with improved documentation.


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 langutils

Author

Ian Eslick

License

BSD

Description

Language utilities

Version

1.0

Dependencies
Source

langutils.asd (file)

Component

src (module)


Next: , Previous: , Up: Top   [Contents][Index]

3 Modules

Modules are listed depth-first from the system components tree.


Previous: , Up: Modules   [Contents][Index]

3.1 langutils/src

Parent

langutils (system)

Location

src/

Components

Next: , Previous: , Up: Top   [Contents][Index]

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

4.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.1 langutils.asd

Location

langutils.asd

Systems

langutils (system)

Packages

langutils.system


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.2 langutils/src/package.lisp

Parent

src (module)

Location

src/package.lisp

Packages

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.3 langutils/src/config.lisp

Dependency

package.lisp (file)

Parent

src (module)

Location

src/config.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.4 langutils/src/tokens.lisp

Dependency

config.lisp (file)

Parent

src (module)

Location

src/tokens.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.5 langutils/src/reference.lisp

Dependency

tokens.lisp (file)

Parent

src (module)

Location

src/reference.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.6 langutils/src/stopwords.lisp

Dependency

reference.lisp (file)

Parent

src (module)

Location

src/stopwords.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.7 langutils/src/my-meta.lisp

Dependency

stopwords.lisp (file)

Parent

src (module)

Location

src/my-meta.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.8 langutils/src/tokenize.lisp

Dependency

my-meta.lisp (file)

Parent

src (module)

Location

src/tokenize.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.9 langutils/src/lexicon.lisp

Dependency

tokenize.lisp (file)

Parent

src (module)

Location

src/lexicon.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.10 langutils/src/lemma.lisp

Dependency

lexicon.lisp (file)

Parent

src (module)

Location

src/lemma.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.11 langutils/src/porter.lisp

Dependency

lemma.lisp (file)

Parent

src (module)

Location

src/porter.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.12 langutils/src/contextual-rule-parser.lisp

Dependency

porter.lisp (file)

Parent

src (module)

Location

src/contextual-rule-parser.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.13 langutils/src/tagger-data.lisp

Dependency

contextual-rule-parser.lisp (file)

Parent

src (module)

Location

src/tagger-data.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.14 langutils/src/tagger.lisp

Dependency

tagger-data.lisp (file)

Parent

src (module)

Location

src/tagger.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.15 langutils/src/chunker-constants.lisp

Dependency

tagger.lisp (file)

Parent

src (module)

Location

src/chunker-constants.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.16 langutils/src/chunker.lisp

Dependency

chunker-constants.lisp (file)

Parent

src (module)

Location

src/chunker.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.17 langutils/src/concept.lisp

Dependency

chunker.lisp (file)

Parent

src (module)

Location

src/concept.lisp

Exported Definitions
Internal Definitions

Previous: , Up: Lisp files   [Contents][Index]

4.1.18 langutils/src/init.lisp

Dependency

concept.lisp (file)

Parent

src (module)

Location

src/init.lisp

Exported Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Packages

Packages are listed by definition order.


Next: , Previous: , Up: Packages   [Contents][Index]

5.1 langutils.system

Source

langutils.asd

Use List

Next: , Previous: , Up: Packages   [Contents][Index]

5.2 langutils

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.3 langutils-tokenize

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Previous: , Up: Packages   [Contents][Index]

5.4 my-meta

Source

package.lisp (file)

Use List

common-lisp

Used By List

langutils-tokenize

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

6.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.1 Macros

Macro: with-list-meta (SOURCE-SYMBOL LIST) &body BODY
Package

my-meta

Source

my-meta.lisp (file)

Macro: with-stream-meta (SOURCE-SYMBOL STREAM) &body BODY
Package

my-meta

Source

my-meta.lisp (file)

Macro: with-string-meta (SOURCE-SYMBOL STRING-BUFFER &key START END) &body BODY
Package

my-meta

Source

my-meta.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.2 Functions

Function: associate-concepts PHRASES

Return the list of phrase/list/token-arrays as pairs with the first element being the original and the second being a canonicalized concept instance

Package

langutils

Source

concept.lisp (file)

Function: chunk TEXT

Returns a phrase-list for the provided text

Package

langutils

Source

chunker.lisp (file)

Function: chunk-tokenized TEXT

Returns a phrase-list for the provided tokenized string

Package

langutils

Source

chunker.lisp (file)

Function: clean-langutils ()
Package

langutils

Source

init.lisp (file)

Function: clean-tagger ()
Package

langutils

Source

tagger.lisp (file)

Function: concise-stopword? ID

Identifies id as a ’concise-stopword’ word.
concise-stopwords are a *very* small list of words. Mainly pronouns and determiners

Package

langutils

Source

stopwords.lisp (file)

Function: contains-is? IDS

Tests list of ids for ’is’ words

Package

langutils

Source

stopwords.lisp (file)

Function: disable-meta-syntax ()
Package

my-meta

Source

my-meta.lisp (file)

Function: enable-meta-syntax ()
Package

my-meta

Source

my-meta.lisp (file)

Function: force-concept C
Package

langutils

Source

concept.lisp (file)

Function: get-lemma WORD &key POS NOUN PORTER

Provides the root word string for the provided word string

Package

langutils

Source

lemma.lisp (file)

Function: get-lemma-for-id ID &key POS NOUN PORTER

Returns a lemma id for the provided word id. pos only returns the root for the provided pos type. noun will stem nouns to the singular form by default and porter determines whether the porter algorithm is used for unknown terms. pos type causes the noun argument to be ignored

Package

langutils

Source

lemma.lisp (file)

Function: get-lexicon-case-forms WORD
Package

langutils

Source

lexicon.lisp (file)

Function: get-lexicon-default-pos WORD
Package

langutils

Source

lexicon.lisp (file)

Function: get-lexicon-entry WORD
Package

langutils

Source

lexicon.lisp (file)

Setf Expander

(setf get-lexicon-entry) (setf expander)

Setf Expander: (setf get-lexicon-entry) WORD ENTRY
Package

langutils

Source

lexicon.lisp (file)

Reader

get-lexicon-entry (function)

Writer

set-lexicon-entry (function)

Function: get-token-count ()

Return the current token counter

Package

langutils

Source

tokens.lisp (file)

Function: head-verb PHRASE &key FILTER-COMMON
Package

langutils

Source

chunker.lisp (file)

Function: head-verbs PHRASES &key FILTER-COMMON
Package

langutils

Source

chunker.lisp (file)

Function: id-for-token TOKEN &optional TRIM

This takes string ’tokens’ and returns a unique id for that character sequence - beware of whitespace, etc.

Package

langutils

Source

tokens.lisp (file)

Function: ids-for-tokens TOKENS
Package

langutils

Source

tokens.lisp (file)

Function: in-pos-class? ELEMENT CLASS
Package

langutils

Source

lemma.lisp (file)

Function: init-langutils ()
Package

langutils

Source

init.lisp (file)

Function: init-tagger &optional LEXICAL-RULE-FILE CONTEXTUAL-RULE-FILE
Package

langutils

Source

tagger.lisp (file)

Function: initial-tag ()

Return an initial tag for a given token string using the langutils lexicon and the tagger lexical rules (via guess-tag)

Package

langutils

Source

tagger.lisp (file)

Function: lexicon-entry-id INSTANCE
Function: (setf lexicon-entry-id) VALUE INSTANCE
Package

langutils

Source

lexicon.lisp (file)

Function: lexicon-entry-roots INSTANCE
Function: (setf lexicon-entry-roots) VALUE INSTANCE
Package

langutils

Source

lexicon.lisp (file)

Function: lexicon-entry-surface-forms INSTANCE
Function: (setf lexicon-entry-surface-forms) VALUE INSTANCE
Package

langutils

Source

lexicon.lisp (file)

Function: lexicon-entry-tag ENTRY
Package

langutils

Source

lexicon.lisp (file)

Function: lexicon-entry-tags INSTANCE
Function: (setf lexicon-entry-tags) VALUE INSTANCE
Package

langutils

Source

lexicon.lisp (file)

Function: make-concept TA
Package

langutils

Source

concept.lisp (file)

Function: make-phrase TEXT-ARRAY TAG-ARRAY &optional TYPE

Take two arrays of test and tags and create a phrase that points at a vdoc created from the two arrays

Package

langutils

Source

reference.lisp (file)

Function: make-phrase-from-sentence TOK-STRING &optional TAG-ARRAY
Package

langutils

Source

reference.lisp (file)

Function: make-phrase-from-vdoc DOC START LEN &optional TYPE
Package

langutils

Source

reference.lisp (file)

Function: make-vector-document TEXT &optional TAGS
Package

langutils

Source

reference.lisp (file)

Function: morph-case-surface-forms ROOT &optional POS-CLASS

All cases of morphological surface forms of the provided root

Package

langutils

Source

lemma.lisp (file)

Function: morph-surface-forms ROOT &optional POS-CLASS

Takes a word or id and returns all surface form ids or all forms of class ’pos-class’ where pos-class is a symbol of langutils::V,A,N

Package

langutils

Source

lemma.lisp (file)

Function: morph-surface-forms-text ROOT &optional POS-CLASS
Package

langutils

Source

lemma.lisp (file)

Function: phrase->concept P &key LEMMATIZED

Create a canonical concept from an arbitrary phrase by removing determiners and lemmatizing verbs.

Package

langutils

Source

concept.lisp (file)

Function: phrase-words PHRASE &optional INDEX
Package

langutils

Source

reference.lisp (file)

Function: read-and-tag-file FILE
Package

langutils

Source

tagger.lisp (file)

Function: read-file-as-tagged-document FILE
Package

langutils

Source

tagger.lisp (file)

Function: reset-langutils ()
Package

langutils

Source

init.lisp (file)

Function: root-noun PHRASE
Package

langutils

Source

chunker.lisp (file)

Function: root-nouns PHRASES
Package

langutils

Source

chunker.lisp (file)

Function: stopword? ID

Identifies id as a ’stopword’

Package

langutils

Source

stopwords.lisp (file)

Function: string->concept S &key LEMMATIZED
Package

langutils

Source

concept.lisp (file)

Function: string->token-array STRING
Package

langutils

Source

tokens.lisp (file)

Function: string-concise-stopword? WORD

Check the word if it is a ’concise-stopword’ word.
concise-stopwords are a *very* small list of words. Mainly pronouns and determiners

Package

langutils

Source

stopwords.lisp (file)

Function: string-contains-is? WORDS

Checks the list for a string containing ’is’

Package

langutils

Source

stopwords.lisp (file)

Function: string-stopword? WORD
Package

langutils

Source

stopwords.lisp (file)

Function: string-tag STRING &optional STREAM

Tokenizes and tags the string returning
a standard tagged string using ’/’ as a separator

Package

langutils

Source

reference.lisp (file)

Function: string-tag-tokenized STRING &optional STREAM
Package

langutils

Source

reference.lisp (file)

Function: suspicious-string? STRING

Determine if the alpha-num and number balance is reasonable for lingustic processing or if non-alpha-nums are present

Package

langutils

Source

tokens.lisp (file)

Function: tag STRING
Package

langutils

Source

tagger.lisp (file)

Function: tag-tokenized STRING
Package

langutils

Source

tagger.lisp (file)

Function: token-array->concept TOKENS &key LEMMATIZED
Package

langutils

Source

concept.lisp (file)

Function: token-for-id ID

Return a string token for a given token id

Package

langutils

Source

tokens.lisp (file)

Function: tokenize-stream STREAM &key BY-SENTENCE FRAGMENT &aux INDEX START CH WS STATUS SENTENCE?

Converts a stream into a string and tokenizes, optionally, one sentence
at a time which is nice for large files. Pretty hairy code: a token processor inside a stream scanner. The stream scanner walks the input stream and tokenizes all punctuation (except periods). After a sequences of non-whitespace has been read, the inline tokenizer looks at the end of the string for mis-tokenized words (can ’ t -> ca n’t)

Package

langutils-tokenize

Source

tokenize.lisp (file)

Function: tokenize-string STRING

Returns a fresh, linguistically tokenized string

Package

langutils-tokenize

Source

tokenize.lisp (file)

Function: tokens-for-ids IDS

Return a list of string tokens for each id in ids

Package

langutils

Source

tokens.lisp (file)

Function: vector-document INPUT
Package

langutils

Source

reference.lisp (file)

Function: vector-tag STRING

Returns a ’document’ which is a class containing a pair of vectors representing the string in the internal token format. Handles arbitrary data.

Package

langutils

Source

tagger.lisp (file)

Function: vector-tag-tokenized ()

Returns a document representing the string using the
internal token dictionary; requires the string to be tokenized. Parses the string into tokens (whitespace separators) then populates the two temp arrays above with token id’s and initial tags. Contextual rules are applied and a new vector document is produced which
is a copy of the enclosed data. This is all done at once so good compilers can open-code the array refs and simplify the calling
of the labels functions.

Package

langutils

Source

tagger.lisp (file)

Function: words->concept SLIST &key LEMMATIZED
Package

langutils

Source

concept.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.3 Generic functions

Generic Function: add-word P INDEX WORD TAG
Package

langutils

Methods
Method: add-word (P altered-phrase) INDEX WORD TAG
Source

reference.lisp (file)

Generic Function: change-word P INDEX NEW-TOKEN &optional NEW-POS
Package

langutils

Methods
Method: change-word (P altered-phrase) INDEX NEW-TOKEN &optional NEW-POS
Source

reference.lisp (file)

Method: change-word (P phrase) INDEX NEW-TOKEN &optional NEW-POS
Source

reference.lisp (file)

Generic Function: concat-concepts &rest CONCEPTS
Package

langutils

Methods
Method: concat-concepts &rest CONCEPTS
Source

concept.lisp (file)

Generic Function: concept->string CNAME
Package

langutils

Methods
Method: concept->string (CNAME concept)
Source

concept.lisp (file)

Generic Function: concept->token-array CNAME
Package

langutils

Methods
Method: concept->token-array (CNAME concept)

Concepts are immutable, don’t change them!

Source

concept.lisp (file)

Generic Function: concept->words CNAME
Package

langutils

Methods
Method: concept->words (CNAME concept)
Source

concept.lisp (file)

Generic Function: concept-contains CSUPER CSUB
Package

langutils

Methods
Method: concept-contains (CSUPER concept) (CSUB concept)
Source

concept.lisp (file)

Generic Function: conceptually-equal PH1 PH2
Package

langutils

Methods
Method: conceptually-equal (CN1 concept) (CN2 concept)
Source

concept.lisp (file)

Method: conceptually-equal (CN concept) (PH phrase)
Source

concept.lisp (file)

Method: conceptually-equal (PH phrase) (CN concept)
Source

concept.lisp (file)

Method: conceptually-equal (PH1 phrase) (PH2 phrase)
Source

concept.lisp (file)

Generic Function: document-annotations OBJECT
Generic Function: (setf document-annotations) NEW-VALUE OBJECT
Package

langutils

Methods
Method: document-annotations (VECTOR-DOCUMENT vector-document)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf document-annotations) NEW-VALUE (VECTOR-DOCUMENT vector-document)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: document-tags OBJECT
Generic Function: (setf document-tags) NEW-VALUE OBJECT
Package

langutils

Methods
Method: document-tags (VECTOR-DOCUMENT vector-document)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf document-tags) NEW-VALUE (VECTOR-DOCUMENT vector-document)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: document-text OBJECT
Generic Function: (setf document-text) NEW-VALUE OBJECT
Package

langutils

Methods
Method: document-text (VECTOR-DOCUMENT vector-document)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf document-text) NEW-VALUE (VECTOR-DOCUMENT vector-document)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: find-phrase P DOC &key MATCH START IGNORE-START IGNORE-END LEMMA CONCEPT-TERMS
Package

langutils

Methods
Method: find-phrase (P phrase) (DOC vector-document) &key MATCH START IGNORE-START IGNORE-END LEMMA CONCEPT-TERMS

Find the specified phrase starting at start, matching text and/or tags according to match. The lemma parameter indicates whether the phrases match under the lemma operator and ignore-start and ignore-end causes the search to not match a region within the document

Source

reference.lisp (file)

Generic Function: find-phrase-intervals P DOC &key MATCH START LEMMA CONCEPT-TERMS
Package

langutils

Methods
Method: find-phrase-intervals (P array) (DOC vector-document) &key MATCH START LEMMA CONCEPT-TERMS &aux RESULTS

Find all phrase intervals in the vector document

Source

reference.lisp (file)

Method: find-phrase-intervals (P phrase) (DOC vector-document) &key MATCH START LEMMA CONCEPT-TERMS &aux RESULTS

Find all phrase intervals in the vector document

Source

reference.lisp (file)

Generic Function: get-adverb-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-adverb-chunks (DOC vector-document) &optional INTERVAL

Return a list of all adverbial phrases

Source

chunker.lisp (file)

Generic Function: get-annotation DOC KEY
Package

langutils

Methods
Method: get-annotation (P phrase) KEY

First returned value is the association value or null if none. The second is true if the key exists, nil otherwise

Source

reference.lisp (file)

Method: get-annotation (DOC vector-document) KEY

First returned value is the association value or null if none. The second is true if the key exists, nil otherwise

Source

reference.lisp (file)

Generic Function: get-event-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-event-chunks (DOC vector-document) &optional INTERVAL

Return vx+nx (simple verb arg) phrase objects

Source

chunker.lisp (file)

Generic Function: get-extended-event-chunks1 DOC &optional INTERVAL
Package

langutils

Methods
Method: get-extended-event-chunks1 (DOC vector-document) &optional INTERVAL

Return vx+nx+pp... objects

Source

chunker.lisp (file)

Generic Function: get-extended-event-chunks2 DOC &optional INTERVAL
Package

langutils

Methods
Method: get-extended-event-chunks2 (DOC vector-document) &optional INTERVAL

Return vx+nx+pp... objects

Source

chunker.lisp (file)

Generic Function: get-imperative-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-imperative-chunks (DOC vector-document) &optional INTERVAL
Source

chunker.lisp (file)

Generic Function: get-nx-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-nx-chunks (DOC vector-document) &optional INTERVAL

Return a list of all nx phrases

Source

chunker.lisp (file)

Generic Function: get-p-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-p-chunks (DOC vector-document) &optional INTERVAL

Return a list of all prepositions as phrases

Source

chunker.lisp (file)

Generic Function: get-pp-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-pp-chunks (DOC vector-document) &optional INTERVAL

Return a list of all prepositions as phrases

Source

chunker.lisp (file)

Generic Function: get-tag DOC OFFSET
Package

langutils

Methods
Method: get-tag (PHRASE altered-phrase) INDEX
Source

reference.lisp (file)

Method: get-tag (P phrase) OFFSET
Source

reference.lisp (file)

Method: get-tag (DOC vector-document) OFFSET
Source

reference.lisp (file)

Generic Function: get-token-id DOC OFFSET
Package

langutils

Methods
Method: get-token-id (PHRASE altered-phrase) INDEX
Source

reference.lisp (file)

Method: get-token-id (P phrase) OFFSET
Source

reference.lisp (file)

Method: get-token-id (DOC vector-document) OFFSET
Source

reference.lisp (file)

Generic Function: get-vx-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-vx-chunks (DOC vector-document) &optional INTERVAL

Return a list of all primitive vx phrases - no arguments

Source

chunker.lisp (file)

Generic Function: lemmatize SEQUENCE &key STRIP-DET POS NOUN PORTER LAST-ONLY
Package

langutils

Methods
Method: lemmatize (SEQUENCE array) &key STRIP-DET POS NOUN PORTER LAST-ONLY
Source

lemma.lisp (file)

Method: lemmatize (SEQUENCE list) &key STRIP-DET POS NOUN PORTER LAST-ONLY

Non-destructive lemmatization of provided sequence

Source

lemma.lisp (file)

Generic Function: lemmatize-phrase P &optional OFFSET
Package

langutils

Methods
Method: lemmatize-phrase (P altered-phrase) &optional OFFSET

Destructive lemmatization of a phrase

Source

reference.lisp (file)

Method: lemmatize-phrase (P phrase) &optional OFFSET

Destructive lemmatization of a phrase

Source

reference.lisp (file)

Generic Function: length-of DOC
Package

langutils

Methods
Method: length-of (DOC vector-document)
Source

reference.lisp (file)

Generic Function: make-alterable-phrase P
Package

langutils

Methods
Method: make-alterable-phrase (P phrase)
Source

reference.lisp (file)

Generic Function: phrase->string P &key WITH-TAGS WITH-INFO NEWLINE
Package

langutils

Methods
Method: phrase->string (P phrase) &key WITH-TAGS WITH-INFO NEWLINE
Source

reference.lisp (file)

Generic Function: phrase->token-array P
Package

langutils

Methods
Method: phrase->token-array (P phrase)

Used in conceptnet to index into a node data structure NOTE: could be faster with direct, declared array copy

Source

reference.lisp (file)

Generic Function: phrase-distance P1 P2
Package

langutils

Methods
Method: phrase-distance (P1 phrase) (P2 phrase)

Distance between the nearest end of two phrases

Source

reference.lisp (file)

Generic Function: phrase-document OBJECT
Generic Function: (setf phrase-document) NEW-VALUE OBJECT
Package

langutils

Methods
Method: phrase-document (P altered-phrase)
Source

reference.lisp (file)

Method: phrase-document (PHRASE phrase)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf phrase-document) NEW-VALUE (PHRASE phrase)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: phrase-end OBJECT
Generic Function: (setf phrase-end) NEW-VALUE OBJECT
Package

langutils

Methods
Method: phrase-end (P altered-phrase)
Source

reference.lisp (file)

Method: phrase-end (PHRASE phrase)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf phrase-end) NEW-VALUE (PHRASE phrase)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: phrase-equal PH1 PH2
Package

langutils

Methods
Method: phrase-equal (PH1 phrase) (PH2 phrase)
Source

reference.lisp (file)

Generic Function: phrase-lemmas PH
Package

langutils

Methods
Method: phrase-lemmas (PH phrase)

Returns the lemmatized phrase represented by the underlying phrase

Source

reference.lisp (file)

Generic Function: phrase-length P
Package

langutils

Methods
Method: phrase-length (P altered-phrase)
Source

reference.lisp (file)

Method: phrase-length (P phrase)
Source

reference.lisp (file)

Generic Function: phrase-overlap PH1 PH2
Package

langutils

Methods
Method: phrase-overlap (PH1 phrase) (PH2 phrase)
Source

reference.lisp (file)

Generic Function: phrase-start OBJECT
Generic Function: (setf phrase-start) NEW-VALUE OBJECT
Package

langutils

Methods
Method: phrase-start (P altered-phrase)
Source

reference.lisp (file)

Method: phrase-start (PHRASE phrase)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf phrase-start) NEW-VALUE (PHRASE phrase)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: phrase-type OBJECT
Generic Function: (setf phrase-type) NEW-VALUE OBJECT
Package

langutils

Methods
Method: phrase-type (PHRASE phrase)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf phrase-type) NEW-VALUE (PHRASE phrase)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: print-phrase P &key STREAM WITH-TAGS WITH-INFO NEWLINE
Package

langutils

Methods
Method: print-phrase (P phrase) &key STREAM WITH-TAGS WITH-INFO NEWLINE
Source

reference.lisp (file)

Generic Function: print-phrase-lemmas PH
Package

langutils

Methods
Method: print-phrase-lemmas (PH phrase)
Source

reference.lisp (file)

Generic Function: print-vector-document DOC &key STREAM WITH-TAGS WITH-NEWLINE
Package

langutils

Methods
Method: print-vector-document (DOC vector-document) &key STREAM WITH-TAGS WITH-NEWLINE
Source

reference.lisp (file)

Generic Function: print-window P WSIZE &key STREAM WITH-TAGS WITH-INFO NEWLINE
Package

langutils

Methods
Method: print-window (P phrase) WSIZE &key STREAM WITH-TAGS WITH-INFO NEWLINE
Source

reference.lisp (file)

Generic Function: read-vector-document FILENAME
Package

langutils

Methods
Method: read-vector-document FILENAME
Source

reference.lisp (file)

Generic Function: read-vector-document-to-string DOC &key WITH-TAGS
Package

langutils

Methods
Method: read-vector-document-to-string (DOC vector-document) &key WITH-TAGS
Source

reference.lisp (file)

Generic Function: remove-word P INDEX
Package

langutils

Methods
Method: remove-word (P altered-phrase) INDEX
Source

reference.lisp (file)

Method: remove-word (P phrase) INDEX
Source

reference.lisp (file)

Generic Function: set-annotation DOC KEY VALUE &key METHOD
Package

langutils

Methods
Method: set-annotation (P phrase) KEY VALUE &key METHOD

Add an annotation to object using method :override, :push, :duplicate-key

Source

reference.lisp (file)

Method: set-annotation (DOC vector-document) KEY VALUE &key METHOD

Add an annotation to object using method :override, :push, :duplicate-key

Source

reference.lisp (file)

Generic Function: suspicious-word? WORD
Package

langutils

Methods
Method: suspicious-word? (WORD fixnum)

Find a suspicious word using it’s token id

Source

tokens.lisp (file)

Generic Function: token-vector OBJECT
Package

langutils

Methods
Method: token-vector (CONCEPT concept)

Stores the representation of the concept as an array of token ids

Source

concept.lisp (file)

Generic Function: unset-annotation DOC KEY
Package

langutils

Methods
Method: unset-annotation (P phrase) KEY
Source

reference.lisp (file)

Method: unset-annotation (DOC vector-document) KEY
Source

reference.lisp (file)

Generic Function: vector-document-string DOC &key WITH-TAGS WITH-NEWLINE
Package

langutils

Methods
Method: vector-document-string (DOC vector-document) &key WITH-TAGS WITH-NEWLINE
Source

reference.lisp (file)

Generic Function: vector-document-words DOC
Package

langutils

Methods
Method: vector-document-words (DOC vector-document)
Source

reference.lisp (file)

Generic Function: write-vector-document DOC FILENAME &key WITH-TAGS IF-EXISTS
Package

langutils

Methods
Method: write-vector-document (DOC vector-document) FILENAME &key WITH-TAGS IF-EXISTS
Source

reference.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.4 Structures

Structure: lexicon-entry ()
Package

langutils

Source

lexicon.lisp (file)

Direct superclasses

structure-object (structure)

Direct slots
Slot: tags
Readers

lexicon-entry-tags (function)

Writers

(setf lexicon-entry-tags) (function)

Slot: id
Readers

lexicon-entry-id (function)

Writers

(setf lexicon-entry-id) (function)

Slot: roots
Readers

lexicon-entry-roots (function)

Writers

(setf lexicon-entry-roots) (function)

Slot: surface-forms
Readers

lexicon-entry-surface-forms (function)

Writers

(setf lexicon-entry-surface-forms) (function)

Slot: case-forms
Readers

lexicon-entry-case-forms (function)

Writers

(setf lexicon-entry-case-forms) (function)


Previous: , Up: Exported definitions   [Contents][Index]

6.1.5 Classes

Class: altered-phrase ()
Package

langutils

Source

reference.lisp (file)

Direct superclasses

phrase (class)

Direct methods
Direct slots
Slot: custom-document
Initargs

:custom-document

Readers

altered-phrase-custom-document (generic function)

Writers

(setf altered-phrase-custom-document) (generic function)

Class: phrase ()
Package

langutils

Source

reference.lisp (file)

Direct superclasses

standard-object (class)

Direct subclasses

altered-phrase (class)

Direct methods
Direct slots
Slot: type
Initargs

:type

Readers

phrase-type (generic function)

Writers

(setf phrase-type) (generic function)

Slot: document
Initargs

:document

Readers

phrase-document (generic function)

Writers

(setf phrase-document) (generic function)

Slot: start
Initargs

:start

Readers

phrase-start (generic function)

Writers

(setf phrase-start) (generic function)

Slot: end
Initargs

:end

Readers

phrase-end (generic function)

Writers

(setf phrase-end) (generic function)

Slot: annotations
Initargs

:annotations

Readers

phrase-annotations (generic function)

Writers

(setf phrase-annotations) (generic function)

Class: vector-document ()
Package

langutils

Source

reference.lisp (file)

Direct superclasses

standard-object (class)

Direct methods
Direct slots
Slot: text
Type

(array fixnum)

Initargs

:text

Readers

document-text (generic function)

Writers

(setf document-text) (generic function)

Slot: tags
Type

(array symbol)

Initargs

:tags

Readers

document-tags (generic function)

Writers

(setf document-tags) (generic function)

Slot: annotations
Type

list

Initargs

:annotations

Readers

document-annotations (generic function)

Writers

(setf document-annotations) (generic function)


Previous: , Up: Definitions   [Contents][Index]

6.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.1 Constants

Constant: *max-token-nums*

The maximum number of numbers allowed in a valid token

Package

langutils

Source

tokens.lisp (file)

Constant: *max-token-others*

The maximum number of non alpha-numeric characters in a valid token

Package

langutils

Source

tokens.lisp (file)

Constant: *whitespace-chars*
Package

langutils

Source

tokens.lisp (file)

Constant: adv-pattern
Package

langutils

Source

chunker-constants.lisp (file)

Constant: noun-pattern
Package

langutils

Source

chunker-constants.lisp (file)

Constant: p-pattern
Package

langutils

Source

chunker-constants.lisp (file)

Constant: verb-pattern
Package

langutils

Source

chunker-constants.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.2 Special variables

Special Variable: *add-to-map-hook*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *auto-init*

Whether to call initialize-langutils when the .fasl is loaded

Package

langutils

Source

config.lisp (file)

Special Variable: *common-verbs*
Package

langutils

Source

chunker.lisp (file)

Special Variable: *concept-store-scratch-array*

Allows us to lookup concepts from arrays without allocating lots of unnecessary data

Package

langutils

Source

concept.lisp (file)

Special Variable: *concept-vhash*
Package

langutils

Source

concept.lisp (file)

Special Variable: *concise-stopwords*
Package

langutils

Source

stopwords.lisp (file)

Special Variable: *config-paths*
Package

langutils

Source

config.lisp (file)

Special Variable: *contextual-rule-args*

The templates for parsing contextual rules and constructing matching templates over word/pos arrays

Package

langutils

Source

contextual-rule-parser.lisp (file)

Special Variable: *default-concise-stopwords-file*

Path to a *very* small list of words. Mainly pronouns and determiners

Package

langutils

Source

config.lisp (file)

Special Variable: *default-contextual-rule-file*

Path to the brill contextual rule file

Package

langutils

Source

config.lisp (file)

Special Variable: *default-lexical-rule-file*

Path to the brill lexical rule file

Package

langutils

Source

config.lisp (file)

Special Variable: *default-lexicon-file*

Path to the lexicon file

Package

langutils

Source

config.lisp (file)

Special Variable: *default-stems-file*

Path to the word stems file

Package

langutils

Source

config.lisp (file)

Special Variable: *default-stopwords-file*

Path to a stopwords file

Package

langutils

Source

config.lisp (file)

Special Variable: *default-token-map-file*

Path to the token map file

Package

langutils

Source

config.lisp (file)

Special Variable: *external-token-map*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *id-for-token-hook*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *id-table*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *is-token*
Package

langutils

Source

stopwords.lisp (file)

Special Variable: *lexicon*
Package

langutils

Source

lexicon.lisp (file)

Special Variable: *meta-readtable*
Package

my-meta

Source

my-meta.lisp (file)

Special Variable: *pos-class-map*
Package

langutils

Source

lemma.lisp (file)

Special Variable: *report-status*

Where to print langutils messages; default to none

Package

langutils

Source

config.lisp (file)

Special Variable: *s-token*
Package

langutils

Source

stopwords.lisp (file)

Special Variable: *saved-readtable*
Package

my-meta

Source

my-meta.lisp (file)

Special Variable: *stopwords*
Package

langutils

Source

stopwords.lisp (file)

Special Variable: *suspicious-words*

Memoize known suspicious words that have been tokenized in this hash

Package

langutils

Source

tokens.lisp (file)

Special Variable: *tagger-bigrams*

Bigram hash (not implemented yet)

Package

langutils

Source

tagger.lisp (file)

Special Variable: *tagger-contextual-rules*

Table to hold the contextual rule closures

Package

langutils

Source

tagger.lisp (file)

Special Variable: *tagger-lexical-rules*

Table to hold the lexical rule closures

Package

langutils

Source

tagger.lisp (file)

Special Variable: *tagger-wordlist*

Wordlist hash (not implemented yet)

Package

langutils

Source

tagger.lisp (file)

Special Variable: *temp-phrase*
Package

langutils

Source

reference.lisp (file)

Special Variable: *test*
Package

langutils

Source

reference.lisp (file)

Special Variable: *token-counter*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *token-counter-hook*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *token-dirty-bit*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *token-for-id-hook*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *token-table*
Package

langutils

Source

tokens.lisp (file)

Special Variable: *tokens-load-file*
Package

langutils

Source

tokens.lisp (file)

Special Variable: known-abbreviations
Package

langutils-tokenize

Source

tokenize.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.3 Macros

Macro: def-contextual-rule-parser NAME &body TEMPLATE-LIST

Given a list of structures, defines a generator named ’name’ that takes
a Brill contextual rule list (list of strings) and generates an applicable closure. The closure accepts an argument list of (tokens tags offset) and will apply the rule and related side effect to the two arrays at the provided offset. Patterns are to be given in the form:
("SURROUNDTAG" (match (0 oldtag) (-1 tag1) (+1 tag2)) =>
(setf oldtag newtag))

Package

langutils

Source

contextual-rule-parser.lisp (file)

Macro: list-match X &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Macro: list-match-type X V &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Macro: stream-match X &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Macro: stream-match-type X V &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Macro: string-match X &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Macro: string-match-type X V &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Macro: with-static-memory-allocation () &rest BODY
Package

langutils

Source

lexicon.lisp (file)

Macro: write-log NAME MSG &rest ARGS
Package

langutils

Source

config.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.4 Functions

Function: *get-determiners* ()
Package

langutils

Source

lemma.lisp (file)

Function: add-basic-entry WORD TAGS &key ROOTS SURFACE

Add a word and it’s probability ordered tags to the lexicon

Package

langutils

Source

lexicon.lisp (file)

Function: add-external-mapping ID-FOR-TOKEN TOKEN-FOR-ID ADD-TO-MAP TOKEN-COUNTER
Package

langutils

Source

tokens.lisp (file)

Function: add-root WORD POS-ROOT-PAIR

Add a root form to word if not exists

Package

langutils

Source

lexicon.lisp (file)

Function: add-root-forms WORD POS-ROOT-PAIRS
Package

langutils

Source

lexicon.lisp (file)

Function: add-roots WORD ROOT-PAIRS

Set the root list (pairs of pos_type/root) for the entry for ’word’

Package

langutils

Source

lexicon.lisp (file)

Function: add-surface-form ROOT SURFACE-FORM

Add a surface form to a root word

Package

langutils

Source

lexicon.lisp (file)

Function: add-to-map-hook TOKEN ID
Package

langutils

Source

tokens.lisp (file)

Function: add-unknown-lexicon-entry WORD GUESSED-TAG
Package

langutils

Source

lexicon.lisp (file)

Function: all-vx+nx-phrases PHRASES

Overly hairy function for finding all vx phrases that are followed by nx. Get event chunks is a better way to do this.

Package

langutils

Source

chunker.lisp (file)

Function: alpha-lowercase CH

Return T if the given character is an alpha character

Package

langutils-tokenize

Source

tokenize.lisp (file)

Function: alpha-misc CH
Package

langutils-tokenize

Source

tokenize.lisp (file)

Function: alpha-uppercase CH
Package

langutils-tokenize

Source

tokenize.lisp (file)

Function: apply-contextual-rules ()
Package

langutils

Source

tagger.lisp (file)

Function: apply-rules DATUM RULE-LIST

Apply rules to the values in values presuming that the returned list is also a list of values that can be passed to the next rule

Package

langutils

Source

tagger-data.lisp (file)

Function: clean-lexicon ()
Package

langutils

Source

lexicon.lisp (file)

Function: clean-stopwords ()
Package

langutils

Source

stopwords.lisp (file)

Function: compile-list L &key SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Function: compileit X &key META-PARSER-TYPE SOURCE-SYMBOL
Package

my-meta

Source

my-meta.lisp (file)

Function: consonantp STR I
Package

langutils

Source

porter.lisp (file)

Function: copy-lexicon-entry INSTANCE
Package

langutils

Source

lexicon.lisp (file)

Function: copy-meta INSTANCE
Package

my-meta

Source

my-meta.lisp (file)

Function: cvc STR LIM
Package

langutils

Source

porter.lisp (file)

Function: default-tag ()

Simple default tagging based on capitalization of token string

Package

langutils

Source

tagger.lisp (file)

Function: doublec STR I
Package

langutils

Source

porter.lisp (file)

Function: duplicate-from ()
Package

langutils

Source

tagger.lisp (file)

Function: ends STR ENDING
Package

langutils

Source

porter.lisp (file)

Function: ensure-common-verbs ()
Package

langutils

Source

chunker.lisp (file)

Function: ensure-concept TOKENS
Package

langutils

Source

concept.lisp (file)

Function: ensure-lexicon-entry WORD &key ROOTS SURFACE
Package

langutils

Source

lexicon.lisp (file)

Function: ensure-token-counts ()

Reset token count if not already set

Package

langutils

Source

tokens.lisp (file)

Function: gen-rule-arg-bindings PATTERN

Generate let bindings for the args referenced in the match pattern

Package

langutils

Source

contextual-rule-parser.lisp (file)

Function: gen-rule-arg-decls PATTERN

Generate type declarations for canonical variables from table entry

Package

langutils

Source

contextual-rule-parser.lisp (file)

Function: gen-rule-closure TEMPLATE

Generate the code for the rule closure as one of the cond forms matching the name of the closure pattern to the rule pattern

Package

langutils

Source

contextual-rule-parser.lisp (file)

Function: gen-rule-closure-decl ()

Optimize the compiled closure through type and optimization declarations

Package

langutils

Source

contextual-rule-parser.lisp (file)

Function: gen-rule-match PATTERN

Generate the conditional code to match this rule

Package

langutils

Source

contextual-rule-parser.lisp (file)

Function: get-bind-entry VAR

Given a canonical variable name, create its let binding and extraction expression from the rule file entry

Package

langutils

Source

contextual-rule-parser.lisp (file)

Function: guess-tag ()

Using rules in rule-table guess the tag of the token ’token’

Package

langutils

Source

tagger-data.lisp (file)

Function: handle-config-entry ENTRY
Package

langutils

Source

config.lisp (file)

Function: id-for-token-hook TOKEN
Package

langutils

Source

tokens.lisp (file)

Function: ids-for-string STRING
Package

langutils

Source

tokens.lisp (file)

Function: init-concise-stopwords &optional PATH
Package

langutils

Source

stopwords.lisp (file)

Function: init-lexicon &optional LEXICON-FILE LEMMA-FILE

Populates the lexicon with ’word tag1 tag2’ structured lines from lexicon-file

Package

langutils

Source

lexicon.lisp (file)

Function: init-stopwords &optional PATH
Package

langutils

Source

stopwords.lisp (file)

Function: init-word-test ()
Package

langutils

Source

stopwords.lisp (file)

Function: initialize-tokens ()
Package

langutils

Source

tokens.lisp (file)

Function: lexicon-entry-case-forms INSTANCE
Function: (setf lexicon-entry-case-forms) VALUE INSTANCE
Package

langutils

Source

lexicon.lisp (file)

Function: lexicon-entry-p OBJECT
Package

langutils

Source

lexicon.lisp (file)

Function: load-contextual-rules RULE-FILE &aux RULES
Package

langutils

Source

tagger-data.lisp (file)

Function: load-lexical-rules RULE-FILE &optional BIGRAM-HASH WORD-HASH &aux RULE-LIST

Return a list of closure implementing the lexical rules in rule-file to tag words not found in the lexicon

Package

langutils

Source

tagger-data.lisp (file)

Function: load-tagger-files LEXICAL-RULES CONTEXTUAL-RULES &key BIGRAMS WORDLIST
Package

langutils

Source

tagger.lisp (file)

Function: m STR LIM
Package

langutils

Source

porter.lisp (file)

Function: make-cases WORD
Package

langutils

Source

lexicon.lisp (file)

Function: make-contextual-rule ()
Package

langutils

Source

tagger-data.lisp (file)

Function: make-lexical-rule LIST LH BH WH

Look through list for rule name

Package

langutils

Source

tagger-data.lisp (file)

Function: make-lexicon-entry &key (TAGS TAGS) (ID ID) (ROOTS ROOTS) (SURFACE-FORMS SURFACE-FORMS) (CASE-FORMS CASE-FORMS)
Package

langutils

Source

lexicon.lisp (file)

Function: make-meta &key (CHAR CHAR) (FORM FORM)
Package

my-meta

Source

my-meta.lisp (file)

Function: meta-char INSTANCE
Function: (setf meta-char) VALUE INSTANCE
Package

my-meta

Source

my-meta.lisp (file)

Function: meta-form INSTANCE
Function: (setf meta-form) VALUE INSTANCE
Package

my-meta

Source

my-meta.lisp (file)

Function: meta-p OBJECT
Package

my-meta

Source

my-meta.lisp (file)

Function: meta-reader S C
Package

my-meta

Source

my-meta.lisp (file)

Function: person-token-offset ARRAY
Package

langutils

Source

reference.lisp (file)

Function: print-token-array TOKENS START STOP &key POS POS-START STREAM WITH-TAGS NEWLINE
Package

langutils

Source

reference.lisp (file)

Function: r STR S SFP
Package

langutils

Source

porter.lisp (file)

Function: read-config ()
Package

langutils

Source

config.lisp (file)

Function: read-file-to-string FILE
Package

langutils

Source

tagger.lisp (file)

Function: relative-pathname PATH
Package

langutils

Source

config.lisp (file)

Function: reset-token-counts ()

Reset all the token datastructures to an initialized but empty state

Package

langutils

Source

tokens.lisp (file)

Function: return-vector-doc ()
Package

langutils

Source

tagger.lisp (file)

Function: select-token TOKEN &key STRIP-DET NOUN POS PORTER LEMMA

Internal per-token function

Package

langutils

Source

lemma.lisp (file)

Function: set-lexicon-entry WORD ENTRY
Package

langutils

Source

lexicon.lisp (file)

Setf Expander

(setf get-lexicon-entry) (setf expander)

Function: setto STR SUFFIX
Package

langutils

Source

porter.lisp (file)

Function: stem STR
Package

langutils

Source

porter.lisp (file)

Function: step1ab STR
Package

langutils

Source

porter.lisp (file)

Function: step1c STR
Package

langutils

Source

porter.lisp (file)

Function: step2 STR
Package

langutils

Source

porter.lisp (file)

Function: step3 STR
Package

langutils

Source

porter.lisp (file)

Function: step4 STR
Package

langutils

Source

porter.lisp (file)

Function: step5 STR
Package

langutils

Source

porter.lisp (file)

Function: symbol-name-equal SRC TARGET &key KEY TEST
Package

my-meta

Source

my-meta.lisp (file)

Function: temp-phrase ()
Package

langutils

Source

reference.lisp (file)

Function: test-concept-equality ()
Package

langutils

Source

concept.lisp (file)

Function: test-phrase TEXT

Prints all the phrases found in the text for simple experimenting

Package

langutils

Source

chunker.lisp (file)

Function: test-vector-tag-tokenized STRING
Package

langutils

Source

tagger.lisp (file)

Function: token-array->words TOKENS
Package

langutils

Source

reference.lisp (file)

Function: token-counter-hook ()
Package

langutils

Source

tokens.lisp (file)

Function: token-for-id-hook ID
Package

langutils

Source

tokens.lisp (file)

Function: tokenize-file2 SOURCE-FILE TARGET-FILE &key IF-EXISTS &aux TOTAL REMAINDER

Tokenizes a pure text file a sentence at a time

Package

langutils-tokenize

Source

tokenize.lisp (file)

Function: vowelinstem STR
Package

langutils

Source

porter.lisp (file)

Function: write-temp ()
Package

langutils

Source

tagger.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.5 Generic functions

Generic Function: altered-phrase-custom-document OBJECT
Generic Function: (setf altered-phrase-custom-document) NEW-VALUE OBJECT
Package

langutils

Methods
Method: altered-phrase-custom-document (ALTERED-PHRASE altered-phrase)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf altered-phrase-custom-document) NEW-VALUE (ALTERED-PHRASE altered-phrase)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: clear-concept-cache ()
Package

langutils

Methods
Method: clear-concept-cache ()
Source

concept.lisp (file)

Generic Function: copy-phrase P &optional ANNOTATIONS
Package

langutils

Methods
Method: copy-phrase (P phrase) &optional ANNOTATIONS
Source

reference.lisp (file)

Generic Function: document-window-as-string DOCUMENT START END
Package

langutils

Methods
Method: document-window-as-string DOCUMENT START END
Source

reference.lisp (file)

Generic Function: get-basic-chunks DOC &optional INTERVAL
Package

langutils

Methods
Method: get-basic-chunks (DOC vector-document) &optional INTERVAL

Returns a list of PHRASEs referencing ’doc’ for all supported primitive phrase types

Source

chunker.lisp (file)

Generic Function: lookup-canonical-concept-instance TA
Package

langutils

Methods
Method: lookup-canonical-concept-instance (LF list)

List of fixnums to lookup a concept instance

Source

concept.lisp (file)

Method: lookup-canonical-concept-instance (TA array)

Take a token array and see if there is already a concept instance

Source

concept.lisp (file)

Generic Function: make-document-from-phrase P
Package

langutils

Methods
Method: make-document-from-phrase (P phrase)

Copy referenced phrase data into it’s own document

Source

reference.lisp (file)

Generic Function: phrase-annotations OBJECT
Generic Function: (setf phrase-annotations) NEW-VALUE OBJECT
Package

langutils

Methods
Method: phrase-annotations (PHRASE phrase)

automatically generated reader method

Source

reference.lisp (file)

Method: (setf phrase-annotations) NEW-VALUE (PHRASE phrase)

automatically generated writer method

Source

reference.lisp (file)

Generic Function: register-new-concept-instance C
Package

langutils

Methods
Method: register-new-concept-instance (C concept)
Source

concept.lisp (file)

Generic Function: vector-doc-as-ids DOC
Package

langutils

Methods
Method: vector-doc-as-ids (DOC vector-document)

Converts the word array to ids with shared structure
for the other elements; keeps the data ’in the family’ so the source or destination documents should be short lived

Source

reference.lisp (file)

Generic Function: vector-doc-as-words DOC
Package

langutils

Methods
Method: vector-doc-as-words (DOC vector-document)
Source

reference.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.6 Conditions

Condition: end-of-sentence ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Direct superclasses

condition (condition)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.7 Structures

Structure: meta ()
Package

my-meta

Source

my-meta.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

print-object (method)

Direct slots
Slot: char
Readers

meta-char (function)

Writers

(setf meta-char) (function)

Slot: form
Readers

meta-form (function)

Writers

(setf meta-form) (function)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.8 Classes

Class: concept ()
Package

langutils

Source

concept.lisp (file)

Direct superclasses

standard-object (class)

Direct methods
Direct slots
Slot: token-vector

Stores the representation of the concept as an array of token ids

Type

(array fixnum)

Initargs

:token-vector

Readers

token-vector (generic function)


Previous: , Up: Internal definitions   [Contents][Index]

6.2.9 Types

Type: alpha ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: alpha-lower ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: alpha-upper ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: alphanum ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: digit ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: non-digit ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: non-digit-or-ws ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: non-punc-or-white ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: non-whitespace ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: punctuation ()
Package

langutils-tokenize

Source

tokenize.lisp (file)

Type: whitespace ()
Package

langutils-tokenize

Source

tokenize.lisp (file)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L   M  
Index Entry  Section

F
File, Lisp, langutils.asd: The langutils<dot>asd file
File, Lisp, langutils/src/chunker-constants.lisp: The langutils/src/chunker-constants<dot>lisp file
File, Lisp, langutils/src/chunker.lisp: The langutils/src/chunker<dot>lisp file
File, Lisp, langutils/src/concept.lisp: The langutils/src/concept<dot>lisp file
File, Lisp, langutils/src/config.lisp: The langutils/src/config<dot>lisp file
File, Lisp, langutils/src/contextual-rule-parser.lisp: The langutils/src/contextual-rule-parser<dot>lisp file
File, Lisp, langutils/src/init.lisp: The langutils/src/init<dot>lisp file
File, Lisp, langutils/src/lemma.lisp: The langutils/src/lemma<dot>lisp file
File, Lisp, langutils/src/lexicon.lisp: The langutils/src/lexicon<dot>lisp file
File, Lisp, langutils/src/my-meta.lisp: The langutils/src/my-meta<dot>lisp file
File, Lisp, langutils/src/package.lisp: The langutils/src/package<dot>lisp file
File, Lisp, langutils/src/porter.lisp: The langutils/src/porter<dot>lisp file
File, Lisp, langutils/src/reference.lisp: The langutils/src/reference<dot>lisp file
File, Lisp, langutils/src/stopwords.lisp: The langutils/src/stopwords<dot>lisp file
File, Lisp, langutils/src/tagger-data.lisp: The langutils/src/tagger-data<dot>lisp file
File, Lisp, langutils/src/tagger.lisp: The langutils/src/tagger<dot>lisp file
File, Lisp, langutils/src/tokenize.lisp: The langutils/src/tokenize<dot>lisp file
File, Lisp, langutils/src/tokens.lisp: The langutils/src/tokens<dot>lisp file

L
langutils.asd: The langutils<dot>asd file
langutils/src: The langutils/src module
langutils/src/chunker-constants.lisp: The langutils/src/chunker-constants<dot>lisp file
langutils/src/chunker.lisp: The langutils/src/chunker<dot>lisp file
langutils/src/concept.lisp: The langutils/src/concept<dot>lisp file
langutils/src/config.lisp: The langutils/src/config<dot>lisp file
langutils/src/contextual-rule-parser.lisp: The langutils/src/contextual-rule-parser<dot>lisp file
langutils/src/init.lisp: The langutils/src/init<dot>lisp file
langutils/src/lemma.lisp: The langutils/src/lemma<dot>lisp file
langutils/src/lexicon.lisp: The langutils/src/lexicon<dot>lisp file
langutils/src/my-meta.lisp: The langutils/src/my-meta<dot>lisp file
langutils/src/package.lisp: The langutils/src/package<dot>lisp file
langutils/src/porter.lisp: The langutils/src/porter<dot>lisp file
langutils/src/reference.lisp: The langutils/src/reference<dot>lisp file
langutils/src/stopwords.lisp: The langutils/src/stopwords<dot>lisp file
langutils/src/tagger-data.lisp: The langutils/src/tagger-data<dot>lisp file
langutils/src/tagger.lisp: The langutils/src/tagger<dot>lisp file
langutils/src/tokenize.lisp: The langutils/src/tokenize<dot>lisp file
langutils/src/tokens.lisp: The langutils/src/tokens<dot>lisp file
Lisp File, langutils.asd: The langutils<dot>asd file
Lisp File, langutils/src/chunker-constants.lisp: The langutils/src/chunker-constants<dot>lisp file
Lisp File, langutils/src/chunker.lisp: The langutils/src/chunker<dot>lisp file
Lisp File, langutils/src/concept.lisp: The langutils/src/concept<dot>lisp file
Lisp File, langutils/src/config.lisp: The langutils/src/config<dot>lisp file
Lisp File, langutils/src/contextual-rule-parser.lisp: The langutils/src/contextual-rule-parser<dot>lisp file
Lisp File, langutils/src/init.lisp: The langutils/src/init<dot>lisp file
Lisp File, langutils/src/lemma.lisp: The langutils/src/lemma<dot>lisp file
Lisp File, langutils/src/lexicon.lisp: The langutils/src/lexicon<dot>lisp file
Lisp File, langutils/src/my-meta.lisp: The langutils/src/my-meta<dot>lisp file
Lisp File, langutils/src/package.lisp: The langutils/src/package<dot>lisp file
Lisp File, langutils/src/porter.lisp: The langutils/src/porter<dot>lisp file
Lisp File, langutils/src/reference.lisp: The langutils/src/reference<dot>lisp file
Lisp File, langutils/src/stopwords.lisp: The langutils/src/stopwords<dot>lisp file
Lisp File, langutils/src/tagger-data.lisp: The langutils/src/tagger-data<dot>lisp file
Lisp File, langutils/src/tagger.lisp: The langutils/src/tagger<dot>lisp file
Lisp File, langutils/src/tokenize.lisp: The langutils/src/tokenize<dot>lisp file
Lisp File, langutils/src/tokens.lisp: The langutils/src/tokens<dot>lisp file

M
Module, langutils/src: The langutils/src module

Jump to:   F   L   M  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (   *  
A   C   D   E   F   G   H   I   L   M   P   R   S   T   U   V   W  
Index Entry  Section

(
(setf altered-phrase-custom-document): Internal generic functions
(setf altered-phrase-custom-document): Internal generic functions
(setf document-annotations): Exported generic functions
(setf document-annotations): Exported generic functions
(setf document-tags): Exported generic functions
(setf document-tags): Exported generic functions
(setf document-text): Exported generic functions
(setf document-text): Exported generic functions
(setf get-lexicon-entry): Exported functions
(setf lexicon-entry-case-forms): Internal functions
(setf lexicon-entry-id): Exported functions
(setf lexicon-entry-roots): Exported functions
(setf lexicon-entry-surface-forms): Exported functions
(setf lexicon-entry-tags): Exported functions
(setf meta-char): Internal functions
(setf meta-form): Internal functions
(setf phrase-annotations): Internal generic functions
(setf phrase-annotations): Internal generic functions
(setf phrase-document): Exported generic functions
(setf phrase-document): Exported generic functions
(setf phrase-end): Exported generic functions
(setf phrase-end): Exported generic functions
(setf phrase-start): Exported generic functions
(setf phrase-start): Exported generic functions
(setf phrase-type): Exported generic functions
(setf phrase-type): Exported generic functions

*
*get-determiners*: Internal functions

A
add-basic-entry: Internal functions
add-external-mapping: Internal functions
add-root: Internal functions
add-root-forms: Internal functions
add-roots: Internal functions
add-surface-form: Internal functions
add-to-map-hook: Internal functions
add-unknown-lexicon-entry: Internal functions
add-word: Exported generic functions
add-word: Exported generic functions
all-vx+nx-phrases: Internal functions
alpha-lowercase: Internal functions
alpha-misc: Internal functions
alpha-uppercase: Internal functions
altered-phrase-custom-document: Internal generic functions
altered-phrase-custom-document: Internal generic functions
apply-contextual-rules: Internal functions
apply-rules: Internal functions
associate-concepts: Exported functions

C
change-word: Exported generic functions
change-word: Exported generic functions
change-word: Exported generic functions
chunk: Exported functions
chunk-tokenized: Exported functions
clean-langutils: Exported functions
clean-lexicon: Internal functions
clean-stopwords: Internal functions
clean-tagger: Exported functions
clear-concept-cache: Internal generic functions
clear-concept-cache: Internal generic functions
compile-list: Internal functions
compileit: Internal functions
concat-concepts: Exported generic functions
concat-concepts: Exported generic functions
concept->string: Exported generic functions
concept->string: Exported generic functions
concept->token-array: Exported generic functions
concept->token-array: Exported generic functions
concept->words: Exported generic functions
concept->words: Exported generic functions
concept-contains: Exported generic functions
concept-contains: Exported generic functions
conceptually-equal: Exported generic functions
conceptually-equal: Exported generic functions
conceptually-equal: Exported generic functions
conceptually-equal: Exported generic functions
conceptually-equal: Exported generic functions
concise-stopword?: Exported functions
consonantp: Internal functions
contains-is?: Exported functions
copy-lexicon-entry: Internal functions
copy-meta: Internal functions
copy-phrase: Internal generic functions
copy-phrase: Internal generic functions
cvc: Internal functions

D
def-contextual-rule-parser: Internal macros
default-tag: Internal functions
disable-meta-syntax: Exported functions
document-annotations: Exported generic functions
document-annotations: Exported generic functions
document-tags: Exported generic functions
document-tags: Exported generic functions
document-text: Exported generic functions
document-text: Exported generic functions
document-window-as-string: Internal generic functions
document-window-as-string: Internal generic functions
doublec: Internal functions
duplicate-from: Internal functions

E
enable-meta-syntax: Exported functions
ends: Internal functions
ensure-common-verbs: Internal functions
ensure-concept: Internal functions
ensure-lexicon-entry: Internal functions
ensure-token-counts: Internal functions

F
find-phrase: Exported generic functions
find-phrase: Exported generic functions
find-phrase-intervals: Exported generic functions
find-phrase-intervals: Exported generic functions
find-phrase-intervals: Exported generic functions
force-concept: Exported functions
Function, (setf lexicon-entry-case-forms): Internal functions
Function, (setf lexicon-entry-id): Exported functions
Function, (setf lexicon-entry-roots): Exported functions
Function, (setf lexicon-entry-surface-forms): Exported functions
Function, (setf lexicon-entry-tags): Exported functions
Function, (setf meta-char): Internal functions
Function, (setf meta-form): Internal functions
Function, *get-determiners*: Internal functions
Function, add-basic-entry: Internal functions
Function, add-external-mapping: Internal functions
Function, add-root: Internal functions
Function, add-root-forms: Internal functions
Function, add-roots: Internal functions
Function, add-surface-form: Internal functions
Function, add-to-map-hook: Internal functions
Function, add-unknown-lexicon-entry: Internal functions
Function, all-vx+nx-phrases: Internal functions
Function, alpha-lowercase: Internal functions
Function, alpha-misc: Internal functions
Function, alpha-uppercase: Internal functions
Function, apply-contextual-rules: Internal functions
Function, apply-rules: Internal functions
Function, associate-concepts: Exported functions
Function, chunk: Exported functions
Function, chunk-tokenized: Exported functions
Function, clean-langutils: Exported functions
Function, clean-lexicon: Internal functions
Function, clean-stopwords: Internal functions
Function, clean-tagger: Exported functions
Function, compile-list: Internal functions
Function, compileit: Internal functions
Function, concise-stopword?: Exported functions
Function, consonantp: Internal functions
Function, contains-is?: Exported functions
Function, copy-lexicon-entry: Internal functions
Function, copy-meta: Internal functions
Function, cvc: Internal functions
Function, default-tag: Internal functions
Function, disable-meta-syntax: Exported functions
Function, doublec: Internal functions
Function, duplicate-from: Internal functions
Function, enable-meta-syntax: Exported functions
Function, ends: Internal functions
Function, ensure-common-verbs: Internal functions
Function, ensure-concept: Internal functions
Function, ensure-lexicon-entry: Internal functions
Function, ensure-token-counts: Internal functions
Function, force-concept: Exported functions
Function, gen-rule-arg-bindings: Internal functions
Function, gen-rule-arg-decls: Internal functions
Function, gen-rule-closure: Internal functions
Function, gen-rule-closure-decl: Internal functions
Function, gen-rule-match: Internal functions
Function, get-bind-entry: Internal functions
Function, get-lemma: Exported functions
Function, get-lemma-for-id: Exported functions
Function, get-lexicon-case-forms: Exported functions
Function, get-lexicon-default-pos: Exported functions
Function, get-lexicon-entry: Exported functions
Function, get-token-count: Exported functions
Function, guess-tag: Internal functions
Function, handle-config-entry: Internal functions
Function, head-verb: Exported functions
Function, head-verbs: Exported functions
Function, id-for-token: Exported functions
Function, id-for-token-hook: Internal functions
Function, ids-for-string: Internal functions
Function, ids-for-tokens: Exported functions
Function, in-pos-class?: Exported functions
Function, init-concise-stopwords: Internal functions
Function, init-langutils: Exported functions
Function, init-lexicon: Internal functions
Function, init-stopwords: Internal functions
Function, init-tagger: Exported functions
Function, init-word-test: Internal functions
Function, initial-tag: Exported functions
Function, initialize-tokens: Internal functions
Function, lexicon-entry-case-forms: Internal functions
Function, lexicon-entry-id: Exported functions
Function, lexicon-entry-p: Internal functions
Function, lexicon-entry-roots: Exported functions
Function, lexicon-entry-surface-forms: Exported functions
Function, lexicon-entry-tag: Exported functions
Function, lexicon-entry-tags: Exported functions
Function, load-contextual-rules: Internal functions
Function, load-lexical-rules: Internal functions
Function, load-tagger-files: Internal functions
Function, m: Internal functions
Function, make-cases: Internal functions
Function, make-concept: Exported functions
Function, make-contextual-rule: Internal functions
Function, make-lexical-rule: Internal functions
Function, make-lexicon-entry: Internal functions
Function, make-meta: Internal functions
Function, make-phrase: Exported functions
Function, make-phrase-from-sentence: Exported functions
Function, make-phrase-from-vdoc: Exported functions
Function, make-vector-document: Exported functions
Function, meta-char: Internal functions
Function, meta-form: Internal functions
Function, meta-p: Internal functions
Function, meta-reader: Internal functions
Function, morph-case-surface-forms: Exported functions
Function, morph-surface-forms: Exported functions
Function, morph-surface-forms-text: Exported functions
Function, person-token-offset: Internal functions
Function, phrase->concept: Exported functions
Function, phrase-words: Exported functions
Function, print-token-array: Internal functions
Function, r: Internal functions
Function, read-and-tag-file: Exported functions
Function, read-config: Internal functions
Function, read-file-as-tagged-document: Exported functions
Function, read-file-to-string: Internal functions
Function, relative-pathname: Internal functions
Function, reset-langutils: Exported functions
Function, reset-token-counts: Internal functions
Function, return-vector-doc: Internal functions
Function, root-noun: Exported functions
Function, root-nouns: Exported functions
Function, select-token: Internal functions
Function, set-lexicon-entry: Internal functions
Function, setto: Internal functions
Function, stem: Internal functions
Function, step1ab: Internal functions
Function, step1c: Internal functions
Function, step2: Internal functions
Function, step3: Internal functions
Function, step4: Internal functions
Function, step5: Internal functions
Function, stopword?: Exported functions
Function, string->concept: Exported functions
Function, string->token-array: Exported functions
Function, string-concise-stopword?: Exported functions
Function, string-contains-is?: Exported functions
Function, string-stopword?: Exported functions
Function, string-tag: Exported functions
Function, string-tag-tokenized: Exported functions
Function, suspicious-string?: Exported functions
Function, symbol-name-equal: Internal functions
Function, tag: Exported functions
Function, tag-tokenized: Exported functions
Function, temp-phrase: Internal functions
Function, test-concept-equality: Internal functions
Function, test-phrase: Internal functions
Function, test-vector-tag-tokenized: Internal functions
Function, token-array->concept: Exported functions
Function, token-array->words: Internal functions
Function, token-counter-hook: Internal functions
Function, token-for-id: Exported functions
Function, token-for-id-hook: Internal functions
Function, tokenize-file2: Internal functions
Function, tokenize-stream: Exported functions
Function, tokenize-string: Exported functions
Function, tokens-for-ids: Exported functions
Function, vector-document: Exported functions
Function, vector-tag: Exported functions
Function, vector-tag-tokenized: Exported functions
Function, vowelinstem: Internal functions
Function, words->concept: Exported functions
Function, write-temp: Internal functions

G
gen-rule-arg-bindings: Internal functions
gen-rule-arg-decls: Internal functions
gen-rule-closure: Internal functions
gen-rule-closure-decl: Internal functions
gen-rule-match: Internal functions
Generic Function, (setf altered-phrase-custom-document): Internal generic functions
Generic Function, (setf document-annotations): Exported generic functions
Generic Function, (setf document-tags): Exported generic functions
Generic Function, (setf document-text): Exported generic functions
Generic Function, (setf phrase-annotations): Internal generic functions
Generic Function, (setf phrase-document): Exported generic functions
Generic Function, (setf phrase-end): Exported generic functions
Generic Function, (setf phrase-start): Exported generic functions
Generic Function, (setf phrase-type): Exported generic functions
Generic Function, add-word: Exported generic functions
Generic Function, altered-phrase-custom-document: Internal generic functions
Generic Function, change-word: Exported generic functions
Generic Function, clear-concept-cache: Internal generic functions
Generic Function, concat-concepts: Exported generic functions
Generic Function, concept->string: Exported generic functions
Generic Function, concept->token-array: Exported generic functions
Generic Function, concept->words: Exported generic functions
Generic Function, concept-contains: Exported generic functions
Generic Function, conceptually-equal: Exported generic functions
Generic Function, copy-phrase: Internal generic functions
Generic Function, document-annotations: Exported generic functions
Generic Function, document-tags: Exported generic functions
Generic Function, document-text: Exported generic functions
Generic Function, document-window-as-string: Internal generic functions
Generic Function, find-phrase: Exported generic functions
Generic Function, find-phrase-intervals: Exported generic functions
Generic Function, get-adverb-chunks: Exported generic functions
Generic Function, get-annotation: Exported generic functions
Generic Function, get-basic-chunks: Internal generic functions
Generic Function, get-event-chunks: Exported generic functions
Generic Function, get-extended-event-chunks1: Exported generic functions
Generic Function, get-extended-event-chunks2: Exported generic functions
Generic Function, get-imperative-chunks: Exported generic functions
Generic Function, get-nx-chunks: Exported generic functions
Generic Function, get-p-chunks: Exported generic functions
Generic Function, get-pp-chunks: Exported generic functions
Generic Function, get-tag: Exported generic functions
Generic Function, get-token-id: Exported generic functions
Generic Function, get-vx-chunks: Exported generic functions
Generic Function, lemmatize: Exported generic functions
Generic Function, lemmatize-phrase: Exported generic functions
Generic Function, length-of: Exported generic functions
Generic Function, lookup-canonical-concept-instance: Internal generic functions
Generic Function, make-alterable-phrase: Exported generic functions
Generic Function, make-document-from-phrase: Internal generic functions
Generic Function, phrase->string: Exported generic functions
Generic Function, phrase->token-array: Exported generic functions
Generic Function, phrase-annotations: Internal generic functions
Generic Function, phrase-distance: Exported generic functions
Generic Function, phrase-document: Exported generic functions
Generic Function, phrase-end: Exported generic functions
Generic Function, phrase-equal: Exported generic functions
Generic Function, phrase-lemmas: Exported generic functions
Generic Function, phrase-length: Exported generic functions
Generic Function, phrase-overlap: Exported generic functions
Generic Function, phrase-start: Exported generic functions
Generic Function, phrase-type: Exported generic functions
Generic Function, print-phrase: Exported generic functions
Generic Function, print-phrase-lemmas: Exported generic functions
Generic Function, print-vector-document: Exported generic functions
Generic Function, print-window: Exported generic functions
Generic Function, read-vector-document: Exported generic functions
Generic Function, read-vector-document-to-string: Exported generic functions
Generic Function, register-new-concept-instance: Internal generic functions
Generic Function, remove-word: Exported generic functions
Generic Function, set-annotation: Exported generic functions
Generic Function, suspicious-word?: Exported generic functions
Generic Function, token-vector: Exported generic functions
Generic Function, unset-annotation: Exported generic functions
Generic Function, vector-doc-as-ids: Internal generic functions
Generic Function, vector-doc-as-words: Internal generic functions
Generic Function, vector-document-string: Exported generic functions
Generic Function, vector-document-words: Exported generic functions
Generic Function, write-vector-document: Exported generic functions
get-adverb-chunks: Exported generic functions
get-adverb-chunks: Exported generic functions
get-annotation: Exported generic functions
get-annotation: Exported generic functions
get-annotation: Exported generic functions
get-basic-chunks: Internal generic functions
get-basic-chunks: Internal generic functions
get-bind-entry: Internal functions
get-event-chunks: Exported generic functions
get-event-chunks: Exported generic functions
get-extended-event-chunks1: Exported generic functions
get-extended-event-chunks1: Exported generic functions
get-extended-event-chunks2: Exported generic functions
get-extended-event-chunks2: Exported generic functions
get-imperative-chunks: Exported generic functions
get-imperative-chunks: Exported generic functions
get-lemma: Exported functions
get-lemma-for-id: Exported functions
get-lexicon-case-forms: Exported functions
get-lexicon-default-pos: Exported functions
get-lexicon-entry: Exported functions
get-nx-chunks: Exported generic functions
get-nx-chunks: Exported generic functions
get-p-chunks: Exported generic functions
get-p-chunks: Exported generic functions
get-pp-chunks: Exported generic functions
get-pp-chunks: Exported generic functions
get-tag: Exported generic functions
get-tag: Exported generic functions
get-tag: Exported generic functions
get-tag: Exported generic functions
get-token-count: Exported functions
get-token-id: Exported generic functions
get-token-id: Exported generic functions
get-token-id: Exported generic functions
get-token-id: Exported generic functions
get-vx-chunks: Exported generic functions
get-vx-chunks: Exported generic functions
guess-tag: Internal functions

H
handle-config-entry: Internal functions
head-verb: Exported functions
head-verbs: Exported functions

I
id-for-token: Exported functions
id-for-token-hook: Internal functions
ids-for-string: Internal functions
ids-for-tokens: Exported functions
in-pos-class?: Exported functions
init-concise-stopwords: Internal functions
init-langutils: Exported functions
init-lexicon: Internal functions
init-stopwords: Internal functions
init-tagger: Exported functions
init-word-test: Internal functions
initial-tag: Exported functions
initialize-tokens: Internal functions

L
lemmatize: Exported generic functions
lemmatize: Exported generic functions
lemmatize: Exported generic functions
lemmatize-phrase: Exported generic functions
lemmatize-phrase: Exported generic functions
lemmatize-phrase: Exported generic functions
length-of: Exported generic functions
length-of: Exported generic functions
lexicon-entry-case-forms: Internal functions
lexicon-entry-id: Exported functions
lexicon-entry-p: Internal functions
lexicon-entry-roots: Exported functions
lexicon-entry-surface-forms: Exported functions
lexicon-entry-tag: Exported functions
lexicon-entry-tags: Exported functions
list-match: Internal macros
list-match-type: Internal macros
load-contextual-rules: Internal functions
load-lexical-rules: Internal functions
load-tagger-files: Internal functions
lookup-canonical-concept-instance: Internal generic functions
lookup-canonical-concept-instance: Internal generic functions
lookup-canonical-concept-instance: Internal generic functions

M
m: Internal functions
Macro, def-contextual-rule-parser: Internal macros
Macro, list-match: Internal macros
Macro, list-match-type: Internal macros
Macro, stream-match: Internal macros
Macro, stream-match-type: Internal macros
Macro, string-match: Internal macros
Macro, string-match-type: Internal macros
Macro, with-list-meta: Exported macros
Macro, with-static-memory-allocation: Internal macros
Macro, with-stream-meta: Exported macros
Macro, with-string-meta: Exported macros
Macro, write-log: Internal macros
make-alterable-phrase: Exported generic functions
make-alterable-phrase: Exported generic functions
make-cases: Internal functions
make-concept: Exported functions
make-contextual-rule: Internal functions
make-document-from-phrase: Internal generic functions
make-document-from-phrase: Internal generic functions
make-lexical-rule: Internal functions
make-lexicon-entry: Internal functions
make-meta: Internal functions
make-phrase: Exported functions
make-phrase-from-sentence: Exported functions
make-phrase-from-vdoc: Exported functions
make-vector-document: Exported functions
meta-char: Internal functions
meta-form: Internal functions
meta-p: Internal functions
meta-reader: Internal functions
Method, (setf altered-phrase-custom-document): Internal generic functions
Method, (setf document-annotations): Exported generic functions
Method, (setf document-tags): Exported generic functions
Method, (setf document-text): Exported generic functions
Method, (setf phrase-annotations): Internal generic functions
Method, (setf phrase-document): Exported generic functions
Method, (setf phrase-end): Exported generic functions
Method, (setf phrase-start): Exported generic functions
Method, (setf phrase-type): Exported generic functions
Method, add-word: Exported generic functions
Method, altered-phrase-custom-document: Internal generic functions
Method, change-word: Exported generic functions
Method, change-word: Exported generic functions
Method, clear-concept-cache: Internal generic functions
Method, concat-concepts: Exported generic functions
Method, concept->string: Exported generic functions
Method, concept->token-array: Exported generic functions
Method, concept->words: Exported generic functions
Method, concept-contains: Exported generic functions
Method, conceptually-equal: Exported generic functions
Method, conceptually-equal: Exported generic functions
Method, conceptually-equal: Exported generic functions
Method, conceptually-equal: Exported generic functions
Method, copy-phrase: Internal generic functions
Method, document-annotations: Exported generic functions
Method, document-tags: Exported generic functions
Method, document-text: Exported generic functions
Method, document-window-as-string: Internal generic functions
Method, find-phrase: Exported generic functions
Method, find-phrase-intervals: Exported generic functions
Method, find-phrase-intervals: Exported generic functions
Method, get-adverb-chunks: Exported generic functions
Method, get-annotation: Exported generic functions
Method, get-annotation: Exported generic functions
Method, get-basic-chunks: Internal generic functions
Method, get-event-chunks: Exported generic functions
Method, get-extended-event-chunks1: Exported generic functions
Method, get-extended-event-chunks2: Exported generic functions
Method, get-imperative-chunks: Exported generic functions
Method, get-nx-chunks: Exported generic functions
Method, get-p-chunks: Exported generic functions
Method, get-pp-chunks: Exported generic functions
Method, get-tag: Exported generic functions
Method, get-tag: Exported generic functions
Method, get-tag: Exported generic functions
Method, get-token-id: Exported generic functions
Method, get-token-id: Exported generic functions
Method, get-token-id: Exported generic functions
Method, get-vx-chunks: Exported generic functions
Method, lemmatize: Exported generic functions
Method, lemmatize: Exported generic functions
Method, lemmatize-phrase: Exported generic functions
Method, lemmatize-phrase: Exported generic functions
Method, length-of: Exported generic functions
Method, lookup-canonical-concept-instance: Internal generic functions
Method, lookup-canonical-concept-instance: Internal generic functions
Method, make-alterable-phrase: Exported generic functions
Method, make-document-from-phrase: Internal generic functions
Method, phrase->string: Exported generic functions
Method, phrase->token-array: Exported generic functions
Method, phrase-annotations: Internal generic functions
Method, phrase-distance: Exported generic functions
Method, phrase-document: Exported generic functions
Method, phrase-document: Exported generic functions
Method, phrase-end: Exported generic functions
Method, phrase-end: Exported generic functions
Method, phrase-equal: Exported generic functions
Method, phrase-lemmas: Exported generic functions
Method, phrase-length: Exported generic functions
Method, phrase-length: Exported generic functions
Method, phrase-overlap: Exported generic functions
Method, phrase-start: Exported generic functions
Method, phrase-start: Exported generic functions
Method, phrase-type: Exported generic functions
Method, print-phrase: Exported generic functions
Method, print-phrase-lemmas: Exported generic functions
Method, print-vector-document: Exported generic functions
Method, print-window: Exported generic functions
Method, read-vector-document: Exported generic functions
Method, read-vector-document-to-string: Exported generic functions
Method, register-new-concept-instance: Internal generic functions
Method, remove-word: Exported generic functions
Method, remove-word: Exported generic functions
Method, set-annotation: Exported generic functions
Method, set-annotation: Exported generic functions
Method, suspicious-word?: Exported generic functions
Method, token-vector: Exported generic functions
Method, unset-annotation: Exported generic functions
Method, unset-annotation: Exported generic functions
Method, vector-doc-as-ids: Internal generic functions
Method, vector-doc-as-words: Internal generic functions
Method, vector-document-string: Exported generic functions
Method, vector-document-words: Exported generic functions
Method, write-vector-document: Exported generic functions
morph-case-surface-forms: Exported functions
morph-surface-forms: Exported functions
morph-surface-forms-text: Exported functions

P
person-token-offset: Internal functions
phrase->concept: Exported functions
phrase->string: Exported generic functions
phrase->string: Exported generic functions
phrase->token-array: Exported generic functions
phrase->token-array: Exported generic functions
phrase-annotations: Internal generic functions
phrase-annotations: Internal generic functions
phrase-distance: Exported generic functions
phrase-distance: Exported generic functions
phrase-document: Exported generic functions
phrase-document: Exported generic functions
phrase-document: Exported generic functions
phrase-end: Exported generic functions
phrase-end: Exported generic functions
phrase-end: Exported generic functions
phrase-equal: Exported generic functions
phrase-equal: Exported generic functions
phrase-lemmas: Exported generic functions
phrase-lemmas: Exported generic functions
phrase-length: Exported generic functions
phrase-length: Exported generic functions
phrase-length: Exported generic functions
phrase-overlap: Exported generic functions
phrase-overlap: Exported generic functions
phrase-start: Exported generic functions
phrase-start: Exported generic functions
phrase-start: Exported generic functions
phrase-type: Exported generic functions
phrase-type: Exported generic functions
phrase-words: Exported functions
print-phrase: Exported generic functions
print-phrase: Exported generic functions
print-phrase-lemmas: Exported generic functions
print-phrase-lemmas: Exported generic functions
print-token-array: Internal functions
print-vector-document: Exported generic functions
print-vector-document: Exported generic functions
print-window: Exported generic functions
print-window: Exported generic functions

R
r: Internal functions
read-and-tag-file: Exported functions
read-config: Internal functions
read-file-as-tagged-document: Exported functions
read-file-to-string: Internal functions
read-vector-document: Exported generic functions
read-vector-document: Exported generic functions
read-vector-document-to-string: Exported generic functions
read-vector-document-to-string: Exported generic functions
register-new-concept-instance: Internal generic functions
register-new-concept-instance: Internal generic functions
relative-pathname: Internal functions
remove-word: Exported generic functions
remove-word: Exported generic functions
remove-word: Exported generic functions
reset-langutils: Exported functions
reset-token-counts: Internal functions
return-vector-doc: Internal functions
root-noun: Exported functions
root-nouns: Exported functions

S
select-token: Internal functions
set-annotation: Exported generic functions
set-annotation: Exported generic functions
set-annotation: Exported generic functions
set-lexicon-entry: Internal functions
Setf Expander, (setf get-lexicon-entry): Exported functions
setto: Internal functions
stem: Internal functions
step1ab: Internal functions
step1c: Internal functions
step2: Internal functions
step3: Internal functions
step4: Internal functions
step5: Internal functions
stopword?: Exported functions
stream-match: Internal macros
stream-match-type: Internal macros
string->concept: Exported functions
string->token-array: Exported functions
string-concise-stopword?: Exported functions
string-contains-is?: Exported functions
string-match: Internal macros
string-match-type: Internal macros
string-stopword?: Exported functions
string-tag: Exported functions
string-tag-tokenized: Exported functions
suspicious-string?: Exported functions
suspicious-word?: Exported generic functions
suspicious-word?: Exported generic functions
symbol-name-equal: Internal functions

T
tag: Exported functions
tag-tokenized: Exported functions
temp-phrase: Internal functions
test-concept-equality: Internal functions
test-phrase: Internal functions
test-vector-tag-tokenized: Internal functions
token-array->concept: Exported functions
token-array->words: Internal functions
token-counter-hook: Internal functions
token-for-id: Exported functions
token-for-id-hook: Internal functions
token-vector: Exported generic functions
token-vector: Exported generic functions
tokenize-file2: Internal functions
tokenize-stream: Exported functions
tokenize-string: Exported functions
tokens-for-ids: Exported functions

U
unset-annotation: Exported generic functions
unset-annotation: Exported generic functions
unset-annotation: Exported generic functions

V
vector-doc-as-ids: Internal generic functions
vector-doc-as-ids: Internal generic functions
vector-doc-as-words: Internal generic functions
vector-doc-as-words: Internal generic functions
vector-document: Exported functions
vector-document-string: Exported generic functions
vector-document-string: Exported generic functions
vector-document-words: Exported generic functions
vector-document-words: Exported generic functions
vector-tag: Exported functions
vector-tag-tokenized: Exported functions
vowelinstem: Internal functions

W
with-list-meta: Exported macros
with-static-memory-allocation: Internal macros
with-stream-meta: Exported macros
with-string-meta: Exported macros
words->concept: Exported functions
write-log: Internal macros
write-temp: Internal functions
write-vector-document: Exported generic functions
write-vector-document: Exported generic functions

Jump to:   (   *  
A   C   D   E   F   G   H   I   L   M   P   R   S   T   U   V   W  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   *  
A   C   D   E   F   I   K   N   P   R   S   T   V  
Index Entry  Section

*
*add-to-map-hook*: Internal special variables
*auto-init*: Internal special variables
*common-verbs*: Internal special variables
*concept-store-scratch-array*: Internal special variables
*concept-vhash*: Internal special variables
*concise-stopwords*: Internal special variables
*config-paths*: Internal special variables
*contextual-rule-args*: Internal special variables
*default-concise-stopwords-file*: Internal special variables
*default-contextual-rule-file*: Internal special variables
*default-lexical-rule-file*: Internal special variables
*default-lexicon-file*: Internal special variables
*default-stems-file*: Internal special variables
*default-stopwords-file*: Internal special variables
*default-token-map-file*: Internal special variables
*external-token-map*: Internal special variables
*id-for-token-hook*: Internal special variables
*id-table*: Internal special variables
*is-token*: Internal special variables
*lexicon*: Internal special variables
*max-token-nums*: Internal constants
*max-token-others*: Internal constants
*meta-readtable*: Internal special variables
*pos-class-map*: Internal special variables
*report-status*: Internal special variables
*s-token*: Internal special variables
*saved-readtable*: Internal special variables
*stopwords*: Internal special variables
*suspicious-words*: Internal special variables
*tagger-bigrams*: Internal special variables
*tagger-contextual-rules*: Internal special variables
*tagger-lexical-rules*: Internal special variables
*tagger-wordlist*: Internal special variables
*temp-phrase*: Internal special variables
*test*: Internal special variables
*token-counter*: Internal special variables
*token-counter-hook*: Internal special variables
*token-dirty-bit*: Internal special variables
*token-for-id-hook*: Internal special variables
*token-table*: Internal special variables
*tokens-load-file*: Internal special variables
*whitespace-chars*: Internal constants

A
adv-pattern: Internal constants
annotations: Exported classes
annotations: Exported classes

C
case-forms: Exported structures
char: Internal structures
Constant, *max-token-nums*: Internal constants
Constant, *max-token-others*: Internal constants
Constant, *whitespace-chars*: Internal constants
Constant, adv-pattern: Internal constants
Constant, noun-pattern: Internal constants
Constant, p-pattern: Internal constants
Constant, verb-pattern: Internal constants
custom-document: Exported classes

D
document: Exported classes

E
end: Exported classes

F
form: Internal structures

I
id: Exported structures

K
known-abbreviations: Internal special variables

N
noun-pattern: Internal constants

P
p-pattern: Internal constants

R
roots: Exported structures

S
Slot, annotations: Exported classes
Slot, annotations: Exported classes
Slot, case-forms: Exported structures
Slot, char: Internal structures
Slot, custom-document: Exported classes
Slot, document: Exported classes
Slot, end: Exported classes
Slot, form: Internal structures
Slot, id: Exported structures
Slot, roots: Exported structures
Slot, start: Exported classes
Slot, surface-forms: Exported structures
Slot, tags: Exported structures
Slot, tags: Exported classes
Slot, text: Exported classes
Slot, token-vector: Internal classes
Slot, type: Exported classes
Special Variable, *add-to-map-hook*: Internal special variables
Special Variable, *auto-init*: Internal special variables
Special Variable, *common-verbs*: Internal special variables
Special Variable, *concept-store-scratch-array*: Internal special variables
Special Variable, *concept-vhash*: Internal special variables
Special Variable, *concise-stopwords*: Internal special variables
Special Variable, *config-paths*: Internal special variables
Special Variable, *contextual-rule-args*: Internal special variables
Special Variable, *default-concise-stopwords-file*: Internal special variables
Special Variable, *default-contextual-rule-file*: Internal special variables
Special Variable, *default-lexical-rule-file*: Internal special variables
Special Variable, *default-lexicon-file*: Internal special variables
Special Variable, *default-stems-file*: Internal special variables
Special Variable, *default-stopwords-file*: Internal special variables
Special Variable, *default-token-map-file*: Internal special variables
Special Variable, *external-token-map*: Internal special variables
Special Variable, *id-for-token-hook*: Internal special variables
Special Variable, *id-table*: Internal special variables
Special Variable, *is-token*: Internal special variables
Special Variable, *lexicon*: Internal special variables
Special Variable, *meta-readtable*: Internal special variables
Special Variable, *pos-class-map*: Internal special variables
Special Variable, *report-status*: Internal special variables
Special Variable, *s-token*: Internal special variables
Special Variable, *saved-readtable*: Internal special variables
Special Variable, *stopwords*: Internal special variables
Special Variable, *suspicious-words*: Internal special variables
Special Variable, *tagger-bigrams*: Internal special variables
Special Variable, *tagger-contextual-rules*: Internal special variables
Special Variable, *tagger-lexical-rules*: Internal special variables
Special Variable, *tagger-wordlist*: Internal special variables
Special Variable, *temp-phrase*: Internal special variables
Special Variable, *test*: Internal special variables
Special Variable, *token-counter*: Internal special variables
Special Variable, *token-counter-hook*: Internal special variables
Special Variable, *token-dirty-bit*: Internal special variables
Special Variable, *token-for-id-hook*: Internal special variables
Special Variable, *token-table*: Internal special variables
Special Variable, *tokens-load-file*: Internal special variables
Special Variable, known-abbreviations: Internal special variables
start: Exported classes
surface-forms: Exported structures

T
tags: Exported structures
tags: Exported classes
text: Exported classes
token-vector: Internal classes
type: Exported classes

V
verb-pattern: Internal constants

Jump to:   *  
A   C   D   E   F   I   K   N   P   R   S   T   V  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   A   C   D   E   L   M   N   P   S   T   V   W  
Index Entry  Section

A
alpha: Internal types
alpha-lower: Internal types
alpha-upper: Internal types
alphanum: Internal types
altered-phrase: Exported classes

C
Class, altered-phrase: Exported classes
Class, concept: Internal classes
Class, phrase: Exported classes
Class, vector-document: Exported classes
concept: Internal classes
Condition, end-of-sentence: Internal conditions

D
digit: Internal types

E
end-of-sentence: Internal conditions

L
langutils: The langutils system
langutils: The langutils package
langutils-tokenize: The langutils-tokenize package
langutils.system: The langutils<dot>system package
lexicon-entry: Exported structures

M
meta: Internal structures
my-meta: The my-meta package

N
non-digit: Internal types
non-digit-or-ws: Internal types
non-punc-or-white: Internal types
non-whitespace: Internal types

P
Package, langutils: The langutils package
Package, langutils-tokenize: The langutils-tokenize package
Package, langutils.system: The langutils<dot>system package
Package, my-meta: The my-meta package
phrase: Exported classes
punctuation: Internal types

S
Structure, lexicon-entry: Exported structures
Structure, meta: Internal structures
System, langutils: The langutils system

T
Type, alpha: Internal types
Type, alpha-lower: Internal types
Type, alpha-upper: Internal types
Type, alphanum: Internal types
Type, digit: Internal types
Type, non-digit: Internal types
Type, non-digit-or-ws: Internal types
Type, non-punc-or-white: Internal types
Type, non-whitespace: Internal types
Type, punctuation: Internal types
Type, whitespace: Internal types

V
vector-document: Exported classes

W
whitespace: Internal types

Jump to:   A   C   D   E   L   M   N   P   S   T   V   W