The langutils Reference Manual

4.1.1 langutils/langutils.asd

Source

langutils.asd.

Parent Component

langutils (system).

ASDF Systems

langutils.

Packages

langutils.system.

---

4.1.2 langutils/src/package.lisp

Source

langutils.asd.

Parent Component

src (module).

Packages

• my-meta.
• langutils.
• langutils-tokenize.

---

4.1.3 langutils/src/config.lisp

Dependency

package.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Internals

• *auto-init* (special variable).
• *config-paths* (special variable).
• *default-concise-stopwords-file* (special variable).
• *default-contextual-rule-file* (special variable).
• *default-lexical-rule-file* (special variable).
• *default-lexicon-file* (special variable).
• *default-stems-file* (special variable).
• *default-stopwords-file* (special variable).
• *default-token-map-file* (special variable).
• *report-status* (special variable).
• handle-config-entry (function).
• read-config (function).
• relative-pathname (function).
• write-log (macro).

---

4.1.4 langutils/src/tokens.lisp

Dependency

config.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• get-token-count (function).
• id-for-token (function).
• ids-for-tokens (function).
• string->token-array (function).
• suspicious-string? (function).
• suspicious-word? (method).
• token-for-id (function).
• tokens-for-ids (function).

Internals

• *add-to-map-hook* (special variable).
• *external-token-map* (special variable).
• *id-for-token-hook* (special variable).
• *id-table* (special variable).
• *max-token-nums* (constant).
• *max-token-others* (constant).
• *suspicious-words* (special variable).
• *token-counter* (special variable).
• *token-counter-hook* (special variable).
• *token-dirty-bit* (special variable).
• *token-for-id-hook* (special variable).
• *token-table* (special variable).
• *tokens-load-file* (special variable).
• *whitespace-chars* (constant).
• add-external-mapping (function).
• add-to-map-hook (function).
• ensure-token-counts (function).
• id-for-token-hook (function).
• ids-for-string (function).
• initialize-tokens (function).
• reset-token-counts (function).
• token-counter-hook (function).
• token-for-id-hook (function).

---

4.1.5 langutils/src/reference.lisp

Dependency

tokens.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• add-word (method).
• altered-phrase (class).
• change-word (method).
• change-word (method).
• document-annotations (reader method).
• (setf document-annotations) (writer method).
• document-tags (reader method).
• (setf document-tags) (writer method).
• document-text (reader method).
• (setf document-text) (writer method).
• find-phrase (method).
• find-phrase-intervals (method).
• find-phrase-intervals (method).
• get-annotation (method).
• get-annotation (method).
• get-tag (method).
• get-tag (method).
• get-tag (method).
• get-token-id (method).
• get-token-id (method).
• get-token-id (method).
• lemmatize-phrase (method).
• lemmatize-phrase (method).
• length-of (method).
• make-alterable-phrase (method).
• make-phrase (function).
• make-phrase-from-sentence (function).
• make-phrase-from-vdoc (function).
• make-vector-document (function).
• phrase (class).
• phrase->string (method).
• phrase->token-array (method).
• phrase-distance (method).
• phrase-document (method).
• phrase-document (reader method).
• (setf phrase-document) (writer method).
• phrase-end (method).
• phrase-end (reader method).
• (setf phrase-end) (writer method).
• phrase-equal (method).
• phrase-lemmas (method).
• phrase-length (method).
• phrase-length (method).
• phrase-overlap (method).
• phrase-start (method).
• phrase-start (reader method).
• (setf phrase-start) (writer method).
• phrase-type (reader method).
• (setf phrase-type) (writer method).
• phrase-words (function).
• print-object (method).
• print-phrase (method).
• print-phrase-lemmas (method).
• print-vector-document (method).
• print-window (method).
• read-vector-document (method).
• read-vector-document-to-string (method).
• remove-word (method).
• remove-word (method).
• set-annotation (method).
• set-annotation (method).
• string-tag (function).
• string-tag-tokenized (function).
• unset-annotation (method).
• unset-annotation (method).
• vector-document (function).
• vector-document (class).
• vector-document-string (method).
• vector-document-words (method).
• write-vector-document (method).

Internals

• *temp-phrase* (special variable).
• *test* (special variable).
• altered-phrase-custom-document (reader method).
• (setf altered-phrase-custom-document) (writer method).
• copy-phrase (method).
• document-window-as-string (method).
• make-document-from-phrase (method).
• person-token-offset (function).
• phrase-annotations (reader method).
• (setf phrase-annotations) (writer method).
• print-token-array (function).
• temp-phrase (function).
• token-array->words (function).
• vector-doc-as-ids (method).
• vector-doc-as-words (method).

---

4.1.6 langutils/src/stopwords.lisp

Dependency

reference.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• concise-stopword? (function).
• contains-is? (function).
• stopword? (function).
• string-concise-stopword? (function).
• string-contains-is? (function).
• string-stopword? (function).

Internals

• *concise-stopwords* (special variable).
• *is-token* (special variable).
• *s-token* (special variable).
• *stopwords* (special variable).
• clean-stopwords (function).
• init-concise-stopwords (function).
• init-stopwords (function).
• init-word-test (function).

---

4.1.7 langutils/src/my-meta.lisp

Dependency

stopwords.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• disable-meta-syntax (function).
• enable-meta-syntax (function).
• print-object (method).
• with-list-meta (macro).
• with-stream-meta (macro).
• with-string-meta (macro).

Internals

• *meta-readtable* (special variable).
• *saved-readtable* (special variable).
• compile-list (function).
• compileit (function).
• copy-meta (function).
• list-match (macro).
• list-match-type (macro).
• make-meta (function).
• meta (structure).
• meta-char (reader).
• (setf meta-char) (writer).
• meta-form (reader).
• (setf meta-form) (writer).
• meta-p (function).
• meta-reader (function).
• stream-match (macro).
• stream-match-type (macro).
• string-match (macro).
• string-match-type (macro).
• symbol-name-equal (function).

---

4.1.8 langutils/src/tokenize.lisp

Dependency

my-meta.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• tokenize-stream (function).
• tokenize-string (function).

Internals

• alpha (type).
• alpha-lower (type).
• alpha-lowercase (function).
• alpha-misc (function).
• alpha-upper (type).
• alpha-uppercase (function).
• alphanum (type).
• digit (type).
• end-of-sentence (condition).
• known-abbreviations (special variable).
• non-digit (type).
• non-digit-or-ws (type).
• non-punc-or-white (type).
• non-whitespace (type).
• punctuation (type).
• tokenize-file2 (function).
• whitespace (type).

---

4.1.9 langutils/src/lexicon.lisp

Dependency

tokenize.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• get-lexicon-case-forms (function).
• get-lexicon-default-pos (function).
• (setf get-lexicon-entry) (setf expander).
• get-lexicon-entry (function).
• lexicon-entry (structure).
• lexicon-entry-id (reader).
• (setf lexicon-entry-id) (writer).
• lexicon-entry-roots (reader).
• (setf lexicon-entry-roots) (writer).
• lexicon-entry-surface-forms (reader).
• (setf lexicon-entry-surface-forms) (writer).
• lexicon-entry-tag (function).
• lexicon-entry-tags (reader).
• (setf lexicon-entry-tags) (writer).

Internals

• *lexicon* (special variable).
• add-basic-entry (function).
• add-root (function).
• add-root-forms (function).
• add-roots (function).
• add-surface-form (function).
• add-unknown-lexicon-entry (function).
• clean-lexicon (function).
• copy-lexicon-entry (function).
• ensure-lexicon-entry (function).
• init-lexicon (function).
• lexicon-entry-case-forms (reader).
• (setf lexicon-entry-case-forms) (writer).
• lexicon-entry-p (function).
• make-cases (function).
• make-lexicon-entry (function).
• set-lexicon-entry (function).
• with-static-memory-allocation (macro).

---

4.1.10 langutils/src/lemma.lisp

Dependency

lexicon.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• get-lemma (function).
• get-lemma-for-id (function).
• in-pos-class? (function).
• lemmatize (method).
• lemmatize (method).
• morph-case-surface-forms (function).
• morph-surface-forms (function).
• morph-surface-forms-text (function).

Internals

• *get-determiners* (function).
• *pos-class-map* (special variable).
• select-token (function).

---

4.1.11 langutils/src/porter.lisp

Dependency

lemma.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Internals

• consonantp (function).
• cvc (function).
• doublec (function).
• ends (function).
• m (function).
• r (function).
• setto (function).
• stem (function).
• step1ab (function).
• step1c (function).
• step2 (function).
• step3 (function).
• step4 (function).
• step5 (function).
• vowelinstem (function).

---

4.1.12 langutils/src/contextual-rule-parser.lisp

Dependency

porter.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Internals

• *contextual-rule-args* (special variable).
• def-contextual-rule-parser (macro).
• gen-rule-arg-bindings (function).
• gen-rule-arg-decls (function).
• gen-rule-closure (function).
• gen-rule-closure-decl (function).
• gen-rule-match (function).
• get-bind-entry (function).

---

4.1.13 langutils/src/tagger-data.lisp

Dependency

contextual-rule-parser.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Internals

• apply-rules (function).
• guess-tag (function).
• load-contextual-rules (function).
• load-lexical-rules (function).
• make-contextual-rule (function).
• make-lexical-rule (function).

---

4.1.14 langutils/src/tagger.lisp

Dependency

tagger-data.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• clean-tagger (function).
• init-tagger (function).
• initial-tag (function).
• read-and-tag-file (function).
• read-file-as-tagged-document (function).
• tag (function).
• tag-tokenized (function).
• vector-tag (function).
• vector-tag-tokenized (function).

Internals

• *tagger-bigrams* (special variable).
• *tagger-contextual-rules* (special variable).
• *tagger-lexical-rules* (special variable).
• *tagger-wordlist* (special variable).
• apply-contextual-rules (function).
• default-tag (function).
• duplicate-from (function).
• load-tagger-files (function).
• read-file-to-string (function).
• return-vector-doc (function).
• test-vector-tag-tokenized (function).
• write-temp (function).

---

4.1.15 langutils/src/chunker-constants.lisp

Dependency

tagger.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Internals

• adv-pattern (constant).
• noun-pattern (constant).
• p-pattern (constant).
• verb-pattern (constant).

---

4.1.16 langutils/src/chunker.lisp

Dependency

chunker-constants.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• chunk (function).
• chunk-tokenized (function).
• get-adverb-chunks (method).
• get-event-chunks (method).
• get-extended-event-chunks1 (method).
• get-extended-event-chunks2 (method).
• get-imperative-chunks (method).
• get-nx-chunks (method).
• get-p-chunks (method).
• get-pp-chunks (method).
• get-vx-chunks (method).
• head-verb (function).
• head-verbs (function).
• root-noun (function).
• root-nouns (function).

Internals

• *common-verbs* (special variable).
• all-vx+nx-phrases (function).
• ensure-common-verbs (function).
• get-basic-chunks (method).
• test-phrase (function).

---

4.1.17 langutils/src/concept.lisp

Dependency

chunker.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• associate-concepts (function).
• concat-concepts (method).
• concept->string (method).
• concept->token-array (method).
• concept->words (method).
• concept-contains (method).
• conceptually-equal (method).
• conceptually-equal (method).
• conceptually-equal (method).
• conceptually-equal (method).
• force-concept (function).
• make-concept (function).
• phrase->concept (function).
• print-object (method).
• string->concept (function).
• token-array->concept (function).
• token-vector (reader method).
• words->concept (function).

Internals

• *concept-store-scratch-array* (special variable).
• *concept-vhash* (special variable).
• clear-concept-cache (method).
• concept (class).
• ensure-concept (function).
• lookup-canonical-concept-instance (method).
• lookup-canonical-concept-instance (method).
• register-new-concept-instance (method).
• test-concept-equality (function).

---

4.1.18 langutils/src/init.lisp

Dependency

concept.lisp (file).

Source

langutils.asd.

Parent Component

src (module).

Public Interface

• clean-langutils (function).
• init-langutils (function).
• reset-langutils (function).

---

6.1.1 Macros

Macro: with-list-meta ((source-symbol list) &body body) ¶

Package

my-meta.

Source

my-meta.lisp.

Macro: with-stream-meta ((source-symbol stream) &body body) ¶

Package

my-meta.

Source

my-meta.lisp.

Macro: with-string-meta ((source-symbol string-buffer &key start end) &body body) ¶

Package

my-meta.

Source

my-meta.lisp.

---

6.1.2 Setf expanders

Setf Expander: (setf get-lexicon-entry) (word) ¶

Package

langutils.

Source

lexicon.lisp.

Reader

get-lexicon-entry (function).

Writer

set-lexicon-entry (function).

---

6.1.3 Ordinary functions

Function: associate-concepts (phrases) ¶

Return the list of phrase/list/token-arrays as pairs with the first element being the original and the second being a canonicalized concept instance

Package

langutils.

Source

concept.lisp.

Function: chunk (text) ¶

Returns a phrase-list for the provided text

Package

langutils.

Source

chunker.lisp.

Function: chunk-tokenized (text) ¶

Returns a phrase-list for the provided tokenized string

Package

langutils.

Source

chunker.lisp.

Function: clean-langutils () ¶

Package

langutils.

Source

init.lisp.

Function: clean-tagger () ¶

Package

langutils.

Source

tagger.lisp.

Function: concise-stopword? (id) ¶

Identifies id as a ’concise-stopword’ word.
concise-stopwords are a *very* small list of words. Mainly pronouns and determiners

Package

langutils.

Source

stopwords.lisp.

Function: contains-is? (ids) ¶

Tests list of ids for ’is’ words

Package

langutils.

Source

stopwords.lisp.

Function: disable-meta-syntax () ¶

Package

my-meta.

Source

my-meta.lisp.

Function: enable-meta-syntax () ¶

Package

my-meta.

Source

my-meta.lisp.

Function: force-concept (c) ¶

Package

langutils.

Source

concept.lisp.

Function: get-lemma (word &key pos noun porter) ¶

Provides the root word string for the provided word string

Package

langutils.

Source

lemma.lisp.

Function: get-lemma-for-id (id &key pos noun porter) ¶

Returns a lemma id for the provided word id. pos only returns the root for the provided pos type. noun will stem nouns to the singular form by default and porter determines whether the porter algorithm is used for unknown terms. pos type causes the noun argument to be ignored

Package

langutils.

Source

lemma.lisp.

Function: get-lexicon-case-forms (word) ¶

Package

langutils.

Source

lexicon.lisp.

Function: get-lexicon-default-pos (word) ¶

Package

langutils.

Source

lexicon.lisp.

Function: get-lexicon-entry (word) ¶

Package

langutils.

Source

lexicon.lisp.

Setf expander for this function

(setf get-lexicon-entry).

Function: get-token-count () ¶

Return the current token counter

Package

langutils.

Source

tokens.lisp.

Function: head-verb (phrase &key filter-common) ¶

Package

langutils.

Source

chunker.lisp.

Function: head-verbs (phrases &key filter-common) ¶

Package

langutils.

Source

chunker.lisp.

Function: id-for-token (token &optional trim) ¶

This takes string ’tokens’ and returns a unique id for that character sequence - beware of whitespace, etc.

Package

langutils.

Source

tokens.lisp.

Function: ids-for-tokens (tokens) ¶

Package

langutils.

Source

tokens.lisp.

Function: in-pos-class? (element class) ¶

Package

langutils.

Source

lemma.lisp.

Function: init-langutils () ¶

Package

langutils.

Source

init.lisp.

Function: init-tagger (&optional lexical-rule-file contextual-rule-file) ¶

Package

langutils.

Source

tagger.lisp.

Function: initial-tag (token) ¶

Return an initial tag for a given token string using the langutils lexicon and the tagger lexical rules (via guess-tag)

Package

langutils.

Source

tagger.lisp.

Reader: lexicon-entry-id (instance) ¶

Writer: (setf lexicon-entry-id) (instance) ¶

Package

langutils.

Source

lexicon.lisp.

Target Slot

id.

Reader: lexicon-entry-roots (instance) ¶

Writer: (setf lexicon-entry-roots) (instance) ¶

Package

langutils.

Source

lexicon.lisp.

Target Slot

roots.

Reader: lexicon-entry-surface-forms (instance) ¶

Writer: (setf lexicon-entry-surface-forms) (instance) ¶

Package

langutils.

Source

lexicon.lisp.

Target Slot

surface-forms.

Function: lexicon-entry-tag (entry) ¶

Package

langutils.

Source

lexicon.lisp.

Reader: lexicon-entry-tags (instance) ¶

Writer: (setf lexicon-entry-tags) (instance) ¶

Package

langutils.

Source

lexicon.lisp.

Target Slot

tags.

Function: make-concept (ta) ¶

Package

langutils.

Source

concept.lisp.

Function: make-phrase (text-array tag-array &optional type) ¶

Take two arrays of test and tags and create a phrase that points at a vdoc created from the two arrays

Package

langutils.

Source

reference.lisp.

Function: make-phrase-from-sentence (tok-string &optional tag-array) ¶

Package

langutils.

Source

reference.lisp.

Function: make-phrase-from-vdoc (doc start len &optional type) ¶

Package

langutils.

Source

reference.lisp.

Function: make-vector-document (text &optional tags) ¶

Package

langutils.

Source

reference.lisp.

Function: morph-case-surface-forms (root &optional pos-class) ¶

All cases of morphological surface forms of the provided root

Package

langutils.

Source

lemma.lisp.

Function: morph-surface-forms (root &optional pos-class) ¶

Takes a word or id and returns all surface form ids or all forms of class ’pos-class’ where pos-class is a symbol of langutils::V,A,N

Package

langutils.

Source

lemma.lisp.

Function: morph-surface-forms-text (root &optional pos-class) ¶

Package

langutils.

Source

lemma.lisp.

Function: phrase->concept (p &key lemmatized) ¶

Create a canonical concept from an arbitrary phrase by removing determiners and lemmatizing verbs.

Package

langutils.

Source

concept.lisp.

Function: phrase-words (phrase &optional index) ¶

Package

langutils.

Source

reference.lisp.

Function: read-and-tag-file (file) ¶

Package

langutils.

Source

tagger.lisp.

Function: read-file-as-tagged-document (file) ¶

Package

langutils.

Source

tagger.lisp.

Function: reset-langutils () ¶

Package

langutils.

Source

init.lisp.

Function: root-noun (phrase) ¶

Package

langutils.

Source

chunker.lisp.

Function: root-nouns (phrases) ¶

Package

langutils.

Source

chunker.lisp.

Function: stopword? (id) ¶

Identifies id as a ’stopword’

Package

langutils.

Source

stopwords.lisp.

Function: string->concept (s &key lemmatized) ¶

Package

langutils.

Source

concept.lisp.

Function: string->token-array (string) ¶

Package

langutils.

Source

tokens.lisp.

Function: string-concise-stopword? (word) ¶

Check the word if it is a ’concise-stopword’ word.
concise-stopwords are a *very* small list of words. Mainly pronouns and determiners

Package

langutils.

Source

stopwords.lisp.

Function: string-contains-is? (words) ¶

Checks the list for a string containing ’is’

Package

langutils.

Source

stopwords.lisp.

Function: string-stopword? (word) ¶

Package

langutils.

Source

stopwords.lisp.

Function: string-tag (string &optional stream) ¶

Tokenizes and tags the string returning
a standard tagged string using ’/’ as a separator

Package

langutils.

Source

reference.lisp.

Function: string-tag-tokenized (string &optional stream) ¶

Package

langutils.

Source

reference.lisp.

Function: suspicious-string? (string) ¶

Determine if the alpha-num and number balance is reasonable for lingustic processing or if non-alpha-nums are present

Package

langutils.

Source

tokens.lisp.

Function: tag (string) ¶

Package

langutils.

Source

tagger.lisp.

Function: tag-tokenized (string) ¶

Package

langutils.

Source

tagger.lisp.

Function: token-array->concept (tokens &key lemmatized) ¶

Package

langutils.

Source

concept.lisp.

Function: token-for-id (id) ¶

Return a string token for a given token id

Package

langutils.

Source

tokens.lisp.

Function: tokenize-stream (stream &key by-sentence fragment) ¶

Converts a stream into a string and tokenizes, optionally, one sentence
at a time which is nice for large files. Pretty hairy code: a token processor inside a stream scanner. The stream scanner walks the input stream and tokenizes all punctuation (except periods). After a sequences of non-whitespace has been read, the inline tokenizer looks at the end of the string for mis-tokenized words (can ’ t -> ca n’t)

Package

langutils-tokenize.

Source

tokenize.lisp.

Function: tokenize-string (string) ¶

Returns a fresh, linguistically tokenized string

Package

langutils-tokenize.

Source

tokenize.lisp.

Function: tokens-for-ids (ids) ¶

Return a list of string tokens for each id in ids

Package

langutils.

Source

tokens.lisp.

Function: vector-document (input) ¶

Package

langutils.

Source

reference.lisp.

Function: vector-tag (string) ¶

Returns a ’document’ which is a class containing a pair of vectors representing the string in the internal token format. Handles arbitrary data.

Package

langutils.

Source

tagger.lisp.

Function: vector-tag-tokenized (string &key end-tokens) ¶

Returns a document representing the string using the
internal token dictionary; requires the string to be tokenized. Parses the string into tokens (whitespace separators) then populates the two temp arrays above with token id’s and initial tags. Contextual rules are applied and a new vector document is produced which
is a copy of the enclosed data. This is all done at once so good compilers can open-code the array refs and simplify the calling
of the labels functions.

Package

langutils.

Source

tagger.lisp.

Function: words->concept (slist &key lemmatized) ¶

Package

langutils.

Source

concept.lisp.

---

6.1.4 Generic functions

Generic Function: add-word (p index word tag) ¶

Package

langutils.

Methods

Method: add-word ((p altered-phrase) index word tag) ¶

Source

reference.lisp.

Generic Function: change-word (p index new-token &optional new-pos) ¶

Package

langutils.

Methods

Method: change-word ((p altered-phrase) index new-token &optional new-pos) ¶

Source

reference.lisp.

Method: change-word ((p phrase) index new-token &optional new-pos) ¶

Source

reference.lisp.

Generic Function: concat-concepts (&rest concepts) ¶

Package

langutils.

Methods

Method: concat-concepts (&rest concepts) ¶

Source

concept.lisp.

Generic Function: concept->string (cname) ¶

Package

langutils.

Methods

Method: concept->string ((cname concept)) ¶

Source

concept.lisp.

Generic Function: concept->token-array (cname) ¶

Package

langutils.

Methods

Method: concept->token-array ((cname concept)) ¶

Concepts are immutable, don’t change them!

Source

concept.lisp.

Generic Function: concept->words (cname) ¶

Package

langutils.

Methods

Method: concept->words ((cname concept)) ¶

Source

concept.lisp.

Generic Function: concept-contains (csuper csub) ¶

Package

langutils.

Methods

Method: concept-contains ((csuper concept) (csub concept)) ¶

Source

concept.lisp.

Generic Function: conceptually-equal (ph1 ph2) ¶

Package

langutils.

Methods

Method: conceptually-equal ((cn1 concept) (cn2 concept)) ¶

Source

concept.lisp.

Method: conceptually-equal ((cn concept) (ph phrase)) ¶

Source

concept.lisp.

Method: conceptually-equal ((ph phrase) (cn concept)) ¶

Source

concept.lisp.

Method: conceptually-equal ((ph1 phrase) (ph2 phrase)) ¶

Source

concept.lisp.

Generic Reader: document-annotations (object) ¶

Package

langutils.

Methods

Reader Method: document-annotations ((vector-document vector-document)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

annotations.

Generic Writer: (setf document-annotations) (object) ¶

Package

langutils.

Methods

Writer Method: (setf document-annotations) ((vector-document vector-document)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

annotations.

Generic Reader: document-tags (object) ¶

Package

langutils.

Methods

Reader Method: document-tags ((vector-document vector-document)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

tags.

Generic Writer: (setf document-tags) (object) ¶

Package

langutils.

Methods

Writer Method: (setf document-tags) ((vector-document vector-document)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

tags.

Generic Reader: document-text (object) ¶

Package

langutils.

Methods

Reader Method: document-text ((vector-document vector-document)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

text.

Generic Writer: (setf document-text) (object) ¶

Package

langutils.

Methods

Writer Method: (setf document-text) ((vector-document vector-document)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

text.

Generic Function: find-phrase (p doc &key match start ignore-start ignore-end lemma concept-terms) ¶

Package

langutils.

Methods

Method: find-phrase ((p phrase) (doc vector-document) &key match start ignore-start ignore-end lemma concept-terms) ¶

Find the specified phrase starting at start, matching text and/or tags according to match. The lemma parameter indicates whether the phrases match under the lemma operator and ignore-start and ignore-end causes the search to not match a region within the document

Source

reference.lisp.

Generic Function: find-phrase-intervals (p doc &key match start lemma concept-terms) ¶

Package

langutils.

Methods

Method: find-phrase-intervals ((p array) (doc vector-document) &key match start lemma concept-terms) ¶

Find all phrase intervals in the vector document

Source

reference.lisp.

Method: find-phrase-intervals ((p phrase) (doc vector-document) &key match start lemma concept-terms) ¶

Find all phrase intervals in the vector document

Source

reference.lisp.

Generic Function: get-adverb-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-adverb-chunks ((doc vector-document) &optional interval) ¶

Return a list of all adverbial phrases

Source

chunker.lisp.

Generic Function: get-annotation (doc key) ¶

Package

langutils.

Methods

Method: get-annotation ((p phrase) key) ¶

First returned value is the association value or null if none. The second is true if the key exists, nil otherwise

Source

reference.lisp.

Method: get-annotation ((doc vector-document) key) ¶

First returned value is the association value or null if none. The second is true if the key exists, nil otherwise

Source

reference.lisp.

Generic Function: get-event-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-event-chunks ((doc vector-document) &optional interval) ¶

Return vx+nx (simple verb arg) phrase objects

Source

chunker.lisp.

Generic Function: get-extended-event-chunks1 (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-extended-event-chunks1 ((doc vector-document) &optional interval) ¶

Return vx+nx+pp... objects

Source

chunker.lisp.

Generic Function: get-extended-event-chunks2 (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-extended-event-chunks2 ((doc vector-document) &optional interval) ¶

Return vx+nx+pp... objects

Source

chunker.lisp.

Generic Function: get-imperative-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-imperative-chunks ((doc vector-document) &optional interval) ¶

Source

chunker.lisp.

Generic Function: get-nx-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-nx-chunks ((doc vector-document) &optional interval) ¶

Return a list of all nx phrases

Source

chunker.lisp.

Generic Function: get-p-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-p-chunks ((doc vector-document) &optional interval) ¶

Return a list of all prepositions as phrases

Source

chunker.lisp.

Generic Function: get-pp-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-pp-chunks ((doc vector-document) &optional interval) ¶

Return a list of all prepositions as phrases

Source

chunker.lisp.

Generic Function: get-tag (doc offset) ¶

Package

langutils.

Methods

Method: get-tag ((phrase altered-phrase) index) ¶

Source

reference.lisp.

Method: get-tag ((p phrase) offset) ¶

Source

reference.lisp.

Method: get-tag ((doc vector-document) offset) ¶

Source

reference.lisp.

Generic Function: get-token-id (doc offset) ¶

Package

langutils.

Methods

Method: get-token-id ((phrase altered-phrase) index) ¶

Source

reference.lisp.

Method: get-token-id ((p phrase) offset) ¶

Source

reference.lisp.

Method: get-token-id ((doc vector-document) offset) ¶

Source

reference.lisp.

Generic Function: get-vx-chunks (doc &optional interval) ¶

Package

langutils.

Methods

Method: get-vx-chunks ((doc vector-document) &optional interval) ¶

Return a list of all primitive vx phrases - no arguments

Source

chunker.lisp.

Generic Function: lemmatize (sequence &key strip-det pos noun porter last-only) ¶

Package

langutils.

Methods

Method: lemmatize ((sequence array) &key strip-det pos noun porter last-only) ¶

Source

lemma.lisp.

Method: lemmatize ((sequence list) &key strip-det pos noun porter last-only) ¶

Non-destructive lemmatization of provided sequence

Source

lemma.lisp.

Generic Function: lemmatize-phrase (p &optional offset) ¶

Package

langutils.

Methods

Method: lemmatize-phrase ((p altered-phrase) &optional offset) ¶

Destructive lemmatization of a phrase

Source

reference.lisp.

Method: lemmatize-phrase ((p phrase) &optional offset) ¶

Destructive lemmatization of a phrase

Source

reference.lisp.

Generic Function: length-of (doc) ¶

Package

langutils.

Methods

Method: length-of ((doc vector-document)) ¶

Source

reference.lisp.

Generic Function: make-alterable-phrase (p) ¶

Package

langutils.

Methods

Method: make-alterable-phrase ((p phrase)) ¶

Source

reference.lisp.

Generic Function: phrase->string (p &key with-tags with-info newline) ¶

Package

langutils.

Methods

Method: phrase->string ((p phrase) &key with-tags with-info newline) ¶

Source

reference.lisp.

Generic Function: phrase->token-array (p) ¶

Package

langutils.

Methods

Method: phrase->token-array ((p phrase)) ¶

Used in conceptnet to index into a node data structure NOTE: could be faster with direct, declared array copy

Source

reference.lisp.

Generic Function: phrase-distance (p1 p2) ¶

Package

langutils.

Methods

Method: phrase-distance ((p1 phrase) (p2 phrase)) ¶

Distance between the nearest end of two phrases

Source

reference.lisp.

Generic Function: phrase-document (object) ¶

Package

langutils.

Methods

Method: phrase-document ((p altered-phrase)) ¶

Source

reference.lisp.

Reader Method: phrase-document ((phrase phrase)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

document.

Generic Writer: (setf phrase-document) (object) ¶

Package

langutils.

Methods

Writer Method: (setf phrase-document) ((phrase phrase)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

document.

Generic Function: phrase-end (object) ¶

Package

langutils.

Methods

Method: phrase-end ((p altered-phrase)) ¶

Source

reference.lisp.

Reader Method: phrase-end ((phrase phrase)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

end.

Generic Writer: (setf phrase-end) (object) ¶

Package

langutils.

Methods

Writer Method: (setf phrase-end) ((phrase phrase)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

end.

Generic Function: phrase-equal (ph1 ph2) ¶

Package

langutils.

Methods

Method: phrase-equal ((ph1 phrase) (ph2 phrase)) ¶

Source

reference.lisp.

Generic Function: phrase-lemmas (ph) ¶

Package

langutils.

Methods

Method: phrase-lemmas ((ph phrase)) ¶

Returns the lemmatized phrase represented by the underlying phrase

Source

reference.lisp.

Generic Function: phrase-length (p) ¶

Package

langutils.

Methods

Method: phrase-length ((p altered-phrase)) ¶

Source

reference.lisp.

Method: phrase-length ((p phrase)) ¶

Source

reference.lisp.

Generic Function: phrase-overlap (ph1 ph2) ¶

Package

langutils.

Methods

Method: phrase-overlap ((ph1 phrase) (ph2 phrase)) ¶

Source

reference.lisp.

Generic Function: phrase-start (object) ¶

Package

langutils.

Methods

Method: phrase-start ((p altered-phrase)) ¶

Source

reference.lisp.

Reader Method: phrase-start ((phrase phrase)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

start.

Generic Writer: (setf phrase-start) (object) ¶

Package

langutils.

Methods

Writer Method: (setf phrase-start) ((phrase phrase)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

start.

Generic Reader: phrase-type (object) ¶

Package

langutils.

Methods

Reader Method: phrase-type ((phrase phrase)) ¶

automatically generated reader method

Source

reference.lisp.

Target Slot

type.

Generic Writer: (setf phrase-type) (object) ¶

Package

langutils.

Methods

Writer Method: (setf phrase-type) ((phrase phrase)) ¶

automatically generated writer method

Source

reference.lisp.

Target Slot

type.

Generic Function: print-phrase (p &key stream with-tags with-info newline) ¶

Package

langutils.

Methods

Method: print-phrase ((p phrase) &key stream with-tags with-info newline) ¶

Source

reference.lisp.

Generic Function: print-phrase-lemmas (ph) ¶

Package

langutils.

Methods

Method: print-phrase-lemmas ((ph phrase)) ¶

Source

reference.lisp.

Generic Function: print-vector-document (doc &key stream with-tags with-newline) ¶

Package

langutils.

Methods

Method: print-vector-document ((doc vector-document) &key stream with-tags with-newline) ¶

Source

reference.lisp.

Generic Function: print-window (p wsize &key stream with-tags with-info newline) ¶

Package

langutils.

Methods

Method: print-window ((p phrase) wsize &key stream with-tags with-info newline) ¶

Source

reference.lisp.

Generic Function: read-vector-document (filename) ¶

Package

langutils.

Methods

Method: read-vector-document (filename) ¶

Source

reference.lisp.

Generic Function: read-vector-document-to-string (doc &key with-tags) ¶

Package

langutils.

Methods

Method: read-vector-document-to-string ((doc vector-document) &key with-tags) ¶

Source

reference.lisp.

Generic Function: remove-word (p index) ¶

Package

langutils.

Methods

Method: remove-word ((p altered-phrase) index) ¶

Source

reference.lisp.

Method: remove-word ((p phrase) index) ¶

Source

reference.lisp.

Generic Function: set-annotation (doc key value &key method) ¶

Package

langutils.

Methods

Method: set-annotation ((p phrase) key value &key method) ¶

Add an annotation to object using method :override, :push, :duplicate-key

Source

reference.lisp.

Method: set-annotation ((doc vector-document) key value &key method) ¶

Add an annotation to object using method :override, :push, :duplicate-key

Source

reference.lisp.

Generic Function: suspicious-word? (word) ¶

Package

langutils.

Methods

Method: suspicious-word? ((word fixnum)) ¶

Find a suspicious word using it’s token id

Source

tokens.lisp.

Generic Reader: token-vector (object) ¶

Package

langutils.

Methods

Reader Method: token-vector ((concept concept)) ¶

Stores the representation of the concept as an array of token ids

Source

concept.lisp.

Target Slot

token-vector.

Generic Function: unset-annotation (doc key) ¶

Package

langutils.

Methods

Method: unset-annotation ((p phrase) key) ¶

Source

reference.lisp.

Method: unset-annotation ((doc vector-document) key) ¶

Source

reference.lisp.

Generic Function: vector-document-string (doc &key with-tags with-newline) ¶

Package

langutils.

Methods

Method: vector-document-string ((doc vector-document) &key with-tags with-newline) ¶

Source

reference.lisp.

Generic Function: vector-document-words (doc) ¶

Package

langutils.

Methods

Method: vector-document-words ((doc vector-document)) ¶

Source

reference.lisp.

Generic Function: write-vector-document (doc filename &key with-tags if-exists) ¶

Package

langutils.

Methods

Method: write-vector-document ((doc vector-document) filename &key with-tags if-exists) ¶

Source

reference.lisp.

---

6.1.5 Standalone methods

Method: print-object ((object meta) stream) ¶

Source

my-meta.lisp.

Method: print-object ((p phrase) stream) ¶

Source

reference.lisp.

Method: print-object ((cn concept) stream) ¶

Source

concept.lisp.

---

6.1.6 Structures

Structure: lexicon-entry ¶

Package

langutils.

Source

lexicon.lisp.

Direct superclasses

structure-object.

Direct slots

Slot: tags ¶

Readers

lexicon-entry-tags.

Writers

(setf lexicon-entry-tags).

Slot: id ¶

Readers

lexicon-entry-id.

Writers

(setf lexicon-entry-id).

Slot: roots ¶

Readers

lexicon-entry-roots.

Writers

(setf lexicon-entry-roots).

Slot: surface-forms ¶

Readers

lexicon-entry-surface-forms.

Writers

(setf lexicon-entry-surface-forms).

Slot: case-forms ¶

Readers

lexicon-entry-case-forms.

Writers

(setf lexicon-entry-case-forms).

---