The lassie Reference Manual

Table of Contents

The lassie Reference Manual

This is the lassie Reference Manual, version 0.0.1, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 13:58:26 2020 GMT+0.

1 Introduction

Lassie is a Common Lisp library for Latent Semantic Indexing by
Ravenpack, Gabor Melis.

It's under the MIT licence. See COPYING.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 lassie

Author

Gabor Melis

License

MIT

Description

Library for Latent Semantic Indexing.

Version

0.0.1

Dependency

fsvd

Source

lassie.asd (file)

Components

• package.lisp (file)
• indexer.lisp (file)
• normalizer.lisp (file)
• mapper.lisp (file)
• assemble.lisp (file)
• lsa.lisp (file)
• lsa-extra.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 lassie.asd

Location

lassie.asd

Systems

lassie (system)

3.1.2 lassie/package.lisp

Parent

lassie (system)

Location

package.lisp

Packages

• lassie
• lassie.indexer
• lassie.assembler

3.1.3 lassie/indexer.lisp

Dependency

package.lisp (file)

Parent

lassie (system)

Location

indexer.lisp

Exported Definitions

• ->index (generic function)
• ->index (method)
• ->index (method)
• ->index (method)
• <-index (generic function)
• <-index (method)
• <-index (method)
• make-counting-indexer (function)
• make-hashing-indexer (function)
• make-random-indexer (function)

Internal Definitions

• %make-hashing-indexer (function)
• %make-random-indexer (function)
• alist->hashing-indexer (function)
• copy-counting-indexer (function)
• copy-hashing-indexer (function)
• copy-random-indexer (function)
• counting-indexer (structure)
• counting-indexer-count (function)
• (setf counting-indexer-count) (function)
• counting-indexer-p (function)
• hashing-indexer (structure)
• hashing-indexer->alist (function)
• hashing-indexer-index->object (function)
• (setf hashing-indexer-index->object) (function)
• hashing-indexer-next-index (function)
• (setf hashing-indexer-next-index) (function)
• hashing-indexer-object->index (function)
• (setf hashing-indexer-object->index) (function)
• hashing-indexer-p (function)
• make-index-vector (function)
• random-indexer (structure)
• random-indexer-length (function)
• (setf random-indexer-length) (function)
• random-indexer-n (function)
• (setf random-indexer-n) (function)
• random-indexer-object->index (function)
• (setf random-indexer-object->index) (function)
• random-indexer-p (function)
• test-hashing-indexer (function)

3.1.4 lassie/normalizer.lisp

Dependency

indexer.lisp (file)

Parent

lassie (system)

Location

normalizer.lisp

Exported Definitions

• column-power-normalizer (class)
• normalize-document-vector (generic function)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-matrix (generic function)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-term-vector (generic function)
• null-normalizer (class)
• pmi-normalizer (class)
• row-centering-normalizer (class)
• sign-normalizer (class)
• tf-idf-normalizer (class)

Internal Definitions

• class-counts (method)
• column-norm (function)
• document-class-fn (method)
• hash-table= (function)
• idfs (method)
• inverse-document-frequency (function)
• make-column-power-normalizer (function)
• make-null-normalizer (function)
• make-tf-idf-normalizer (function)
• n-columns (function)
• n-documents (method)
• n-documents-with-term (function)
• n-rows (function)
• norm (function)
• normalize-column (function)
• normalize-vector (function)
• power (method)
• row-average (function)
• row-averages (method)
• sum-column (function)
• sum-matrix (function)
• sum-row (function)
• sum-vector (function)
• term-counts (method)
• term-counts-per-class (method)
• term-frequency (function)
• term-total (method)
• term-total-per-class (method)
• test-pmi-normalizer (function)
• test-tf-idf-normalizer (function)

3.1.5 lassie/mapper.lisp

Dependency

normalizer.lisp (file)

Parent

lassie (system)

Location

mapper.lisp

Internal Definitions

• compose-mappers (function)
• concatente-mappers (function)
• curry-mapper (function)
• encode-mapper (function)
• make-encoded-term-document-lister (function)
• make-encoded-term-document-mapper (function)
• make-mapper (function)
• map-lines (generic function)
• map-lines (method)
• map-lines (method)
• map-lines (method)
• null-mapper (function)
• test-compose-mappers (function)

3.1.6 lassie/assemble.lisp

Dependency

mapper.lisp (file)

Parent

lassie (system)

Location

assemble.lisp

Exported Definitions

• assemble-co-occurrence-matrix (generic function)
• assemble-co-occurrence-matrix (method)
• assemble-co-occurrence-matrix (method)
• assemble-document-vector (generic function)
• assemble-document-vector (method)
• assemble-document-vector (method)
• assemble-term-vector (generic function)
• assemble-term-vector (method)
• lsa-assembler (class)
• ri-term-assembler (class)

Internal Definitions

• assemble-occurence-vector (function)
• incf-and-maybe-grow (function)
• make-lsa-assembler (function)
• make-ri-term-assembler (function)
• n-documents (method)
• n-terms (method)

3.1.7 lassie/lsa.lisp

Dependency

assemble.lisp (file)

Parent

lassie (system)

Location

lsa.lisp

Exported Definitions

• construct-document-vector (function)
• construct-term-vector (function)
• document->vector (function)
• document-features (function)
• document-indexer (method)
• document-mapper (method)
• document-vector-features (function)
• load-lsa (function)
• lsa (function)
• lsa (class)
• normalizer (method)
• save-lsa (function)
• svd (method)
• term->vector (function)
• term-features (function)
• term-indexer (method)
• term-mapper (method)
• term-vector-features (function)

Internal Definitions

• assembler (method)
• construct-lsa-vector (function)
• coordinate (function)
• extract-lsa-features (function)
• extract-svd-features (function)
• inner* (function)
• single-float-vector (type)

3.1.8 lassie/lsa-extra.lisp

Dependency

lsa.lisp (file)

Parent

lassie (system)

Location

lsa-extra.lisp

Exported Definitions

• cosine-similarity (function)
• most-similar-documents (function)

Internal Definitions

insert-into-sorted-vector (function)

4 Packages

Packages are listed by definition order.

4.1 lassie

The core functionality of Lassie.

Source

package.lisp (file)

Use List

• lassie.assembler
• lassie.indexer
• common-lisp

Exported Definitions

• construct-document-vector (function)
• construct-term-vector (function)
• cosine-similarity (function)
• document->vector (function)
• document-features (function)
• document-indexer (generic function)
• document-indexer (method)
• document-mapper (generic function)
• document-mapper (method)
• document-vector-features (function)
• load-lsa (function)
• lsa (function)
• lsa (class)
• most-similar-documents (function)
• normalizer (generic function)
• normalizer (method)
• save-lsa (function)
• svd (generic function)
• svd (method)
• term->vector (function)
• term-features (function)
• term-indexer (generic function)
• term-indexer (method)
• term-mapper (generic function)
• term-mapper (method)
• term-vector-features (function)

Internal Definitions

• assembler (generic function)
• assembler (method)
• compose-mappers (function)
• concatente-mappers (function)
• construct-lsa-vector (function)
• coordinate (function)
• curry-mapper (function)
• encode-mapper (function)
• extract-lsa-features (function)
• extract-svd-features (function)
• inner* (function)
• insert-into-sorted-vector (function)
• make-encoded-term-document-lister (function)
• make-encoded-term-document-mapper (function)
• make-mapper (function)
• map-lines (generic function)
• map-lines (method)
• map-lines (method)
• map-lines (method)
• null-mapper (function)
• single-float-vector (type)
• test-compose-mappers (function)

4.2 lassie.indexer

Indexers provide a - sometimes reversible - mapping
from objects and indices. The word ‘index’ is used here in a very general sense, random indexers, for instance, map to a set of indices. Within Lassie they are used in conjunction with assemblers that know how to change the co-occurence matrix when encountering given an index. They can be printed and read readably.

Source

package.lisp (file)

Use List

common-lisp

Used By List

lassie

Exported Definitions

• ->index (generic function)
• ->index (method)
• ->index (method)
• ->index (method)
• <-index (generic function)
• <-index (method)
• <-index (method)
• make-counting-indexer (function)
• make-hashing-indexer (function)
• make-random-indexer (function)

Internal Definitions

• %make-hashing-indexer (function)
• %make-random-indexer (function)
• alist->hashing-indexer (function)
• copy-counting-indexer (function)
• copy-hashing-indexer (function)
• copy-random-indexer (function)
• counting-indexer (structure)
• counting-indexer-count (function)
• (setf counting-indexer-count) (function)
• counting-indexer-p (function)
• hashing-indexer (structure)
• hashing-indexer->alist (function)
• hashing-indexer-index->object (function)
• (setf hashing-indexer-index->object) (function)
• hashing-indexer-next-index (function)
• (setf hashing-indexer-next-index) (function)
• hashing-indexer-object->index (function)
• (setf hashing-indexer-object->index) (function)
• hashing-indexer-p (function)
• make-index-vector (function)
• random-indexer (structure)
• random-indexer-length (function)
• (setf random-indexer-length) (function)
• random-indexer-n (function)
• (setf random-indexer-n) (function)
• random-indexer-object->index (function)
• (setf random-indexer-object->index) (function)
• random-indexer-p (function)
• test-hashing-indexer (function)

4.3 lassie.assembler

Different assemblers and normalizers that one plug
into Lassie. Assemblers to construct a co-occurence matrix or document vector from a corpus, and normalizers to perform post processing on the assembled data. Normalizers can be printed and read readably.

Source

package.lisp (file)

Use List

common-lisp

Used By List

lassie

Exported Definitions

• assemble-co-occurrence-matrix (generic function)
• assemble-co-occurrence-matrix (method)
• assemble-co-occurrence-matrix (method)
• assemble-document-vector (generic function)
• assemble-document-vector (method)
• assemble-document-vector (method)
• assemble-term-vector (generic function)
• assemble-term-vector (method)
• column-power-normalizer (class)
• lsa-assembler (class)
• normalize-document-vector (generic function)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-document-vector (method)
• normalize-matrix (generic function)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-matrix (method)
• normalize-term-vector (generic function)
• null-normalizer (class)
• pmi-normalizer (class)
• ri-term-assembler (class)
• row-centering-normalizer (class)
• sign-normalizer (class)
• tf-idf-normalizer (class)

Internal Definitions

• assemble-occurence-vector (function)
• class-counts (generic function)
• class-counts (method)
• column-norm (function)
• document-class-fn (generic function)
• document-class-fn (method)
• hash-table= (function)
• idfs (generic function)
• idfs (method)
• incf-and-maybe-grow (function)
• inverse-document-frequency (function)
• make-column-power-normalizer (function)
• make-lsa-assembler (function)
• make-null-normalizer (function)
• make-ri-term-assembler (function)
• make-tf-idf-normalizer (function)
• n-columns (function)
• n-documents (generic function)
• n-documents (method)
• n-documents (method)
• n-documents-with-term (function)
• n-rows (function)
• n-terms (generic function)
• n-terms (method)
• norm (function)
• normalize-column (function)
• normalize-vector (function)
• power (generic function)
• power (method)
• row-average (function)
• row-averages (generic function)
• row-averages (method)
• sum-column (function)
• sum-matrix (function)
• sum-row (function)
• sum-vector (function)
• term-counts (generic function)
• term-counts (method)
• term-counts-per-class (generic function)
• term-counts-per-class (method)
• term-frequency (function)
• term-total (generic function)
• term-total (method)
• term-total-per-class (generic function)
• term-total-per-class (method)
• test-pmi-normalizer (function)
• test-tf-idf-normalizer (function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Functions

Function: construct-document-vector LSA FEATURES

Construct a document vector from FEATURES. Inverse of DOCUMENT-VECTOR-FEATURES.

Package

lassie

Source

lsa.lisp (file)

Function: construct-term-vector LSA FEATURES

Construct a term vector from FEATURES. Inverse of TERM-VECTOR-FEATURES.

Package

lassie

Source

lsa.lisp (file)

Function: cosine-similarity X Y &key N

Package

lassie

Source

lsa-extra.lisp (file)

Function: document->vector LSA DOCUMENT

Turn DOCUMENT into a document vector.

Package

lassie

Source

lsa.lisp (file)

Function: document-features LSA DOCUMENT

Convenience function that returns the features of DOCUMENT after turning into into a vector with LSA.

Package

lassie

Source

lsa.lisp (file)

Function: document-vector-features LSA VECTOR-OR-INDEX

Return the feature vector for the document given by document VECTOR or INDEX.

Package

lassie

Source

lsa.lisp (file)

Function: load-lsa &key FILENAME SVD-FILENAME

Return the lsa loaded from FILENAME and SVD-FILENAME.

Package

lassie

Source

lsa.lisp (file)

Function: lsa &key TERM-MAPPER DOCUMENT-MAPPER TERM-LISTER DOCUMENT-LISTER TERM-INDEXER DOCUMENT-INDEXER ASSEMBLER NORMALIZER SUPERVISOR LEARNING-RATE NORMALIZATION-FACTOR

Perform LSA and return the lsa object that contains the SVD and remembers the mappers, indexers, ASSEMBLER and NORMALIZER for easy querying later by for example DOCUMENT-FEATURES.

This fat function assembles the co-occurrence matrix by iterating over all terms by TERM-LISTER and all documents by DOCUMENT-LISTER (either may be NIL). If DOCUMENT-LISTER is provided then DOCUMENT-MAPPER is employed to iterate over the terms of each document. Similarly TERM-MAPPER complements TERM-LISTER. TERM-INDEXER and DOCUMENT-INDEXER provide a - sometimes invertible - mapping from terms/documents to indices.

After the initial construction the mappers and indexers are stored in the LSA instance because they are needed to assemble term/document vectors later.

Finally the co-occurrence matrix is decomposed into singular vector pairs that define the feature spaces.

SUPERVISOR is a FSVD supervisor on which FSVD:SUPERVISE-SVD is invoked to control iteration (see FSVD:SVD). The lsa instance being constructed is passed as the :LSA argument to allow inspecting, saving, etc.

Package

lassie

Source

lsa.lisp (file)

Function: make-counting-indexer &key (COUNT COUNT)

Package

lassie.indexer

Source

indexer.lisp (file)

Function: make-hashing-indexer &key TEST

Package

lassie.indexer

Source

indexer.lisp (file)

Function: make-random-indexer &key LENGTH N TEST

Package

lassie.indexer

Source

indexer.lisp (file)

Function: most-similar-documents LSA DOCUMENT-FEATURES &key N TEST FILTER MEASURE

Return a vector of index and similarity pairs of the - at most N - documents whose features are most similar to DOCUMENT-FEATURES according to the similarity MEASURE.

Package

lassie

Source

lsa-extra.lisp (file)

Function: save-lsa LSA &key FILENAME SVD-FILENAME

Save LSA to FILENAME and its svd to SVD-FILENAME.

Package

lassie

Source

lsa.lisp (file)

Function: term->vector LSA TERM

Turn TERM into a document vector.

Package

lassie

Source

lsa.lisp (file)

Function: term-features LSA TERM

Convenience function that returns the features of TERM after turning into into a vector with LSA.

Package

lassie

Source

lsa.lisp (file)

Function: term-vector-features LSA VECTOR-OR-INDEX

Return the feature vector for the term given by term VECTOR or INDEX.

Package

lassie

Source

lsa.lisp (file)

5.1.2 Generic functions

Generic Function: ->index INDEXER OBJECT &key ALLOCATE-NEW-INDEX-P

Return an index representing OBJECT.

Package

lassie.indexer

Source

indexer.lisp (file)

Methods

Method: ->index (INDEXER random-indexer) OBJECT &key ALLOCATE-NEW-INDEX-P

Method: ->index (INDEXER hashing-indexer) OBJECT &key ALLOCATE-NEW-INDEX-P

Method: ->index (INDEXER counting-indexer) OBJECT &key ALLOCATE-NEW-INDEX-P

Generic Function: <-index INDEXER INDEX

Return the object that is encoded to INDEX.

Package

lassie.indexer

Source

indexer.lisp (file)

Methods

Method: <-index (INDEXER hashing-indexer) INDEX

Method: <-index (INDEXER counting-indexer) OBJECT

Generic Function: assemble-co-occurrence-matrix ASSEMBLER LISTER

Assemble MATRIX and remember how to perform the
same kind of activity on subsequent calls to ASSEMBLE-TERM-VECTOR and ASSEMBLE-DOCUMENT-VECTOR.

Package

lassie.assembler

Source

assemble.lisp (file)

Methods

Method: assemble-co-occurrence-matrix (ASSEMBLER ri-term-assembler) LISTER

Method: assemble-co-occurrence-matrix (ASSEMBLER lsa-assembler) LISTER

Generic Function: assemble-document-vector ASSEMBLER LISTER

Iterate over terms of LISTER and assemble a
document vector in the same way as the matrix was assembled previously.

Package

lassie.assembler

Source

assemble.lisp (file)

Methods

Method: assemble-document-vector (ASSEMBLER ri-term-assembler) LISTER

Method: assemble-document-vector (ASSEMBLER lsa-assembler) LISTER

Generic Function: assemble-term-vector ASSEMBLER LISTER

Iterate over documents of LISTER and assemble a
term vector in the same way as the matrix was assembled previously.

Package

lassie.assembler

Source

assemble.lisp (file)

Methods

Method: assemble-term-vector (ASSEMBLER lsa-assembler) LISTER

Generic Function: document-indexer OBJECT

Package

lassie

Methods

Method: document-indexer (LSA lsa)

Document indexer.

Source

lsa.lisp (file)

Generic Function: document-mapper OBJECT

Package

lassie

Methods

Method: document-mapper (LSA lsa)

A mapper over all terms that occur in a given document.

Source

lsa.lisp (file)

Generic Function: normalize-document-vector NORMALIZER DOCUMENT-VECTOR DOCUMENT

Returned the normalized DOCUMENT-VECTOR. Possibly desctructive.

Package

lassie.assembler

Source

normalizer.lisp (file)

Methods

Method: normalize-document-vector (NORMALIZER column-power-normalizer) DOCUMENT-VECTOR DOCUMENT

Method: normalize-document-vector (NORMALIZER row-centering-normalizer) DOCUMENT-VECTOR DOCUMENT

Method: normalize-document-vector (NORMALIZER null-normalizer) DOCUMENT-VECTOR DOCUMENT

Method: normalize-document-vector (NORMALIZER sign-normalizer) DOCUMENT-VECTOR DOCUMENT

Method: normalize-document-vector (NORMALIZER pmi-normalizer) DOCUMENT-VECTOR DOCUMENT

Method: normalize-document-vector (NORMALIZER tf-idf-normalizer) DOCUMENT-VECTOR DOCUMENT

Generic Function: normalize-matrix NORMALIZER MATRIX

Return the normalized MATRIX possibly destructively
and remember how to perform the same kind normalizations on subsequent calls to NORMALIZE-TERM and NORMALIZE-DOCUMENT.

Package

lassie.assembler

Source

normalizer.lisp (file)

Methods

Method: normalize-matrix (NORMALIZER column-power-normalizer) MATRIX

Method: normalize-matrix (NORMALIZER row-centering-normalizer) MATRIX

Method: normalize-matrix (NORMALIZER null-normalizer) MATRIX

Method: normalize-matrix (NORMALIZER sign-normalizer) MATRIX

Method: normalize-matrix (NORMALIZER pmi-normalizer) MATRIX

Method: normalize-matrix (NORMALIZER tf-idf-normalizer) MATRIX

Generic Function: normalize-term-vector NORMALIZER TERM-VECTOR TERM

Return the normalized TERM-VECTOR. Possibly desctructive.

Package

lassie.assembler

Source

normalizer.lisp (file)

Generic Function: normalizer OBJECT

Package

lassie

Methods

Method: normalizer (LSA lsa)

Performs some last minute transformations on the assembled matrix.

Source

lsa.lisp (file)

Generic Function: svd OBJECT

Package

lassie

Methods

Method: svd (LSA lsa)

The singular value decomposition.

Source

lsa.lisp (file)

Generic Function: term-indexer OBJECT

Package

lassie

Methods

Method: term-indexer (LSA lsa)

Term indexer.

Source

lsa.lisp (file)

Generic Function: term-mapper OBJECT

Package

lassie

Methods

Method: term-mapper (LSA lsa)

A mapper over all documents in which a given term occurs.

Source

lsa.lisp (file)

5.1.3 Classes

Class: column-power-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• normalize-document-vector (method)
• normalize-matrix (method)
• print-object (method)
• power (method)

Direct slots

Slot: power

Initargs

:power

Initform

2

Readers

power (generic function)

Class: lsa ()

This is not much more than a convenience class that
remembers how the SVD was produced to be able to extract features later, or just to know what a given row or column corresponds to.

Package

lassie

Source

lsa.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• svd (method)
• normalizer (method)
• assembler (method)
• document-indexer (method)
• term-indexer (method)
• document-mapper (method)
• term-mapper (method)

Direct slots

Slot: term-mapper

A mapper over all documents in which a given term occurs.

Initargs

:term-mapper

Readers

term-mapper (generic function)

Slot: document-mapper

A mapper over all terms that occur in a given document.

Initargs

:document-mapper

Readers

document-mapper (generic function)

Slot: term-indexer

Term indexer.

Initargs

:term-indexer

Readers

term-indexer (generic function)

Slot: document-indexer

Document indexer.

Initargs

:document-indexer

Readers

document-indexer (generic function)

Slot: assembler

Turns co-occurrences into a matrix, term and document vectors.

Initargs

:assembler

Readers

assembler (generic function)

Slot: normalizer

Performs some last minute transformations on the assembled matrix.

Initargs

:normalizer

Readers

normalizer (generic function)

Slot: svd

The singular value decomposition.

Initargs

:svd

Readers

svd (generic function)

Class: lsa-assembler ()

The standard assembler that adds ...

Package

lassie.assembler

Source

assemble.lisp (file)

Direct superclasses

standard-object (class)

Direct subclasses

ri-term-assembler (class)

Direct methods

• assemble-document-vector (method)
• assemble-term-vector (method)
• assemble-co-occurrence-matrix (method)
• print-object (method)
• n-documents (method)
• n-terms (method)

Direct slots

Slot: n-terms

Initargs

:n-terms

Readers

n-terms (generic function)

Slot: n-documents

Initargs

:n-documents

Readers

n-documents (generic function)

Class: null-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• normalize-document-vector (method)
• normalize-matrix (method)
• print-object (method)

Class: pmi-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• normalize-document-vector (method)
• normalize-matrix (method)
• n-documents (method)
• term-total-per-class (method)
• term-counts-per-class (method)
• class-counts (method)
• term-counts (method)
• term-total (method)
• document-class-fn (method)

Direct slots

Slot: document-class-fn

Initargs

:document-class-fn

Initform

(error "document-class-fn is required.")

Readers

document-class-fn (generic function)

Slot: term-total

Initargs

:term-total

Readers

term-total (generic function)

Slot: term-counts

Initargs

:term-counts

Readers

term-counts (generic function)

Slot: class-counts

Initargs

:class-counts

Readers

class-counts (generic function)

Slot: term-counts-per-class

Initargs

:term-counts-per-class

Readers

term-counts-per-class (generic function)

Slot: term-total-per-class

Initargs

:term-total-per-class

Readers

term-total-per-class (generic function)

Slot: n-documents

Initargs

:n-documents

Readers

n-documents (generic function)

Class: ri-term-assembler ()

Terms are random indexed, documents are not.

Package

lassie.assembler

Source

assemble.lisp (file)

Direct superclasses

lsa-assembler (class)

Direct methods

• assemble-document-vector (method)
• assemble-co-occurrence-matrix (method)
• print-object (method)

Class: row-centering-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• normalize-document-vector (method)
• normalize-matrix (method)
• row-averages (method)

Direct slots

Slot: row-averages

Readers

row-averages (generic function)

Class: sign-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• normalize-document-vector (method)
• normalize-matrix (method)

Class: tf-idf-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• normalize-document-vector (method)
• normalize-matrix (method)
• print-object (method)
• idfs (method)

Direct slots

Slot: idfs

The inverse document frequencies in the originally assembled matrix.

Initargs

:idfs

Readers

idfs (generic function)

5.2 Internal definitions

5.2.1 Functions

Function: %make-hashing-indexer &key (NEXT-INDEX NEXT-INDEX) (INDEX->OBJECT INDEX->OBJECT) (OBJECT->INDEX OBJECT->INDEX)

Package

lassie.indexer

Source

indexer.lisp (file)

Function: %make-random-indexer &key (LENGTH LENGTH) (N N) (OBJECT->INDEX OBJECT->INDEX)

Package

lassie.indexer

Source

indexer.lisp (file)

Function: alist->hashing-indexer TEST ALIST

Package

lassie.indexer

Source

indexer.lisp (file)

Function: assemble-occurence-vector LISTER SIZE

Return a vector of SIZE whose elements represent the frequency with which their indices were listed by LISTER.

Package

lassie.assembler

Source

assemble.lisp (file)

Function: column-norm MATRIX COLUMN &key POWER

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: compose-mappers &rest MAPPERS

Return a mapper that maps from the same set as the first of MAPPERS maps from and maps to what the last of MAPPERS maps to, composing them in a chain. If MAPPERS is NIL #’FUNCALL, the identity mapper, is returned.

Package

lassie

Source

mapper.lisp (file)

Function: concatente-mappers &rest MAPPERS

Return a mapper that is the concatention of MAPPERS.

Package

lassie

Source

mapper.lisp (file)

Function: construct-lsa-vector LSA FEATURES SV-SIDE

Package

lassie

Source

lsa.lisp (file)

Function: coordinate BASIS VECTOR

Return the length of the projection of VECTOR to BASIS.

Package

lassie

Source

lsa.lisp (file)

Function: copy-counting-indexer INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: copy-hashing-indexer INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: copy-random-indexer INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: counting-indexer-count INSTANCE

Function: (setf counting-indexer-count) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: counting-indexer-p OBJECT

Package

lassie.indexer

Source

indexer.lisp (file)

Function: curry-mapper MAPPER &rest CURRIED-ARGS

What makes a mapper is that the first is a function that is somehow applied to arguments. Currying a mapper leaves the function parameter alone and curries the rest of the parameters.

Package

lassie

Source

mapper.lisp (file)

Function: encode-mapper MAPPER ENCODER &key ALLOCATE-NEW-INDEX-P

Translate MAPPER by encoding its sole argument with ENCODER.

Package

lassie

Source

mapper.lisp (file)

Function: extract-lsa-features LSA VECTOR-OR-INDEX SV-OWN SV-OTHER

Package

lassie

Source

lsa.lisp (file)

Function: extract-svd-features SVD VECTOR FN

Package

lassie

Source

lsa.lisp (file)

Function: hash-table= ALIST HASH-TABLE

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: hashing-indexer->alist INDEXER

Package

lassie.indexer

Source

indexer.lisp (file)

Function: hashing-indexer-index->object INSTANCE

Function: (setf hashing-indexer-index->object) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: hashing-indexer-next-index INSTANCE

Function: (setf hashing-indexer-next-index) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: hashing-indexer-object->index INSTANCE

Function: (setf hashing-indexer-object->index) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: hashing-indexer-p OBJECT

Package

lassie.indexer

Source

indexer.lisp (file)

Function: incf-and-maybe-grow MATRIX DELTA &rest INDICES

Package

lassie.assembler

Source

assemble.lisp (file)

Function: inner* VECTOR1 VECTOR2

Package

lassie

Source

lsa.lisp (file)

Function: insert-into-sorted-vector ITEM VEC &key MAX-LENGTH KEY TEST

Insert ITEM into VECTOR while keeping it sorted by TEST. Extend the vector if needed while respecting MAX-LENGTH

Package

lassie

Source

lsa-extra.lisp (file)

Function: inverse-document-frequency MATRIX TERM

Relative importance of TERM across MATRIX.

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: make-column-power-normalizer &key POWER

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: make-encoded-term-document-lister TERM-MAPPER DOCUMENT-MAPPER TERM-ENCODER DOCUMENT-ENCODER TERM-LISTER DOCUMENT-LISTER

Return a lister that maps to encoded terms and documents. If DOCUMENT-LISTER is not NIL get its documents, encode them and list their terms with DOCUMENT-MAPPER. Act similary with TERM-LISTER and TERM-MAPPER.

Package

lassie

Source

mapper.lisp (file)

Function: make-encoded-term-document-mapper DOCUMENT-MAPPER TERM-ENCODER DOCUMENT-ENCODER

Return a mapper that applies to a document and calls its function argument with two parameters: the encoded term and the encoded document.

Package

lassie

Source

mapper.lisp (file)

Function: make-index-vector N LENGTH

Create a random index vector of LENGTH with N 1s and N -1s. It is stored as a sparse vector (only the indices of non-zero elements where the first N are +1 the rest are -1).

Package

lassie.indexer

Source

indexer.lisp (file)

Function: make-lsa-assembler &key N-TERMS N-DOCUMENTS

Package

lassie.assembler

Source

assemble.lisp (file)

Function: make-mapper &rest SEQUENCES

Return a mapper that maps from SEQUENCES to elements of SEQUENCES.

Package

lassie

Source

mapper.lisp (file)

Function: make-null-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: make-ri-term-assembler ()

Package

lassie.assembler

Source

assemble.lisp (file)

Function: make-tf-idf-normalizer &key IDFS

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: n-columns MATRIX

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: n-documents-with-term MATRIX TERM

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: n-rows MATRIX

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: norm VECTOR &key POWER

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: normalize-column MATRIX COLUMN &key POWER

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: normalize-vector VECTOR &key POWER

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: null-mapper &rest ARGS

Package

lassie

Source

mapper.lisp (file)

Function: random-indexer-length INSTANCE

Function: (setf random-indexer-length) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: random-indexer-n INSTANCE

Function: (setf random-indexer-n) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: random-indexer-object->index INSTANCE

Function: (setf random-indexer-object->index) VALUE INSTANCE

Package

lassie.indexer

Source

indexer.lisp (file)

Function: random-indexer-p OBJECT

Package

lassie.indexer

Source

indexer.lisp (file)

Function: row-average MATRIX ROW

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: sum-column MATRIX COLUMN

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: sum-matrix MATRIX

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: sum-row MATRIX ROW

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: sum-vector VECTOR

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: term-frequency MATRIX TERM DOCUMENT

A normalized measure of how often TERM appears in DOCUMENT.

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: test-compose-mappers ()

Package

lassie

Source

mapper.lisp (file)

Function: test-hashing-indexer ()

Package

lassie.indexer

Source

indexer.lisp (file)

Function: test-pmi-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

Function: test-tf-idf-normalizer ()

Package

lassie.assembler

Source

normalizer.lisp (file)

5.2.2 Generic functions

Generic Function: assembler OBJECT

Package

lassie

Methods

Method: assembler (LSA lsa)

Turns co-occurrences into a matrix, term and document vectors.

Source

lsa.lisp (file)

Generic Function: class-counts OBJECT

Package

lassie.assembler

Methods

Method: class-counts (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: document-class-fn OBJECT

Package

lassie.assembler

Methods

Method: document-class-fn (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: idfs OBJECT

Package

lassie.assembler

Methods

Method: idfs (TF-IDF-NORMALIZER tf-idf-normalizer)

The inverse document frequencies in the originally assembled matrix.

Source

normalizer.lisp (file)

Generic Function: map-lines FUNCTION OBJECT

Package

lassie

Source

mapper.lisp (file)

Methods

Method: map-lines FUNCTION (STREAM stream)

Method: map-lines FUNCTION (PATHNAME pathname)

Method: map-lines FUNCTION (STRING string)

Generic Function: n-documents OBJECT

Package

lassie.assembler

Methods

Method: n-documents (LSA-ASSEMBLER lsa-assembler)

automatically generated reader method

Source

assemble.lisp (file)

Method: n-documents (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: n-terms OBJECT

Package

lassie.assembler

Methods

Method: n-terms (LSA-ASSEMBLER lsa-assembler)

automatically generated reader method

Source

assemble.lisp (file)

Generic Function: power OBJECT

Package

lassie.assembler

Methods

Method: power (COLUMN-POWER-NORMALIZER column-power-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: row-averages OBJECT

Package

lassie.assembler

Methods

Method: row-averages (ROW-CENTERING-NORMALIZER row-centering-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: term-counts OBJECT

Package

lassie.assembler

Methods

Method: term-counts (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: term-counts-per-class OBJECT

Package

lassie.assembler

Methods

Method: term-counts-per-class (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: term-total OBJECT

Package

lassie.assembler

Methods

Method: term-total (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

Generic Function: term-total-per-class OBJECT

Package

lassie.assembler

Methods

Method: term-total-per-class (PMI-NORMALIZER pmi-normalizer)

automatically generated reader method

Source

normalizer.lisp (file)

5.2.3 Structures

Structure: counting-indexer ()

Simply assigns a new index to every object.

Package

lassie.indexer

Source

indexer.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

• <-index (method)
• ->index (method)

Direct slots

Slot: count

Initform

0

Readers

counting-indexer-count (function)

Writers

(setf counting-indexer-count) (function)

Structure: hashing-indexer ()

Package

lassie.indexer

Source

indexer.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

• print-object (method)
• <-index (method)
• ->index (method)

Direct slots

Slot: next-index

Initform

0

Readers

hashing-indexer-next-index (function)

Writers

(setf hashing-indexer-next-index) (function)

Slot: index->object

Readers

hashing-indexer-index->object (function)

Writers

(setf hashing-indexer-index->object) (function)

Slot: object->index

Readers

hashing-indexer-object->index (function)

Writers

(setf hashing-indexer-object->index) (function)

Structure: random-indexer ()

Package

lassie.indexer

Source

indexer.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

->index (method)

Direct slots

Slot: length

Readers

random-indexer-length (function)

Writers

(setf random-indexer-length) (function)

Slot: n

Readers

random-indexer-n (function)

Writers

(setf random-indexer-n) (function)

Slot: object->index

Readers

random-indexer-object->index (function)

Writers

(setf random-indexer-object->index) (function)

5.2.4 Types

Type: single-float-vector ()

Package

lassie

Source

lsa.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions