The cl-conllu Reference Manual

Table of Contents

The cl-conllu Reference Manual

This is the cl-conllu Reference Manual, version 0.3, generated automatically by Declt version 2.4 "Will Decker" on Wed Jun 20 11:02:10 2018 GMT+0.

1 Introduction

#+title: Library for working with CoNLL-U files with CL

The =cl-conllu= is a Common Lisp library to work with [[http://universaldependencies.org/format.html][CoNLL-U]],
licensed under the [[http://www.apache.org/licenses/LICENSE-2.0][Apache license]].

It is developed and tested with SBCL but should probably run with any
other implementation.

* Quicklisp

If don't have quicklisp installed already, follow [[https://www.quicklisp.org/beta/#installation][these steps]].

The =cl-conllu= library is now available from quicklisp distribution,
if you are not planning to change the code, just use:

#+BEGIN_SRC lisp
(ql:quickload :cl-conllu)
#+END_SRC

If you plan on contributing, clone this project to your
=local-projects= quicklisp directory (usually at
=~/quicklisp/local-projects/=) and use the same command as above.

* Reading CoNLL-U files

First you need some CoNLL-U files to start with. If you have none in
mind, you may get some from [[https://github.com/own-pt/bosque-UD/tree/master/documents][here]].

The simplest way to read a file or a directory of =conllu= files is:

#+BEGIN_SRC lisp
CL-USER> (defparameter *sents* (cl-conllu:read-conllu #P"/path/to/my/file/CF1.conllu"))
(# #
 # #
 # #
 #)
#+END_SRC

Each object returned is an instance of a =sentence= class, made up of
=token= objects, which we will describe in the next section.

All read functions accept a =fn-meta= function as argument. This
function collects the metadata from each CoNLL-U sentence, which
usually includes the raw sentence (see [[http://universaldependencies.org/format.html][format]]). The default metadata
collector function is =collect-meta=.

* The Classes

=cl-conllu= has a few central classes: =sentence=, =token=, and
=mtoken=. They are all defined in =data.lisp= file.  When a CoNLL-U
file is read, its contents are turned into instances of these classes.

** Sentences

Every CoNLL-U sentence is turned in an instance of the =sentence=
class by =cl-conllu=. Each instance is characterized by four
properties: =start=, =meta=, =tokens=, and =mtokens=. The =start=
field contains the line number of the file where the sentence block
started.

The =meta= field includes the metainformation regarding the
sentence. This information may vary, as we have discussed in the
previous section, but usually includes the full (raw) sentence and the
sentence ID, as required by the CoNLL-U format specification.

#+BEGIN_EXAMPLE
CL-USER> (cl-conllu:sentence-meta (first *sents*))
(("text" . "PT no governo")
 ("source" . "CETENFolha n=1 cad=Opinião sec=opi sem=94a")
 ("sent_id" . "CF1-1") ("id" . "1"))
#+END_EXAMPLE

The =tokens= are the list of tokens that together form the sentence,
and they are themselves instances of the =token= class.

The =mtokens= (meta-tokens) are also instances of their own =mtoken=
class, and they are used for multiword tokens (vámonos = vamos + nos).

** Tokens

Instances of the =token= class have one property for each field/column
in the CoNLL-U format's sentences, that is:

- ID :: Word index, integer starting at 1 for each new sentence; may
        be a range for multiword tokens; may be a decimal number for
        empty nodes.
- FORM :: Word form or punctuation symbol.
- LEMMA :: Lemma of word form.
- UPOSTAG :: Universal part-of-speech tag.
- XPOSTAG :: Language-specific part-of-speech tag; underscore if not
             available.
- FEATS :: List of morphological features from the universal feature
           inventory or from a defined language-specific extension;
           underscore if not available.
- HEAD :: Head of the current word, which is either a value of ID or
          zero if the token is the root (0).
- DEPREL :: Universal dependency relation to the HEAD (=root= iff HEAD
            is 0) or a defined language-specific subtype of one.
- DEPS :: Enhanced dependency graph in the form of a list of
          head-deprel pairs.
- MISC :: Any other annotation.

* Visualizing CoNLL-U sentences

To visualize CoNLL-U sentences, we use the =conllu-visualize=
subpackage. The function =tree-sentence= receives an instance of a
sentence object and (optionally) an output stream, and outputs to the
stream the sentence's metadata and its tree structure:

#+BEGIN_EXAMPLE
(conllu.draw:tree-sentence (nth 5 *frases*))
text = Eles se dizem oposição, mas ainda não informaram o que vão combater.
source = CETENFolha n=1 cad=Opinião sec=opi sem=94a
sent_id = CF1-7
id = 6
─┮ 
 │ ╭─╼ Eles nsubj 
 │ ├─╼ se expl 
 ╰─┾ dizem root 
   ├─╼ oposição xcomp 
   │ ╭─╼ , punct 
   │ ├─╼ mas cc 
   │ │ ╭─╼ ainda advmod 
   │ ├─┶ não advmod 
   ├─┾ informaram conj 
   │ │   ╭─╼ o det 
   │ │ ╭─┶ que obj 
   │ │ ├─╼ vão aux 
   │ ╰─┶ combater ccomp 
   ╰─╼ . punct 
#+END_EXAMPLE

* Querying CoNLL-U files

Queries can be executed with 

#+BEGIN_SRC lisp
  (query '(nsubj (advcl (and (upostag "VERB") (lemma "correr"))
			(upostag "VERB" )) 
		 (upostag "PROP"))
	  *sents*)
#+END_SRC

* How to cite

http://arademaker.github.io/bibliography/tilic-stil-2017.html

#+BEGIN_EXAMPLE
@inproceedings{tilic-stil-2017,
  author = {Muniz, Henrique and Chalub, Fabricio and Rademaker, Alexandre},
  title = {CL-CONLLU: dependências universais em Common Lisp},
  booktitle = {V Workshop de Iniciação Científica em Tecnologia da
                    Informação e da Linguagem Humana (TILic)},
  year = {2017},
  address = {Uberlândia, MG, Brazil},
  note = {https://sites.google.com/view/tilic2017/}
}
#+END_EXAMPLE

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 cl-conllu

Author

Fabricio Chalub <fchalub@br.ibm.com>

License

Apache 2.0

Description

Common Lisp corpus conllu utilities

Version

0.3

Dependencies

• cl-ppcre
• uuid
• alexandria
• cl-log
• split-sequence
• xmls
• yason
• lispbuilder-lexer
• wilbur

Source

cl-conllu.asd (file)

Components

• packages.lisp (file)
• data.lisp (file)
• read-write.lisp (file)
• evaluate.lisp (file)
• evaluate-template.lisp (file)
• query.lisp (file)
• utils.lisp (file)
• rdf.lisp (file)
• rdf-wilbur.lisp (file)
• command-line.lisp (file)
• rules.lisp (file)
• conllu-prolog.lisp (file)
• niceline.lisp (file)
• tag-converter.lisp (file)
• draw.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 cl-conllu.asd

Location

cl-conllu.asd

Systems

cl-conllu (system)

3.1.2 cl-conllu/packages.lisp

Parent

cl-conllu (system)

Location

packages.lisp

Packages

• conllu.draw
• conllu.converters.tags
• conllu.converters.niceline
• conllu.rdf
• conllu.prolog
• cl-conllu

3.1.3 cl-conllu/data.lisp

Dependencies

• packages.lisp (file)
• packages.lisp (file)

Parent

cl-conllu (system)

Location

data.lisp

Exported Definitions

• adjust-sentence (function)
• mtoken (class)
• mtoken-end (method)
• (setf mtoken-end) (method)
• mtoken-form (method)
• (setf mtoken-form) (method)
• mtoken-start (method)
• (setf mtoken-start) (method)
• sentence (class)
• sentence->text (function)
• sentence-binary-tree (function)
• sentence-equal (function)
• sentence-hash-table (function)
• sentence-id (function)
• sentence-meta (method)
• (setf sentence-meta) (method)
• sentence-meta-value (function)
• sentence-mtokens (method)
• (setf sentence-mtokens) (method)
• sentence-start (method)
• (setf sentence-start) (method)
• sentence-text (function)
• sentence-tokens (method)
• (setf sentence-tokens) (method)
• sentence-valid? (function)
• token (class)
• token-deprel (method)
• (setf token-deprel) (method)
• token-deps (method)
• (setf token-deps) (method)
• token-feats (method)
• (setf token-feats) (method)
• token-form (method)
• (setf token-form) (method)
• token-head (method)
• (setf token-head) (method)
• token-id (method)
• (setf token-id) (method)
• token-lemma (method)
• (setf token-lemma) (method)
• token-misc (method)
• (setf token-misc) (method)
• token-upostag (method)
• (setf token-upostag) (method)
• token-xpostag (method)
• (setf token-xpostag) (method)

Internal Definitions

• *token-fields* (special variable)
• insert-token (function)
• is-descendant? (function)
• mtoken->tokens (function)
• mtoken-equal (function)
• mtoken-misc (method)
• (setf mtoken-misc) (method)
• remove-token (function)
• sentence-matrix (function)
• sentence-root (function)
• sentence-size (function)
• set-head (function)
• simple-deprel (function)
• token-children (function)
• token-equal (function)
• token-parent (function)

3.1.4 cl-conllu/read-write.lisp

Dependencies

• data.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

read-write.lisp

Exported Definitions

• make-sentence (function)
• read-conllu (function)
• read-directory (function)
• read-file (function)
• read-stream (function)
• write-conllu (function)
• write-conllu-to-stream (function)

Internal Definitions

• collect-meta (function)
• line->mtoken (function)
• line->token (function)
• list-to-tsv (function)
• write-mtoken (function)
• write-sentence (function)
• write-token (function)

3.1.5 cl-conllu/evaluate.lisp

Dependencies

• read-write.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

evaluate.lisp

Internal Definitions

• *deprel-value-list* (special variable)
• attachment-score-by-sentence (function)
• attachment-score-by-word (function)
• confusion-matrix (function)
• exact-match (function)
• exact-match-score (function)
• exact-match-sentence (function)
• format-matrix (function)
• non-projectivity-accuracy (function)
• non-projectivity-precision (function)
• non-projectivity-recall (function)
• precision (function)
• recall (function)
• sentence-average-score (function)
• sentence-diff (function)
• token-diff (function)
• word-average-score (function)

3.1.6 cl-conllu/evaluate-template.lisp

Dependencies

• evaluate.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

evaluate-template.lisp

Internal Definitions

• add-diff-log (function)
• allowed-value (function)
• confusion-table-access-cell (function)
• confusion-table-add-column (function)
• confusion-table-add-pair (function)
• confusion-table-rows (function)
• diff-sentences (function)
• format-cell-html (function)
• format-dependency (function)
• format-file-name (function)
• format-link-html (function)
• format-row-html (function)
• format-sentence-text (function)
• format-table-html (function)
• get-diff-file-path (function)
• get-token-parent (function)
• html-dependency-pair (function)
• html-diff-log (function)
• html-make-header (function)
• html-make-info-line (function)
• html-set-bold (function)
• html-set-font-color (function)
• html-write-charset (function)
• html-write-style-css (function)
• report-confusion-table (function)
• report-diff (function)
• write-confusion-table (function)
• write-diffs-to-files (function)

3.1.7 cl-conllu/query.lisp

Dependencies

• evaluate-template.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

query.lisp

Exported Definitions

• query (function)
• query-as-json (function)

Internal Definitions

• *deprels* (special variable)
• and% (function)
• children (function)
• eval-query (function)
• or% (function)
• r~ (function)
• t~ (function)

3.1.8 cl-conllu/utils.lisp

Dependencies

• query.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

utils.lisp

Exported Definitions

• diff (function)
• levenshtein (function)
• non-projective? (function)

Internal Definitions

• find-min (function)
• insert-at (function)
• mappend (function)
• print-diff (function)

3.1.9 cl-conllu/rdf.lisp

Dependencies

• utils.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

rdf.lisp

Exported Definitions

• convert-rdf (function)
• convert-rdf-file (function)

Internal Definitions

• components->string (function)
• convert-sentence-to-turtle (function)
• escape-string (function)
• escape-turtle-char (function)
• make-dep (function)
• make-featurename (function)
• make-features (function)
• make-features-bnode (function)
• make-literal (function)
• make-metadata (function)
• make-metadata-bnode (function)
• make-token-id (function)
• make-upos (function)
• unspecified-field? (function)

3.1.10 cl-conllu/rdf-wilbur.lisp

Dependencies

• rdf.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

rdf-wilbur.lisp

Exported Definitions

convert-to-rdf (function)

Internal Definitions

• convert-features-to-rdf (function)
• convert-sentence-metadata (function)
• convert-sentence-to-rdf (function)
• convert-token-to-rdf (function)

3.1.11 cl-conllu/command-line.lisp

Dependencies

• rdf-wilbur.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

command-line.lisp

Internal Definitions

• adjust-conllu (function)
• modify-conllu (function)
• select-sentence (function)
• write-selected-sentence (function)

3.1.12 cl-conllu/rules.lisp

Dependencies

• command-line.lisp (file)
• utils.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

rules.lisp

Exported Definitions

• apply-rules (function)
• apply-rules-from-files (function)

Internal Definitions

• add-value (function)
• apply-changes (function)
• apply-conditions-in-token (function)
• apply-rhs (function)
• apply-rule-in-sentence (function)
• apply-rule-in-tokens (function)
• apply-rules-in-sentence (function)
• apply-rules-in-sentence-aux (function)
• equal-op (function)
• intern-pattern (function)
• intern-rule (function)
• intern-sides (function)
• match-token (function)
• match? (function)
• member-op (function)
• modify-value (function)
• read-rules (function)
• regex-op (function)
• rhs (function)
• rhs-vars (function)
• rls (function)
• rls-vars (function)
• valid-vars (function)
• variable-p (function)

3.1.13 cl-conllu/conllu-prolog.lisp

Dependencies

• rules.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

conllu-prolog.lisp

Exported Definitions

convert-filename (function)

Internal Definitions

• *clauses* (special variable)
• *dependencies* (special variable)
• clean-dep-rel (function)
• clean-whitespace (function)
• emit-prolog (function)
• is-root (function)
• make-id (function)
• process-features (function)
• process-tokens (function)
• prolog-string (function)
• toprologid (function)
• valid-line (function)
• write-prolog (function)

3.1.14 cl-conllu/niceline.lisp

Dependencies

• conllu-prolog.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

niceline.lisp

Internal Definitions

• add-field-value (function)
• consume-tags (function)
• convert-directory (function)
• malformed-line (method)
• malformed-line-error (condition)
• niceline (function)
• niceline-matcher (special variable)
• read-niceline (function)
• read-niceline-file (function)
• read-niceline-stream (function)

3.1.15 cl-conllu/tag-converter.lisp

Dependencies

• niceline.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

tag-converter.lisp

Exported Definitions

• read-file-tag-suffix (function)
• read-sentence-tag-suffix (function)
• write-sentence-tag-suffix-to-stream (function)
• write-sentences-tag-suffix (function)
• write-sentences-tag-suffix-to-stream (function)

Internal Definitions

write-token-tag-suffix (function)

3.1.16 cl-conllu/draw.lisp

Dependencies

• tag-converter.lisp (file)
• data.lisp (file)

Parent

cl-conllu (system)

Location

draw.lisp

Exported Definitions

tree-sentence (function)

Internal Definitions

• add (function)
• between (function)
• children (function)
• collect-slots (function)
• draw-descendent (function)
• draw-father (function)
• draw-father-lines (function)
• draw-no-descendets (function)
• draw-node (function)
• draw-node-values (function)
• is-not-just-a-leaf (function)
• make-branchs (function)
• make-root-lines (function)
• make-tree (function)
• maximum-between (function)
• root-leaf (function)

4 Packages

Packages are listed by definition order.

4.1 conllu.draw

Source

packages.lisp (file)

Use List

• cl-conllu
• common-lisp

Exported Definitions

tree-sentence (function)

Internal Definitions

• add (function)
• between (function)
• children (function)
• collect-slots (function)
• draw-descendent (function)
• draw-father (function)
• draw-father-lines (function)
• draw-no-descendets (function)
• draw-node (function)
• draw-node-values (function)
• is-not-just-a-leaf (function)
• make-branchs (function)
• make-root-lines (function)
• make-tree (function)
• maximum-between (function)
• root-leaf (function)

4.2 conllu.converters.tags

Source

packages.lisp (file)

Use List

• cl-ppcre
• cl-conllu
• common-lisp

Exported Definitions

• read-file-tag-suffix (function)
• read-sentence-tag-suffix (function)
• write-sentence-tag-suffix-to-stream (function)
• write-sentences-tag-suffix (function)
• write-sentences-tag-suffix-to-stream (function)

Internal Definitions

write-token-tag-suffix (function)

4.3 conllu.converters.niceline

Source

packages.lisp (file)

Use List

• lispbuilder-lexer
• cl-conllu
• common-lisp

Internal Definitions

• add-field-value (function)
• consume-tags (function)
• convert-directory (function)
• malformed-line (generic function)
• malformed-line (method)
• malformed-line-error (condition)
• niceline (function)
• niceline-matcher (special variable)
• read-niceline (function)
• read-niceline-file (function)
• read-niceline-stream (function)

4.4 conllu.rdf

Source

packages.lisp (file)

Use List

• split-sequence
• cl-conllu
• alexandria.0.dev
• wilbur
• common-lisp

Internal Definitions

• components->string (function)
• convert-features-to-rdf (function)
• convert-sentence-metadata (function)
• convert-sentence-to-rdf (function)
• convert-sentence-to-turtle (function)
• convert-token-to-rdf (function)
• escape-string (function)
• escape-turtle-char (function)
• make-dep (function)
• make-featurename (function)
• make-features (function)
• make-features-bnode (function)
• make-literal (function)
• make-metadata (function)
• make-metadata-bnode (function)
• make-token-id (function)
• make-upos (function)
• unspecified-field? (function)

4.5 conllu.prolog

Source

packages.lisp (file)

Use List

• cl-conllu
• split-sequence
• alexandria.0.dev
• common-lisp

Exported Definitions

convert-filename (function)

Internal Definitions

• *clauses* (special variable)
• *dependencies* (special variable)
• clean-dep-rel (function)
• clean-whitespace (function)
• emit-prolog (function)
• is-root (function)
• make-id (function)
• process-features (function)
• process-tokens (function)
• prolog-string (function)
• toprologid (function)
• valid-line (function)
• write-prolog (function)

4.6 cl-conllu

Source

packages.lisp (file)

Use List

• com.ravenbrook.common-lisp-log
• split-sequence
• cl-ppcre
• common-lisp

Used By List

• conllu.draw
• conllu.converters.tags
• conllu.converters.niceline
• conllu.rdf
• conllu.prolog

Exported Definitions

• adjust-sentence (function)
• apply-rules (function)
• apply-rules-from-files (function)
• convert-rdf (function)
• convert-rdf-file (function)
• convert-to-rdf (function)
• diff (function)
• levenshtein (function)
• make-sentence (function)
• mtoken (class)
• mtoken-end (generic function)
• mtoken-end (method)
• (setf mtoken-end) (method)
• (setf mtoken-end) (generic function)
• mtoken-form (generic function)
• mtoken-form (method)
• (setf mtoken-form) (method)
• (setf mtoken-form) (generic function)
• mtoken-start (generic function)
• mtoken-start (method)
• (setf mtoken-start) (method)
• (setf mtoken-start) (generic function)
• non-projective? (function)
• query (function)
• query-as-json (function)
• read-conllu (function)
• read-directory (function)
• read-file (function)
• read-stream (function)
• sentence (class)
• sentence->text (function)
• sentence-binary-tree (function)
• sentence-equal (function)
• sentence-hash-table (function)
• sentence-id (function)
• sentence-meta (generic function)
• sentence-meta (method)
• (setf sentence-meta) (method)
• (setf sentence-meta) (generic function)
• sentence-meta-value (function)
• sentence-mtokens (generic function)
• sentence-mtokens (method)
• (setf sentence-mtokens) (method)
• (setf sentence-mtokens) (generic function)
• sentence-start (generic function)
• sentence-start (method)
• (setf sentence-start) (method)
• (setf sentence-start) (generic function)
• sentence-text (function)
• sentence-tokens (generic function)
• sentence-tokens (method)
• (setf sentence-tokens) (method)
• (setf sentence-tokens) (generic function)
• sentence-valid? (function)
• token (class)
• token-deprel (generic function)
• token-deprel (method)
• (setf token-deprel) (method)
• (setf token-deprel) (generic function)
• token-deps (generic function)
• token-deps (method)
• (setf token-deps) (method)
• (setf token-deps) (generic function)
• token-feats (generic function)
• token-feats (method)
• (setf token-feats) (method)
• (setf token-feats) (generic function)
• token-form (generic function)
• token-form (method)
• (setf token-form) (method)
• (setf token-form) (generic function)
• token-head (generic function)
• token-head (method)
• (setf token-head) (method)
• (setf token-head) (generic function)
• token-id (generic function)
• token-id (method)
• (setf token-id) (method)
• (setf token-id) (generic function)
• token-lemma (generic function)
• token-lemma (method)
• (setf token-lemma) (method)
• (setf token-lemma) (generic function)
• token-misc (generic function)
• token-misc (method)
• (setf token-misc) (method)
• (setf token-misc) (generic function)
• token-upostag (generic function)
• token-upostag (method)
• (setf token-upostag) (method)
• (setf token-upostag) (generic function)
• token-xpostag (generic function)
• token-xpostag (method)
• (setf token-xpostag) (method)
• (setf token-xpostag) (generic function)
• write-conllu (function)
• write-conllu-to-stream (function)

Internal Definitions

• *deprel-value-list* (special variable)
• *deprels* (special variable)
• *token-fields* (special variable)
• add-diff-log (function)
• add-value (function)
• adjust-conllu (function)
• allowed-value (function)
• and% (function)
• apply-changes (function)
• apply-conditions-in-token (function)
• apply-rhs (function)
• apply-rule-in-sentence (function)
• apply-rule-in-tokens (function)
• apply-rules-in-sentence (function)
• apply-rules-in-sentence-aux (function)
• attachment-score-by-sentence (function)
• attachment-score-by-word (function)
• children (function)
• collect-meta (function)
• confusion-matrix (function)
• confusion-table-access-cell (function)
• confusion-table-add-column (function)
• confusion-table-add-pair (function)
• confusion-table-rows (function)
• diff-sentences (function)
• equal-op (function)
• eval-query (function)
• exact-match (function)
• exact-match-score (function)
• exact-match-sentence (function)
• find-min (function)
• format-cell-html (function)
• format-dependency (function)
• format-file-name (function)
• format-link-html (function)
• format-matrix (function)
• format-row-html (function)
• format-sentence-text (function)
• format-table-html (function)
• get-diff-file-path (function)
• get-token-parent (function)
• html-dependency-pair (function)
• html-diff-log (function)
• html-make-header (function)
• html-make-info-line (function)
• html-set-bold (function)
• html-set-font-color (function)
• html-write-charset (function)
• html-write-style-css (function)
• insert-at (function)
• insert-token (function)
• intern-pattern (function)
• intern-rule (function)
• intern-sides (function)
• is-descendant? (function)
• line->mtoken (function)
• line->token (function)
• list-to-tsv (function)
• mappend (function)
• match-token (function)
• match? (function)
• member-op (function)
• modify-conllu (function)
• modify-value (function)
• mtoken->tokens (function)
• mtoken-equal (function)
• mtoken-misc (generic function)
• mtoken-misc (method)
• (setf mtoken-misc) (method)
• (setf mtoken-misc) (generic function)
• non-projectivity-accuracy (function)
• non-projectivity-precision (function)
• non-projectivity-recall (function)
• or% (function)
• precision (function)
• print-diff (function)
• read-rules (function)
• recall (function)
• regex-op (function)
• remove-token (function)
• report-confusion-table (function)
• report-diff (function)
• rhs (function)
• rhs-vars (function)
• rls (function)
• rls-vars (function)
• r~ (function)
• select-sentence (function)
• sentence-average-score (function)
• sentence-diff (function)
• sentence-matrix (function)
• sentence-root (function)
• sentence-size (function)
• set-head (function)
• simple-deprel (function)
• token-children (function)
• token-diff (function)
• token-equal (function)
• token-parent (function)
• t~ (function)
• valid-vars (function)
• variable-p (function)
• word-average-score (function)
• write-confusion-table (function)
• write-diffs-to-files (function)
• write-mtoken (function)
• write-selected-sentence (function)
• write-sentence (function)
• write-token (function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Functions

Function: adjust-sentence SENTENCE

Receives a sentence and reenumerate IDs and HEAD values of each token so that their order (as in sentence-tokens) is respected.

Package

cl-conllu

Source

data.lisp (file)

Function: apply-rules SENTENCES RULES RECURSIVE

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-rules-from-files CONLLU-FILE RULES-FILE NEW-CONLLU-FILE LOG-FILE &key RECURSIVE

Package

cl-conllu

Source

rules.lisp (file)

Function: convert-filename CONTEXT FILENAME-IN FILENAME-OUT

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Function: convert-rdf CORPUSNAME STREAM CONLLS TEXT-FN ID-FN

Converts the collection of sentences (as generated by READ-CONLLU) in CONLL, using the function TEXT-FN to extract the text of each sentence and ID-FN to extract the id of each sentence (we need this as there is no standardized way of knowing this.) Also the generated Turtle file contains a lot of duplication so when you import it into your triple-store, make sure you remove all duplicate triples afterwards.

Package

cl-conllu

Source

rdf.lisp (file)

Function: convert-rdf-file FILE-IN FILE-OUT

Package

cl-conllu

Source

rdf.lisp (file)

Function: convert-to-rdf SENTENCES &key TEXT-FN ID-FN CORPUSNAME NAMESPACE-STRING STREAM RDF-FORMAT CONLL-NAMESPACE

Converts a list of sentences (e.g. as generated by READ-CONLLU)
in SENTENCES, using the function TEXT-FN to extract the text of each sentence and ID-FN to extract the id of each sentence (we need this as there is no standardized way of knowing this.)

Currently only ntriples is supported as RDF-FORMAT.

Package

cl-conllu

Source

rdf-wilbur.lisp (file)

Function: diff SENTENCES-A SENTENCES-B &key TEST KEY

Package

cl-conllu

Source

utils.lisp (file)

Function: levenshtein S1 S2 &key TEST

Package

cl-conllu

Source

utils.lisp (file)

Function: make-sentence LINENO LINES FN-META

Package

cl-conllu

Source

read-write.lisp (file)

Function: non-projective? SENTENCE

Verifies if a sentence tree is projective. Intuitively, this means that, keeping word order, there’s no two dependency arcs that cross.
More formally, let i -> j mean that j’s head is node i. Let ’->*’ be the transitive closure of ’->’.

A tree if projective when, for each node i, j: if i -> j, then for each node k between i and j (i < k < j or j < k < i), i ->* k.

References:
- Nivre, Joakim; Inductive Dependency Parsing, 2006
- https://en.wikipedia.org/wiki/Discontinuity_(linguistics)

Package

cl-conllu

Source

utils.lisp (file)

Function: query QUERY SENTENCES

Package

cl-conllu

Source

query.lisp (file)

Function: query-as-json A-QUERY SENTENCES

Package

cl-conllu

Source

query.lisp (file)

Function: read-conllu INPUT &key FN-META

Package

cl-conllu

Source

read-write.lisp (file)

Function: read-directory PATH &key FN-META

Package

cl-conllu

Source

read-write.lisp (file)

Function: read-file PATH &key FN-META

Package

cl-conllu

Source

read-write.lisp (file)

Function: read-file-tag-suffix FILENAME &key TAG SEPARATOR

Package

conllu.converters.tags

Source

tag-converter.lisp (file)

Function: read-sentence-tag-suffix STREAM FIELD SEPARATOR

Writes as sentence object input from STREAM as FORM.SEPARATOR.TAGVALUE (without dots), followed by a whitespace character.

Example:

;; Consider the file example.txt, with contents:
;; Pudim_NOUN é_VERB bom_ADJ ._PUNCT
;; E_CONJ torta_NOUN também_ADV ._PUNCT

(with-open-file (s "./example.txt")
(write-conllu-to-stream (read-sentence-tag-suffix s ’upostag "_"))) 1 Pudim _ NOUN _ _ _ _ _ _
2 é _ VERB _ _ _ _ _ _
3 bom _ ADJ _ _ _ _ _ _
4 . _ PUNCT _ _ _ _ _ _

1 E _ CONJ _ _ _ _ _ _
2 torta _ NOUN _ _ _ _ _ _
3 também _ ADV _ _ _ _ _ _
4 . _ PUNCT _ _ _ _ _ _

Package

conllu.converters.tags

Source

tag-converter.lisp (file)

Function: read-stream STREAM &key FN-META

Package

cl-conllu

Source

read-write.lisp (file)

Function: sentence->text SENTENCE &key IGNORE-MTOKENS SPECIAL-FORMAT-TEST SPECIAL-FORMAT-FUNCTION

Receives SENTENCE, a sentence object, and returns a string reconstructed from its tokens and mtokens.

If IGNORE-MTOKENS, then tokens’ forms are used. Else, tokens with id contained in a mtoken are not used, with mtoken’s form being used instead.

It is possible to special format some tokens. In order to do so, both SPECIAL-FORMAT-TEST and SPECIAL-FORMAT-FUNCTION should be passed. Then for each object (token or mtoken) for which SPECIAL-FORMAT-TEST returns a non-nil result, its form is modified by SPECIAL-FORMAT-FUNCTION in the final string.

Example:
(sentence-tokens *sentence*)
=> (#<TOKEN The/DET #1-det-3> #<TOKEN US/PROPN #2-compound-3> #<TOKEN troops/NOUN #3-nsubj-4> #<TOKEN fired/VERB #4-root-0> #<TOKEN into/ADP #5-case-8> #<TOKEN the/DET #6-det-8>
#<TOKEN hostile/ADJ #7-amod-8> #<TOKEN crowd/NOUN #8-obl-4> #<TOKEN ,/PUNCT #9-punct-4> #<TOKEN killing/VERB #10-advcl-4> #<TOKEN 4/NUM #11-obj-10> #<TOKEN ./PUNCT #12-punct-4>)

(sentence->text
sentence)
=> "The US troops fired into the hostile crowd, killing 4."

(sentence->text
sentence
:special-format-test #’(lambda (token)
(eq (token-upostag token)
"VERB"))
:special-format-function (lambda (string)
(format nil "*~a*" (string-upcase string))))
=> "The US troops *FIRED* into the hostile crowd, *KILLING* 4."

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-binary-tree SENTENCE

Based on the idea from [1], it produces a tree view of the sentence, still need to improve the priorities of children.

Code at https://github.com/sivareddyg/UDepLambda in file src/deplambda/parser/TreeTransformer.java method ’binarizeTree’

[1] Siva Reddy, O. Tackstrom, M. Collins, T. Kwiatkowski, D. Das, M. Steedman, and M. Lapataw, Transforming Dependency Structures to Logical Forms for Semantic Parsing, Transactions of the Association for Computational Linguistics, pp. 127–140, Apr. 2016.

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-equal SENT-1 SENT-2

Tests if, for each slot, sent-1 has the same values as sent-2. For tokens and multiword tokens, it uses token-equal and mtoken-equal, respectively.

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-hash-table SENTENCE

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-id SENTENCE

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-meta-value SENTENCE META-FIELD

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-text SENTENCE

Package

cl-conllu

Source

data.lisp (file)

Function: sentence-valid? SENTENCE

Package

cl-conllu

Source

data.lisp (file)

Function: tree-sentence SENTENCE &key STREAM SHOW-META

Package

conllu.draw

Source

draw.lisp (file)

Function: write-conllu SENTENCES FILENAME &key IF-EXISTS

Package

cl-conllu

Source

read-write.lisp (file)

Function: write-conllu-to-stream SENTENCES &optional OUT

Package

cl-conllu

Source

read-write.lisp (file)

Function: write-sentence-tag-suffix-to-stream SENTENCE &key STREAM TAG SEPARATOR

Writes sentence as CoNLL-U file in STREAM as FORM.SEPARATOR.TAGVALUE (without
dots), followed by a whitespace character.

If TAG is NIL, then writes only FORMs, followed by a whitepsace character.

Example:
;; supposing sentence already defined
(write-sentence-tag-suffix-to-stream (sentence :tag ’xpostag :separator "_")) Pierre_NNP Vinken_NNP ,_, 61_CD years_NNS old_JJ ,_, will_MD join_VB the_DT board_NN as_IN a_DT nonexecutive_JJ director_NN Nov._NNP 29_CD ._.
=> NIL

Package

conllu.converters.tags

Source

tag-converter.lisp (file)

Function: write-sentences-tag-suffix SENTENCES FILENAME &key TAG SEPARATOR IF-EXISTS

See documentation for write-sentence-tag-suffix-to-stream

Package

conllu.converters.tags

Source

tag-converter.lisp (file)

Function: write-sentences-tag-suffix-to-stream SENTENCES &key STREAM TAG SEPARATOR

See documentation for write-sentence-tag-suffix-to-stream

Package

conllu.converters.tags

Source

tag-converter.lisp (file)

5.1.2 Generic functions

Generic Function: mtoken-end OBJECT

Generic Function: (setf mtoken-end) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: mtoken-end (MTOKEN mtoken)

automatically generated reader method

Source

data.lisp (file)

Method: (setf mtoken-end) NEW-VALUE (MTOKEN mtoken)

automatically generated writer method

Source

data.lisp (file)

Generic Function: mtoken-form OBJECT

Generic Function: (setf mtoken-form) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: mtoken-form (MTOKEN mtoken)

automatically generated reader method

Source

data.lisp (file)

Method: (setf mtoken-form) NEW-VALUE (MTOKEN mtoken)

automatically generated writer method

Source

data.lisp (file)

Generic Function: mtoken-start OBJECT

Generic Function: (setf mtoken-start) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: mtoken-start (MTOKEN mtoken)

automatically generated reader method

Source

data.lisp (file)

Method: (setf mtoken-start) NEW-VALUE (MTOKEN mtoken)

automatically generated writer method

Source

data.lisp (file)

Generic Function: sentence-meta OBJECT

Generic Function: (setf sentence-meta) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: sentence-meta (SENTENCE sentence)

automatically generated reader method

Source

data.lisp (file)

Method: (setf sentence-meta) NEW-VALUE (SENTENCE sentence)

automatically generated writer method

Source

data.lisp (file)

Generic Function: sentence-mtokens OBJECT

Generic Function: (setf sentence-mtokens) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: sentence-mtokens (SENTENCE sentence)

automatically generated reader method

Source

data.lisp (file)

Method: (setf sentence-mtokens) NEW-VALUE (SENTENCE sentence)

automatically generated writer method

Source

data.lisp (file)

Generic Function: sentence-start OBJECT

Generic Function: (setf sentence-start) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: sentence-start (SENTENCE sentence)

automatically generated reader method

Source

data.lisp (file)

Method: (setf sentence-start) NEW-VALUE (SENTENCE sentence)

automatically generated writer method

Source

data.lisp (file)

Generic Function: sentence-tokens OBJECT

Generic Function: (setf sentence-tokens) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: sentence-tokens (SENTENCE sentence)

automatically generated reader method

Source

data.lisp (file)

Method: (setf sentence-tokens) NEW-VALUE (SENTENCE sentence)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-deprel OBJECT

Generic Function: (setf token-deprel) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-deprel (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-deprel) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-deps OBJECT

Generic Function: (setf token-deps) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-deps (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-deps) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-feats OBJECT

Generic Function: (setf token-feats) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-feats (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-feats) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-form OBJECT

Generic Function: (setf token-form) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-form (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-form) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-head OBJECT

Generic Function: (setf token-head) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-head (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-head) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-id OBJECT

Generic Function: (setf token-id) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-id (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-id) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-lemma OBJECT

Generic Function: (setf token-lemma) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-lemma (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-lemma) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-misc OBJECT

Generic Function: (setf token-misc) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-misc (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-misc) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-upostag OBJECT

Generic Function: (setf token-upostag) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-upostag (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-upostag) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

Generic Function: token-xpostag OBJECT

Generic Function: (setf token-xpostag) NEW-VALUE OBJECT

Package

cl-conllu

Methods

Method: token-xpostag (TOKEN token)

automatically generated reader method

Source

data.lisp (file)

Method: (setf token-xpostag) NEW-VALUE (TOKEN token)

automatically generated writer method

Source

data.lisp (file)

5.1.3 Classes

Class: mtoken ()

Package

cl-conllu

Source

data.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• mtoken-misc (method)
• mtoken-misc (method)
• mtoken-form (method)
• mtoken-form (method)
• mtoken-end (method)
• mtoken-end (method)
• mtoken-start (method)
• mtoken-start (method)

Direct slots

Slot: start

Initargs

:start

Readers

mtoken-start (generic function)

Writers

(setf mtoken-start) (generic function)

Slot: end

Initargs

:end

Readers

mtoken-end (generic function)

Writers

(setf mtoken-end) (generic function)

Slot: form

Initargs

:form

Readers

mtoken-form (generic function)

Writers

(setf mtoken-form) (generic function)

Slot: misc

Initargs

:misc

Initform

"_"

Readers

mtoken-misc (generic function)

Writers

(setf mtoken-misc) (generic function)

Class: sentence ()

Package

cl-conllu

Source

data.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• encode (method)
• print-object (method)
• sentence-mtokens (method)
• sentence-mtokens (method)
• sentence-tokens (method)
• sentence-tokens (method)
• sentence-meta (method)
• sentence-meta (method)
• sentence-start (method)
• sentence-start (method)

Direct slots

Slot: start

Initargs

:start

Initform

0

Readers

sentence-start (generic function)

Writers

(setf sentence-start) (generic function)

Slot: meta

Initargs

:meta

Readers

sentence-meta (generic function)

Writers

(setf sentence-meta) (generic function)

Slot: tokens

Initargs

:tokens

Readers

sentence-tokens (generic function)

Writers

(setf sentence-tokens) (generic function)

Slot: mtokens

Initargs

:mtokens

Readers

sentence-mtokens (generic function)

Writers

(setf sentence-mtokens) (generic function)

Class: token ()

Package

cl-conllu

Source

data.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• encode (method)
• print-object (method)
• token-misc (method)
• token-misc (method)
• token-deps (method)
• token-deps (method)
• token-deprel (method)
• token-deprel (method)
• token-head (method)
• token-head (method)
• token-feats (method)
• token-feats (method)
• token-xpostag (method)
• token-xpostag (method)
• token-upostag (method)
• token-upostag (method)
• token-lemma (method)
• token-lemma (method)
• token-form (method)
• token-form (method)
• token-id (method)
• token-id (method)

Direct slots

Slot: id

Initargs

:id

Readers

token-id (generic function)

Writers

(setf token-id) (generic function)

Slot: form

Initargs

:form

Readers

token-form (generic function)

Writers

(setf token-form) (generic function)

Slot: lemma

Initargs

:lemma

Readers

token-lemma (generic function)

Writers

(setf token-lemma) (generic function)

Slot: upostag

Initargs

:upostag

Initform

"_"

Readers

token-upostag (generic function)

Writers

(setf token-upostag) (generic function)

Slot: xpostag

Initargs

:xpostag

Initform

"_"

Readers

token-xpostag (generic function)

Writers

(setf token-xpostag) (generic function)

Slot: feats

Initargs

:feats

Initform

"_"

Readers

token-feats (generic function)

Writers

(setf token-feats) (generic function)

Slot: head

Initargs

:head

Initform

"_"

Readers

token-head (generic function)

Writers

(setf token-head) (generic function)

Slot: deprel

Initargs

:deprel

Initform

"_"

Readers

token-deprel (generic function)

Writers

(setf token-deprel) (generic function)

Slot: deps

Initargs

:deps

Initform

"_"

Readers

token-deps (generic function)

Writers

(setf token-deps) (generic function)

Slot: misc

Initargs

:misc

Initform

"_"

Readers

token-misc (generic function)

Writers

(setf token-misc) (generic function)

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *clauses*

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Special Variable: *dependencies*

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Special Variable: *deprel-value-list*

List of the 37 universal syntactic relations in UD.

Package

cl-conllu

Source

evaluate.lisp (file)

Special Variable: *deprels*

Package

cl-conllu

Source

query.lisp (file)

Special Variable: *token-fields*

Package

cl-conllu

Source

data.lisp (file)

Special Variable: niceline-matcher

Package

conllu.converters.niceline

Source

niceline.lisp (file)

5.2.2 Functions

Function: add LINES SPACES ID TEXT

Package

conllu.draw

Source

draw.lisp (file)

Function: add-diff-log P-VAL G-VAL LOG DIFF-TABLE

Recieves a pair of values g-val and p-val, creates the html section for reporting this diff

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: add-field-value TOKEN FIELD VALUE

Package

conllu.converters.niceline

Source

niceline.lisp (file)

Function: add-value FIELD NEW-VALUE TOKEN REST-CONDITIONS

Package

cl-conllu

Source

rules.lisp (file)

Function: adjust-conllu INPUT-FILENAME OUTPUT

Package

cl-conllu

Source

command-line.lisp (file)

Function: allowed-value ALLOWED-VALUES VALUE

Checks if the value is in the allowed list. An empty list signals that any value is allowed.

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: and% QUERY1 QUERY2 &key TKS

Package

cl-conllu

Source

query.lisp (file)

Function: apply-changes TOKENS RULE

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-conditions-in-token CONDITIONS TOKEN

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-rhs BINDINGS RHS

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-rule-in-sentence A-SENTENCE RULE SENT-ID

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-rule-in-tokens TOKENS RULE SENT-ID &optional MATCHS

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-rules-in-sentence A-SENTENCE RULES SENT-ID RECURSIVE &optional OLD-RULES-APPLIED

Package

cl-conllu

Source

rules.lisp (file)

Function: apply-rules-in-sentence-aux A-SENTENCE RULES SENT-ID &optional APPLIED?

Package

cl-conllu

Source

rules.lisp (file)

Function: attachment-score-by-sentence LIST-SENT1 LIST-SENT2 &key LABELED IGNORE-PUNCT SIMPLE-DEP

Attachment score by sentence (macro-average).

The attachment score is the percentage of words that have correct arcs to their heads. The unlabeled attachment score (UAS) considers only who is the head of the token, while the labeled attachment score (LAS) considers both the head and the arc label (dependency label / syntactic class).

In order to choose between labeled or unlabeled,
set the key argument LABELED.

References:
- Dependency Parsing. Kubler, Mcdonald and Nivre (pp.79-80)

Package

cl-conllu

Source

evaluate.lisp (file)

Function: attachment-score-by-word LIST-SENT1 LIST-SENT2 &key LABELED IGNORE-PUNCT SIMPLE-DEP

Attachment score by word (micro-average).

The attachment score is the percentage of words that have correct arcs to their heads. The unlabeled attachment score (UAS) considers only who is the head of the token, while the labeled attachment score (LAS) considers both the head and the arc label (dependency label / syntactic class).

In order to choose between labeled or unlabeled,
set the key argument LABELED.

References:
- Dependency Parsing. Kubler, Mcdonald and Nivre (pp.79-80)

Package

cl-conllu

Source

evaluate.lisp (file)

Function: between LIST BOT TOP

Package

conllu.draw

Source

draw.lisp (file)

Function: children FATHER TOKENS

Package

conllu.draw

Source

draw.lisp (file)

Function: children TKS ID

Package

cl-conllu

Source

query.lisp (file)

Function: clean-dep-rel STR

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Function: clean-whitespace LINE

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Function: collect-meta LINES

Package

cl-conllu

Source

read-write.lisp (file)

Function: collect-slots TOKEN SLOTS ACUM

Package

conllu.draw

Source

draw.lisp (file)

Function: components->string COMPONENTS

Package

conllu.rdf

Source

rdf.lisp (file)

Function: confusion-matrix LIST-SENT1 LIST-SENT2 &key NORMALIZE

Returns a hash table where keys are lists (deprel1 deprel2) and values are fraction of classifications as deprel1 of a word that originally was deprel2.

We assume that LIST-SENT1 is the classified result
and LIST-SENT2 is the list of golden (correct) sentences.

Package

cl-conllu

Source

evaluate.lisp (file)

Function: confusion-table-access-cell ROW COL TABLE

Returns the value of the confusion table defined by the row and col

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: confusion-table-add-column CONFUSION-TABLE NEW-COLUMN

Adds a new column and row to the confusion table.

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: confusion-table-add-pair G-VALUE P-VALUE CONFUSION-TABLE

Increments the respective (row,column) of the confusion-table by 1, If the row or column does not exist, create it.

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: confusion-table-rows TABLE

Returns the rows of the confusion table

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: consume-tags TAGS TK

Package

conllu.converters.niceline

Source

niceline.lisp (file)

Function: convert-directory PATH

Package

conllu.converters.niceline

Source

niceline.lisp (file)

Function: convert-features-to-rdf FEATURES-STRING

Input: string
Output: list of feature nodes

Returns a list of nodes to be used as objects in triples with the predicate "conll:feats".

Examples:
(let ((wilbur:*nodes* (make-instance ’wilbur:dictionary))
(wilbur:*db* (make-instance ’wilbur:db)))
(wilbur:add-namespace "olia-sys" "http://purl.org/olia/system.owl#") (wilbur:add-namespace "conll" "http://br.ibm.com/conll/LEMMA") (convert-features-to-rdf "Mood=Ind|Tense=Past|VerbForm=Fin" (node "sentence1"))) =>
(!"http://br.ibm.com/conll/LEMMA#verbFormFin" !"http://br.ibm.com/conll/LEMMA#tensePast" !"http://br.ibm.com/conll/LEMMA#moodInd")

Package

conllu.rdf

Source

rdf-wilbur.lisp (file)

Function: convert-sentence-metadata METADATA SENTENCE-NODE

Input: list of pairs (name value), node
Output: List of triples

Example:
(let ((wilbur:*db* (make-instance ’wilbur:db))
(metadata ’(("sent_id" . "test")
("text" . "The US troops fired into the hostile crowd, killing 4.")))
(sentence-node (node "sentence")))
(convert-sentence-metadata metadata sentence-node))
=>
((#<WILBUR:TRIPLE !"sentence" !conll:metadata/sent_id #"test" {10048AA2E3}>)
(#<WILBUR:TRIPLE !"sentence" !conll:metadata/text #"The US troops fired into the hostile crowd, killing 4." {10048AA753}>))

Package

conllu.rdf

Source

rdf-wilbur.lisp (file)

Function: convert-sentence-to-rdf SENTENCE TEXT SENTENCE-ID CORPUS-ID-NODE

Package

conllu.rdf

Source

rdf-wilbur.lisp (file)

Function: convert-sentence-to-turtle STREAM CONLL TEXT ID

Package

conllu.rdf

Source

rdf.lisp (file)

Function: convert-token-to-rdf TOKEN SENTENCE-ID SENTENCE-NODE

Package

conllu.rdf

Source

rdf-wilbur.lisp (file)

Function: diff-sentences P-SENTENCE G-SENTENCE EVALUATION-FUNCTION CONFUSION-TABLE DIFF-TABLE &optional G-COLUMNS P-COLUMNS

Evaluates the tokens using the recieved evaluation-function.
Optional columns g-columns/p-columns for golden and predicted values.
The differences found are reported in the diff-table and confusion-table recieved

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: draw-descendent ID LINES SPACES BOT TOP N

Package

conllu.draw

Source

draw.lisp (file)

Function: draw-father ID LINES SPACES BOT TOP NODE N

Package

conllu.draw

Source

draw.lisp (file)

Function: draw-father-lines ID LINES SPACES BOT TOP

Package

conllu.draw

Source

draw.lisp (file)

Function: draw-no-descendets ID LINES SPACES N

Package

conllu.draw

Source

draw.lisp (file)

Function: draw-node ID LINES SPACES NODE

Package

conllu.draw

Source

draw.lisp (file)

Function: draw-node-values ID LINES SPACES NODE

Package

conllu.draw

Source

draw.lisp (file)

Function: emit-prolog CLAUSE TEXT

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Function: equal-op FIELD REGEX REST-PATTERN TOKEN

Package

cl-conllu

Source

rules.lisp (file)

Function: escape-string STRING CHAR-ESCAPE-FUNCTION

Package

conllu.rdf

Source

rdf.lisp (file)

Function: escape-turtle-char CHAR

Package

conllu.rdf

Source

rdf.lisp (file)

Function: eval-query EXPRESSION &key TKS

Package

cl-conllu

Source

query.lisp (file)

Function: exact-match LIST-SENT1 LIST-SENT2 &key COMPARED-FIELDS IDENTITY-FIELDS TEST SIMPLE-DEP IGNORE-PUNCT

Returns the list of sentences of LIST-SENT1 that are an exact match to the corresponding sentence of LIST-SENT2 (same position in list).

LIST-SENT1 and LIST-SENT2 must have the same size with corresponding sentences in order.

Package

cl-conllu

Source

evaluate.lisp (file)

Function: exact-match-score LIST-SENT1 LIST-SENT2 &key COMPARED-FIELDS IDENTITY-FIELDS TEST SIMPLE-DEP IGNORE-PUNCT

Returns the percentage of sentences of LIST-SENT1 that are an exact match to LIST-SENT2.

LIST-SENT1 and LIST-SENT2 must have the same size with corresponding sentences in order.

The typical use case is comparing the result of a tagger (or parser) against a test set, where an exact match is a completely correct tagging (or parse) for the sentence.

References:
- Dependency Parsing. Kubler, Mcdonald and Nivre (p.79)

Package

cl-conllu

Source

evaluate.lisp (file)

Function: exact-match-sentence SENT1 SENT2 &key COMPARED-FIELDS IDENTITY-FIELDS TEST SIMPLE-DEP IGNORE-PUNCT

Compares if two sentences are an exact match.

The typical use case is comparing the result of a tagger (or parser) against a test sentence.

Returns SENT1 if SENT1 and SENT2 agree for all tokens with respect to the COMPARED-FIELDS. Otherwise returns NIL.

If they do not have the same number of tokens or if the tokens do not agree on each IDENTITY-FIELD, then the sentences are not ’the same’ and thus an error is returned.

Package

cl-conllu

Source

evaluate.lisp (file)

Function: find-min LIST &key MIN

Package

cl-conllu

Source

utils.lisp (file)

Function: format-cell-html VAL &optional COLOR

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: format-dependency TOKEN SENT

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: format-file-name P-VALUE G-VALUE

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: format-link-html VAL HREF

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: format-matrix MATRIX

Package

cl-conllu

Source

evaluate.lisp (file)

Function: format-row-html VAL

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: format-sentence-text SENTENCE &optional HIGHLIGHTED-TOKENS

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: format-table-html VAL

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: get-diff-file-path P-VALUE G-VALUE DIFFS-PATH

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: get-token-parent TOKEN SENT

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-dependency-pair P-TOKEN G-TOKEN P-SENTENCE G-SENTENCE

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-diff-log P-VALUE G-VALUE P-TOKEN G-TOKEN P-SENTENCE G-SENTENCE DIFF-TABLE

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-make-header P-VALUE G-VALUE

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-make-info-line TOPIC LINE

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-set-bold TEXT

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-set-font-color TEXT COLOR

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-write-charset STREAM

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: html-write-style-css STREAM

Package

cl-conllu

Source

evaluate-template.lisp (file)

Function: insert-at LST INDEX NEWELT

Package

cl-conllu

Source

utils.lisp (file)

Function: insert-token SENTENCE NEW-TOKEN

Inserts token in a sentence object. It will not be inserted exactly as given: its ID will be the same (place where it’ll be inserted) but its head should point to id value prior to the insertion. It changes the sentence object passed.

Package

cl-conllu

Source

data.lisp (file)

Function: intern-pattern PATTERN

Package

cl-conllu

Source

rules.lisp (file)

Function: intern-rule RULE

Package

cl-conllu

Source

rules.lisp (file)

Function: intern-sides SIDES

Package

cl-conllu

Source

rules.lisp (file)

Function: is-descendant? ID-1 ID-2 SENTENCE &optional ALIST

Package

cl-conllu

Source

data.lisp (file)

Function: is-not-just-a-leaf ID TOKENS

Package

conllu.draw

Source

draw.lisp (file)

Function: is-root STR

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Function: line->mtoken LINE

Package

cl-conllu

Source

read-write.lisp (file)

Function: line->token LINE

Package

cl-conllu

Source

read-write.lisp (file)

Function: list-to-tsv ALIST STREAM

Package

cl-conllu

Source

read-write.lisp (file)

Function: make-branchs FATHER CHILDREN TOKENS LEN-TOKENS LINES SPACES

Package

conllu.draw

Source

draw.lisp (file)

Function: make-dep DEP

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-featurename FEAT

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-features FEATURES &optional VALUE-AS-LITERAL

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-features-bnode FEATURES &optional VALUE-AS-LITERAL

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-id CONTEXT PREFIX ID

Package

conllu.prolog

Source

conllu-prolog.lisp (file)

Function: make-literal STRING

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-metadata METADATA

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-metadata-bnode METADATA

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-root-lines TOKENS ROOT LINES SPACES

Package

conllu.draw

Source

draw.lisp (file)

Function: make-token-id SENTENCE-ID TOKEN-ID

Package

conllu.rdf

Source

rdf.lisp (file)

Function: make-tree SENT

Package

conllu.draw

Source

draw.lisp (file)

Function: make-upos UPOS

Package

conllu.rdf

Source

rdf.lisp (file)

Function: mappend FUNCTION LIST

Receives a function and a list of lists and returns the appended result of the aplication of the function to each list.

Package

cl-conllu

Source

utils.lisp (file)

Function: match-token TOKEN PATTERN

Package

cl-conllu

Source

rules.lisp (file)

Function: match? TOKENS RLS &optional BINDINGS

Package

cl-conllu

Source

rules.lisp (file)

Function: maximum-between LIST BOT TOP

Package

conllu.draw

Source

draw.lisp (file)

Function: member-op FIELD ELMENT REST-PATTERN TOKEN

Package

cl-conllu

Source

rules.lisp (file)

Function: modify-conllu ORIGINAL-FILE CHANGES-FILE OUTPUT &optional ADD-NEWS

The original file contains a set of sentences. The modified file contains some sentences from original modified, this function replaces in original the sentences presented in modified file, matching them using the sentence ids. If the modified file contains sentence not in original, the flag ’add-new’ , if true, says that these sentence must be added in the end of the original file.

Package

cl-conllu

Source

command-line.lisp (file)

Function: modify-value FIELD NEW-VALUE TOKEN REST-CONDITIONS

Package

cl-conllu

Source

rules.lisp (file)

Function: mtoken->tokens SENTENCE MTOKEN

Package

cl-conllu

Source

data.lisp (file)

Function: mtoken-equal MTOKEN-1 MTOKEN-2

Tests if, for each slot, mtoken-1 has the same values as mtoken-2.

Package

cl-conllu

Source

data.lisp (file)

Function: niceline G4 &key START END END-TOKEN END-VALUE &aux G5 G0 G1 G2 G3

Package

conllu.converters.niceline

Source

niceline.lisp (file)

Function: non-projectivity-accuracy LIST-SENT1 LIST-SENT2

Package

cl-conllu

Source

evaluate.lisp (file)

Function: non-projectivity-precision LIST-SENT1 LIST-SENT2

Package

cl-conllu

Source

evaluate.lisp (file)

Function: non-projectivity-recall LIST-SENT1 LIST-SENT2

Package

cl-conllu

Source

evaluate.lisp (file)

Function: or% QUERY1 QUERY2 &key TKS

Package

cl-conllu

Source

query.lisp (file)

Function: precision LIST-SENT1 LIST-SENT2 DEPREL &key ERROR-TYPE SIMPLE-DEP

Restricted to words which are classified as of syntactical class (dependency type to head) DEPREL, returns the precision:
the number of true positives divided by the number of words predicted positive (that is, predicted as of class DEPREL).

We assume that LIST-SENT1 is the classified (predicted) result and LIST-SENT2 is the list of golden (correct) sentences.

ERROR-TYPE defines what is considered an error (a false negative). Some usual values are:
- ’(deprel) :: for the deprel tagging task only
- ’(head) :: for considering errors for each syntactic class - ’(deprel head) :: for considering correct only when both deprel and head are correct.

Package

cl-conllu

Source

evaluate.lisp (file)