This is the csv-validator Reference Manual, version 1.0.0, generated automatically by Declt version 4.0 beta 2 "William Riker" on Mon Feb 26 16:10:16 2024 GMT+0.
csv-validator/csv-validator.asdcsv-validator/src/packages.lispcsv-validator/src/main.lispcsv-validator/src/parallel.lispcsv-validator/src/output.lispcsv-validator/src/csv.lispcsv-validator/src/validator.lispcsv-validator/src/validation_utils.lispThe main system appears first, followed by any subsystem dependency.
csv-validatorValidates tabular CSV data using predefined validations, similar to its Python counterpart ’Great Expectations’.
Koen van den Berg <k.vandenberg@insertdata.nl>
(GIT https://github.com/KoenvdBerg/csv-validator)
BSD-3
1.0.0
local-time (system).
lparallel (system).
parse-float (system).
src (module).
Modules are listed depth-first from the system components tree.
csv-validator/srccsv-validator (system).
packages.lisp (file).
main.lisp (file).
parallel.lisp (file).
output.lisp (file).
csv.lisp (file).
validator.lisp (file).
validation_utils.lisp (file).
Files are sorted by type and then listed depth-first from the systems components trees.
csv-validator/csv-validator.asdcsv-validator/src/packages.lispcsv-validator/src/main.lispcsv-validator/src/parallel.lispcsv-validator/src/output.lispcsv-validator/src/csv.lispcsv-validator/src/validator.lispcsv-validator/src/validation_utils.lispcsv-validator/src/main.lisppackages.lisp (file).
src (module).
validate-csv (function).
csv-validator/src/parallel.lispcsv-validator/src/output.lispparallel.lisp (file).
src (module).
add-index-to-file (function).
cleanup-outdir (function).
join-files-together (function).
list-dir (function).
move-working-dir (function).
split-file-in-n (function).
write-header-validation-file (function).
write-record-to-stream (function).
write-result-file (function).
write-result-header-file (function).
csv-validator/src/csv.lispoutput.lisp (file).
src (module).
record (type).
split-string (function).
to-record (function).
csv-validator/src/validator.lispcsv.lisp (file).
src (module).
get-col-position (function).
get-header-row (function).
get-values (function).
run-record-validation (function).
validate-header (function).
validate-record (function).
csv-validator/src/validation_utils.lispvalidator.lisp (file).
src (module).
check-compare-two-dates (function).
check-date-before-today (function).
check-date-parsable (function).
check-float-string (function).
check-integer-in-range (function).
check-integer-string (function).
check-not-null (function).
check-null (function).
check-number-in-range (function).
check-number-string (function).
check-scientific-number-string (function).
check-tz-parsable (function).
Packages are listed by definition order.
csv-validatorcommon-lisp.
check-compare-two-dates (function).
check-date-before-today (function).
check-date-parsable (function).
check-float-string (function).
check-integer-in-range (function).
check-integer-string (function).
check-not-null (function).
check-null (function).
check-number-in-range (function).
check-number-string (function).
check-scientific-number-string (function).
check-tz-parsable (function).
validate-csv (function).
add-index-to-file (function).
cleanup-outdir (function).
get-col-position (function).
get-header-row (function).
get-values (function).
init (function).
join-files-together (function).
list-dir (function).
move-working-dir (function).
record (type).
run-record-validation (function).
shutdown (function).
split-file-in-n (function).
split-string (function).
to-record (function).
validate-header (function).
validate-record (function).
write-header-validation-file (function).
write-record-to-stream (function).
write-result-file (function).
write-result-header-file (function).
Definitions are sorted by export status, category, package, and then by lexicographic order.
Checks if the incoming date string x is before y
returns
—-
nil if x > y
t if x < y or either x or y not date parsable
Checks if the incoming string is a date that is before today
returns
—-
nil if x is parsable and not before today
t if x ain’t parsable or before today
Checks if the incoming string is parsable as a date.
returns
—-
date-object if parsable, else nil
Checks if the incoming string is a float
returns
—-
t if float, else nil
Checks if the incoming string is an integer between range-start and
range-end. If no range-start, checks if x < range-end. If no
range-end, checks x > range-start.
returns
—-
nil if x outside of range
t if x isn’t integer or within range
Checks if the incoming string is an integer
returns
—-
t if integer, else nil
Checks if the incoming string is not null or null-like
Checks if the incoming string can be considered null or null-like
returns
—-
t if null, else nil
Checks if the incoming string is a number between range-start and
range-end. If no range-start, checks if x < range-end. If no
range-end, checks x > range-start.
returns
—-
nil if x outside of range
t if x isn’t integer or within range
Checks in the incoming string is either an integer or a float
returns
—-
t if number, else nil
Checks if the incoming string is a scientific number
returns
—-
t if float, else nil
Checks if the incoming string is parsable as a timezone date.
returns
—-
date-object if parsable, else nil
Main function that is used to apply the csv-validator to a dataset
args
—-
in: string, filepath to the input csv file
outdir: string, filepath to the output directory where the results will be stored
validation-suite: the suite to use to validate the input csv to
threads: integer, indicates the amount of threads to use. Best is between 1-4
delim: character, the csv delimiter
returns
—-
nil
Adds an index column to the infile at column position 0 with correct delimiter (first column)
Cleans up the outdir
Obtains the position of a column name in a list of columns
args
—-
col-name: string, describes the column name
cols: list(string), list of column names to find the position in
returns
—-
int column position
Obtains the header row from the infile
args
—-
infile: string, filepath to the input csv file
delim: character, the csv delimiter
returns
—-
header row as a list. E.g. ’("header1" "header2")
Obtains the values from a record based on the depends field in the
validation suite
args
—-
header: list(string), list that contains the headers as strings
spec: a single specification of a validation suite as a list
record: list(string), list that holds the values of that record. Every value is string.
returns
—-
list of values
Initializes the lparallel kernel
Appends multiple csv files together vertically to new file called errors.csv
List files in specified outdir that have wildcard prefix
Move the workind directory to target-dir
Main function for validating a single csv infile in 1 thread
args
—-
in: string, filepath to the input csv file
outdir: string, filepath to the output directory where the results will be stored
suite: the suite to use to validate the input csv to
delim: character, the csv delimiter
returns
—-
nil
Shuts down the kernel
Splits the tabular infile in N=nr-files splits that can be processed individually
Splits a string using the given separator, returns a list with the substrings.
args
—-
separator: character, char that is used to split the input csv with
str: string, the input string that will be split
returns
—-
list with elements between delimiter
example
—-
name;age;gender –> ("name" "age" "gender")
Converts a sequence of elements into a record.
Validates the header against the expectation suite. Filters the
expectation suite so that only the specifications remain that are
also present in the header.
args
—-
header: list(string), list that contains the headers as strings
suite: the suite to use to validate the input csv to
returns
—-
suite but then filtered so that it contains only the specs that are
also in header
Validates record-values based on the logic defined in a single
specification of a validation suite
args
—-
vals: list(string), list of values. Every value is string
spec: a single specification of a validation suite as a list
returns
—-
t if logic holds, else nil
writes the validation file that contains the missing headers. A
missing header is a column that was defined in the validation-suite,
but couldn’t be found back in the input csv
args
—-
missing-header: list, contains each spec of the validation
suite that couldn’t be found back in the input csv file
outdir: string, filepath to the output directory where the results
will be stored
returns
—-
string, filepath to the header validation file
Writes the result of a record-validation to an output file
args
—-
str: stream, output stream
index: string, the index (row number) of the input csv
spec: list, a single specification of a validation suite as a list
vals: list(string), the values that are relevant for this result
returns
—-
nil
Writes individual validation result files and combines them together.
args
—-
outdir: string, filepath to the output directory where the results
will be stored
missing-header: a list that contains each spec of the validation
returns
—-
string, filepath to the final result file that contains all validations
Writes the header row of the validation result file. This is being done here so that the csv-validator can be run in parallel without having to keep the csv header of each output into account.
args
—-
outdir: string, filepath to the output directory where the results
will be stored
returns
—-
string, filepath to the file that contains the header row
| Jump to: | A C F G I J L M R S T V W |
|---|
| Jump to: | A C F G I J L M R S T V W |
|---|
| Jump to: | C F M O P R S T V |
|---|
| Jump to: | C F M O P R S T V |
|---|