The pgloader Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The pgloader Reference Manual

This is the pgloader Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Fri Jun 26 11:57:15 2020 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

PGLoader

Build Status Join the chat at https://gitter.im/dimitri/pgloader Docker Build Status Read The Docs Status

pgloader is a data loading tool for PostgreSQL, using the COPY command.

Its main advantage over just using COPY or \copy, and over using a Foreign Data Wrapper, is its transaction behaviour, where pgloader will keep a separate file of rejected data, but continue trying to copy good data in your database.

The default PostgreSQL behaviour is transactional, which means that any erroneous line in the input data (file or remote database) will stop the entire bulk load for the table.

pgloader also implements data reformatting, a typical example of that being the transformation of MySQL datestamps 0000-00-00 and 0000-00-00 00:00:00 to PostgreSQL NULL value (because our calendar never had a year zero).

Versioning

pgloader version 1.x is quite old and was developed in TCL. When faced with maintaining that code, the new emerging development team (hi!) picked python instead because that made sense at the time. So pgloader version 2.x was written in python.

The current version of pgloader is the 3.x series, which is written in Common Lisp for better development flexibility, runtime performance, and support of real threading.

The versioning is now following the Emacs model, where any X.0 release number means you're using a development version (alpha, beta, or release candidate). The next stable versions are going to be 3.1 then 3.2 etc.

When using a development snapshot rather than a released version the version number includes the git hash (in its abbreviated form):

LICENCE

pgloader is available under The PostgreSQL Licence.

INSTALL

You can install pgloader directly from apt.postgresql.org and from official debian repositories, see packages.debian.org/pgloader.

$ apt-get install pgloader

You can also use a docker image for pgloader at https://hub.docker.com/r/dimitri/pgloader/:

$ docker pull dimitri/pgloader
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --version
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --help

Build from sources

pgloader is now a Common Lisp program, tested using the SBCL (>= 1.2.5) and Clozure CL implementations with Quicklisp.

When building from sources, you should always build from the current git HEAD as it's basically the only source that is managed in a way to ensure it builds aginst current set of dependencies versions.

Building from sources on debian

$ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev
$ cd /path/to/pgloader
$ make pgloader
$ ./build/bin/pgloader --help

Building from sources on RedHat/CentOS

See "Redhat / CentOS" in INSTALL.md

Building from sources on macOS

When using brew, it should be a simple brew install --HEAD pgloader.

When using macports, then we have a situation to deal with with shared objects pgloader depends on, as reported in issue #161 at https://github.com/dimitri/pgloader/issues/161#issuecomment-201162647:

I was able to get a clean build without having to disable compression after symlinking /usr/local/lib to /opt/local/lib. Note that I did not have anything installed to /usr/local/lib so I didn't lose anything here.

Building from sources on Windows

Building pgloader on Windows is supported, thanks to Common Lisp implementations being available on that platform, and to the Common Lisp Standard for making it easy to write actually portable code.

It is recommended to have a look at the issues labelled with Windows support if you run into trouble when building pgloader:

https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows%20support%22%20

Building Docker image from sources

You can build a Docker image from source using SBCL by default:

$ docker build .

Or Clozure CL (CCL):

$ docker build -f Dockerfile.ccl .

More options when building from source

The Makefile target pgloader knows how to produce a Self Contained Binary file for pgloader, found at ./build/bin/pgloader:

$ make pgloader

By default, the Makefile uses SBCL to compile your binary image, though it's possible to build using CCL.

$ make CL=ccl pgloader

If using SBCL and it supports core compression, the make process will use it to generate a smaller binary. To force disabling core compression, you may use:

$ make COMPRESS_CORE=no pgloader

The --compress-core is unique to SBCL, so not used when CC is different from the sbcl value.

You can also tweak the default amount of memory that the pgloader image will allow itself using when running through your data (don't ask for more than your current RAM tho):

$ make DYNSIZE=8192 pgloader

The make pgloader command when successful outputs a ./build/bin/pgloader file for you to use.

Usage

You can either give a command file to pgloader or run it all from the command line, see the pgloader quick start on https://pgloader.readthedocs.io for more details.

$ ./build/bin/pgloader --help
$ ./build/bin/pgloader <file.load>

For example, for a full migration from SQLite:

$ createdb newdb
$ pgloader ./test/sqlite/sqlite.db postgresql:///newdb

Or for a full migration from MySQL, including schema definition (tables, indexes, foreign keys, comments) and parallel loading of the corrected data:

$ createdb pagila
$ pgloader mysql://user@localhost/sakila postgresql:///pagil

Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 pgloader

Author

Dimitri Fontaine <dim@tapoueh.org>

License

The PostgreSQL Licence

Description

Load data into PostgreSQL

Dependencies
Source

pgloader.asd (file)

Component

src (module)


Next: , Previous: , Up: Top   [Contents][Index]

3 Modules

Modules are listed depth-first from the system components tree.


Next: , Previous: , Up: Modules   [Contents][Index]

3.1 pgloader/src

Parent

pgloader (system)

Location

src/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.2 pgloader/src/monkey

Parent

src (module)

Location

src/monkey/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.3 pgloader/src/utils

Dependencies
Parent

src (module)

Location

src/utils/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.4 pgloader/src/pgsql

Dependencies
Parent

src (module)

Location

src/pgsql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.5 pgloader/src/sources

Dependencies
Parent

src (module)

Location

src/sources/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.6 pgloader/src/sources/common

Parent

sources (module)

Location

src/sources/common/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.7 pgloader/src/sources/csv

Dependency

common (module)

Parent

sources (module)

Location

src/sources/csv/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.8 pgloader/src/sources/fixed

Dependency

common (module)

Parent

sources (module)

Location

src/sources/fixed/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.9 pgloader/src/sources/db3

Dependencies
Parent

sources (module)

Location

src/sources/db3/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.10 pgloader/src/sources/ixf

Dependency

common (module)

Parent

sources (module)

Location

src/sources/ixf/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.11 pgloader/src/sources/sqlite

Dependency

common (module)

Parent

sources (module)

Location

src/sources/sqlite/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.12 pgloader/src/sources/mssql

Dependency

common (module)

Parent

sources (module)

Location

src/sources/mssql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.13 pgloader/src/sources/mysql

Dependency

common (module)

Parent

sources (module)

Location

src/sources/mysql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.14 pgloader/src/sources/pgsql

Dependency

common (module)

Parent

sources (module)

Location

src/sources/pgsql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.15 pgloader/src/pg-copy

Dependencies
Parent

src (module)

Location

src/pg-copy/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.16 pgloader/src/load

Dependencies
Parent

src (module)

Location

src/load/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.17 pgloader/src/parsers

Dependencies
Parent

src (module)

Location

src/parsers/

Components

Previous: , Up: Modules   [Contents][Index]

3.18 pgloader/src/regress

Dependencies
Parent

src (module)

Location

src/regress/

Component

regress.lisp (file)


Next: , Previous: , Up: Top   [Contents][Index]

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

4.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.1 pgloader.asd

Location

pgloader.asd

Systems

pgloader (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.2 pgloader/src/params.lisp

Parent

src (module)

Location

src/params.lisp

Packages

pgloader.params

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.3 pgloader/src/package.lisp

Dependency

params.lisp (file)

Parent

src (module)

Location

src/package.lisp

Packages

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.4 pgloader/src/monkey/bind.lisp

Parent

monkey (module)

Location

src/monkey/bind.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.5 pgloader/src/monkey/mssql.lisp

Parent

monkey (module)

Location

src/monkey/mssql.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.6 pgloader/src/utils/charsets.lisp

Parent

utils (module)

Location

src/utils/charsets.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.7 pgloader/src/utils/logs.lisp

Parent

utils (module)

Location

src/utils/logs.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.8 pgloader/src/utils/utils.lisp

Parent

utils (module)

Location

src/utils/utils.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.9 pgloader/src/utils/state.lisp

Parent

utils (module)

Location

src/utils/state.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.10 pgloader/src/utils/transforms.lisp

Parent

utils (module)

Location

src/utils/transforms.lisp

Packages

pgloader.user-symbols

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.11 pgloader/src/utils/read-sql-files.lisp

Parent

utils (module)

Location

src/utils/read-sql-files.lisp

Exported Definitions

read-queries (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.12 pgloader/src/utils/queries.lisp

Parent

utils (module)

Location

src/utils/queries.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.13 pgloader/src/utils/quoting.lisp

Dependency

utils.lisp (file)

Parent

utils (module)

Location

src/utils/quoting.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.14 pgloader/src/utils/catalog.lisp

Dependency

quoting.lisp (file)

Parent

utils (module)

Location

src/utils/catalog.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.15 pgloader/src/utils/alter-table.lisp

Dependency

catalog.lisp (file)

Parent

utils (module)

Location

src/utils/alter-table.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.16 pgloader/src/utils/citus.lisp

Dependency

catalog.lisp (file)

Parent

utils (module)

Location

src/utils/citus.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.17 pgloader/src/utils/reject.lisp

Dependency

state.lisp (file)

Parent

utils (module)

Location

src/utils/reject.lisp

Internal Definitions

%process-bad-row (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.18 pgloader/src/utils/pretty-print-state.lisp

Dependency

state.lisp (file)

Parent

utils (module)

Location

src/utils/pretty-print-state.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.19 pgloader/src/utils/report.lisp

Dependencies
Parent

utils (module)

Location

src/utils/report.lisp

Exported Definitions

report-full-summary (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.20 pgloader/src/utils/monitor.lisp

Dependencies
Parent

utils (module)

Location

src/utils/monitor.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.21 pgloader/src/utils/threads.lisp

Dependency

monitor.lisp (file)

Parent

utils (module)

Location

src/utils/threads.lisp

Exported Definitions

make-kernel (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.22 pgloader/src/utils/archive.lisp

Dependency

monitor.lisp (file)

Parent

utils (module)

Location

src/utils/archive.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.23 pgloader/src/utils/connection.lisp

Dependencies
Parent

utils (module)

Location

src/utils/connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.24 pgloader/src/pgsql/connection.lisp

Parent

pgsql (module)

Location

src/pgsql/connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.25 pgloader/src/pgsql/pgsql-ddl.lisp

Dependency

connection.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-ddl.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.26 pgloader/src/pgsql/pgsql-ddl-citus.lisp

Dependency

pgsql-ddl.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-ddl-citus.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.27 pgloader/src/pgsql/pgsql-schema.lisp

Dependency

pgsql-ddl-citus.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.28 pgloader/src/pgsql/merge-catalogs.lisp

Dependencies
Parent

pgsql (module)

Location

src/pgsql/merge-catalogs.lisp

Exported Definitions

merge-catalogs (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.29 pgloader/src/pgsql/pgsql-trigger.lisp

Dependency

merge-catalogs.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-trigger.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.30 pgloader/src/pgsql/pgsql-index-filter.lisp

Dependency

pgsql-trigger.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-index-filter.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.31 pgloader/src/pgsql/pgsql-finalize-catalogs.lisp

Dependency

pgsql-index-filter.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-finalize-catalogs.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.32 pgloader/src/pgsql/pgsql-create-schema.lisp

Dependencies
Parent

pgsql (module)

Location

src/pgsql/pgsql-create-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.33 pgloader/src/sources/common/api.lisp

Parent

common (module)

Location

src/sources/common/api.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.34 pgloader/src/sources/common/methods.lisp

Dependency

api.lisp (file)

Parent

common (module)

Location

src/sources/common/methods.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.35 pgloader/src/sources/common/md-methods.lisp

Dependency

methods.lisp (file)

Parent

common (module)

Location

src/sources/common/md-methods.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.36 pgloader/src/sources/common/matviews.lisp

Dependency

md-methods.lisp (file)

Parent

common (module)

Location

src/sources/common/matviews.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.37 pgloader/src/sources/common/casting-rules.lisp

Dependency

matviews.lisp (file)

Parent

common (module)

Location

src/sources/common/casting-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.38 pgloader/src/sources/common/files-and-pathnames.lisp

Dependency

casting-rules.lisp (file)

Parent

common (module)

Location

src/sources/common/files-and-pathnames.lisp

Exported Definitions
Internal Definitions

filter-directory (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.39 pgloader/src/sources/common/project-fields.lisp

Dependency

files-and-pathnames.lisp (file)

Parent

common (module)

Location

src/sources/common/project-fields.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.40 pgloader/src/sources/csv/csv-guess.lisp

Parent

csv (module)

Location

src/sources/csv/csv-guess.lisp

Exported Definitions

guess-csv-params (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.41 pgloader/src/sources/csv/csv.lisp

Parent

csv (module)

Location

src/sources/csv/csv.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.42 pgloader/src/sources/fixed/fixed-guess.lisp

Parent

fixed (module)

Location

src/sources/fixed/fixed-guess.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.43 pgloader/src/sources/fixed/fixed.lisp

Dependency

fixed-guess.lisp (file)

Parent

fixed (module)

Location

src/sources/fixed/fixed.lisp

Exported Definitions
Internal Definitions

parse-row (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.44 pgloader/src/sources/copy.lisp

Dependencies
Parent

sources (module)

Location

src/sources/copy.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.45 pgloader/src/sources/db3/db3-cast-rules.lisp

Parent

db3 (module)

Location

src/sources/db3/db3-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.46 pgloader/src/sources/db3/db3-connection.lisp

Dependency

db3-cast-rules.lisp (file)

Parent

db3 (module)

Location

src/sources/db3/db3-connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.47 pgloader/src/sources/db3/db3-schema.lisp

Dependency

db3-connection.lisp (file)

Parent

db3 (module)

Location

src/sources/db3/db3-schema.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.48 pgloader/src/sources/db3/db3.lisp

Dependency

db3-schema.lisp (file)

Parent

db3 (module)

Location

src/sources/db3/db3.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.49 pgloader/src/sources/ixf/ixf-cast-rules.lisp

Parent

ixf (module)

Location

src/sources/ixf/ixf-cast-rules.lisp

Exported Definitions

cast (method)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.50 pgloader/src/sources/ixf/ixf-connection.lisp

Dependency

ixf-cast-rules.lisp (file)

Parent

ixf (module)

Location

src/sources/ixf/ixf-connection.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.51 pgloader/src/sources/ixf/ixf-schema.lisp

Dependency

ixf-connection.lisp (file)

Parent

ixf (module)

Location

src/sources/ixf/ixf-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.52 pgloader/src/sources/ixf/ixf.lisp

Dependencies
Parent

ixf (module)

Location

src/sources/ixf/ixf.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.53 pgloader/src/sources/sqlite/sqlite-cast-rules.lisp

Parent

sqlite (module)

Location

src/sources/sqlite/sqlite-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.54 pgloader/src/sources/sqlite/sqlite-connection.lisp

Dependency

sqlite-cast-rules.lisp (file)

Parent

sqlite (module)

Location

src/sources/sqlite/sqlite-connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.55 pgloader/src/sources/sqlite/sqlite-schema.lisp

Dependency

sqlite-connection.lisp (file)

Parent

sqlite (module)

Location

src/sources/sqlite/sqlite-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.56 pgloader/src/sources/sqlite/sqlite.lisp

Dependency

sqlite-schema.lisp (file)

Parent

sqlite (module)

Location

src/sources/sqlite/sqlite.lisp

Exported Definitions
Internal Definitions

parse-value (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.57 pgloader/src/sources/mssql/mssql-cast-rules.lisp

Parent

mssql (module)

Location

src/sources/mssql/mssql-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.58 pgloader/src/sources/mssql/mssql-connection.lisp

Dependency

mssql-cast-rules.lisp (file)

Parent

mssql (module)

Location

src/sources/mssql/mssql-connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.59 pgloader/src/sources/mssql/mssql-schema.lisp

Dependency

mssql-connection.lisp (file)

Parent

mssql (module)

Location

src/sources/mssql/mssql-schema.lisp

Exported Definitions
Internal Definitions

*table-type* (special variable)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.60 pgloader/src/sources/mssql/mssql.lisp

Dependency

mssql-schema.lisp (file)

Parent

mssql (module)

Location

src/sources/mssql/mssql.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.61 pgloader/src/sources/mssql/mssql-index-filters.lisp

Dependency

mssql.lisp (file)

Parent

mssql (module)

Location

src/sources/mssql/mssql-index-filters.lisp

Exported Definitions

translate-index-filter (method)

Internal Definitions

parse-index-filter-clause (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.62 pgloader/src/sources/mysql/mysql-cast-rules.lisp

Parent

mysql (module)

Location

src/sources/mysql/mysql-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.63 pgloader/src/sources/mysql/mysql-connection.lisp

Dependency

mysql-cast-rules.lisp (file)

Parent

mysql (module)

Location

src/sources/mysql/mysql-connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.64 pgloader/src/sources/mysql/mysql-schema.lisp

Dependency

mysql-connection.lisp (file)

Parent

mysql (module)

Location

src/sources/mysql/mysql-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.65 pgloader/src/sources/mysql/mysql.lisp

Dependency

mysql-schema.lisp (file)

Parent

mysql (module)

Location

src/sources/mysql/mysql.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.66 pgloader/src/sources/pgsql/pgsql-cast-rules.lisp

Parent

pgsql (module)

Location

src/sources/pgsql/pgsql-cast-rules.lisp

Exported Definitions
Internal Definitions

pgsql-column-ctype (method)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.67 pgloader/src/sources/pgsql/pgsql.lisp

Dependency

pgsql-cast-rules.lisp (file)

Parent

pgsql (module)

Location

src/sources/pgsql/pgsql.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.68 pgloader/src/pg-copy/copy-batch.lisp

Parent

pg-copy (module)

Location

src/pg-copy/copy-batch.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.69 pgloader/src/pg-copy/copy-format.lisp

Dependency

copy-batch.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-format.lisp

Exported Definitions

format-vector-row (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.70 pgloader/src/pg-copy/copy-db-write.lisp

Dependency

copy-format.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-db-write.lisp

Exported Definitions

copy-init-error (condition)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.71 pgloader/src/pg-copy/copy-rows-in-stream.lisp

Dependency

copy-db-write.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-rows-in-stream.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.72 pgloader/src/pg-copy/copy-rows-in-batch.lisp

Dependency

copy-rows-in-stream.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-rows-in-batch.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.73 pgloader/src/pg-copy/copy-rows-in-batch-through-s3.lisp

Dependency

copy-rows-in-batch.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-rows-in-batch-through-s3.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.74 pgloader/src/pg-copy/copy-retry-batch.lisp

Dependency

copy-rows-in-batch-through-s3.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-retry-batch.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.75 pgloader/src/pg-copy/copy-from-queue.lisp

Dependency

copy-retry-batch.lisp (file)

Parent

pg-copy (module)

Location

src/pg-copy/copy-from-queue.lisp

Exported Definitions

copy-rows-from-queue (function)

Internal Definitions

prepare-copy-parameters (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.76 pgloader/src/load/api.lisp

Parent

load (module)

Location

src/load/api.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.77 pgloader/src/load/copy-data.lisp

Dependency

api.lisp (file)

Parent

load (module)

Location

src/load/copy-data.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.78 pgloader/src/load/load-file.lisp

Dependency

copy-data.lisp (file)

Parent

load (module)

Location

src/load/load-file.lisp

Exported Definitions

copy-database (method)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.79 pgloader/src/load/migrate-database.lisp

Dependency

load-file.lisp (file)

Parent

load (module)

Location

src/load/migrate-database.lisp

Exported Definitions
Internal Definitions

process-catalog (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.80 pgloader/src/parsers/parse-ini.lisp

Parent

parsers (module)

Location

src/parsers/parse-ini.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.81 pgloader/src/parsers/template.lisp

Dependency

parse-ini.lisp (file)

Parent

parsers (module)

Location

src/parsers/template.lisp

Exported Definitions

initialize-context (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.82 pgloader/src/parsers/command-utils.lisp

Dependency

template.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-utils.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.83 pgloader/src/parsers/command-keywords.lisp

Dependency

command-utils.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-keywords.lisp

Internal Definitions

def-keyword-rule (macro)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.84 pgloader/src/parsers/command-regexp.lisp

Dependency

command-keywords.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-regexp.lisp

Internal Definitions

process-quoted-regex (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.85 pgloader/src/parsers/parse-pgpass.lisp

Dependency

command-regexp.lisp (file)

Parent

parsers (module)

Location

src/parsers/parse-pgpass.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.86 pgloader/src/parsers/command-db-uri.lisp

Dependency

parse-pgpass.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-db-uri.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.87 pgloader/src/parsers/command-source.lisp

Dependency

command-db-uri.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-source.lisp

Internal Definitions

filename-character-p (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.88 pgloader/src/parsers/command-options.lisp

Dependency

command-source.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-options.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.89 pgloader/src/parsers/command-sql-block.lisp

Dependency

command-options.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-sql-block.lisp

Exported Definitions

execute-sql-code-block (function)

Internal Definitions

sql-code-block (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.90 pgloader/src/parsers/command-sexp.lisp

Dependency

command-sql-block.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-sexp.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.91 pgloader/src/parsers/command-csv.lisp

Dependency

command-sexp.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-csv.lisp

Exported Definitions

lisp-code-for-loading-from-csv (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.92 pgloader/src/parsers/command-ixf.lisp

Dependency

command-csv.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-ixf.lisp

Exported Definitions

lisp-code-for-loading-from-ixf (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.93 pgloader/src/parsers/command-fixed.lisp

Dependency

command-ixf.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-fixed.lisp

Exported Definitions

lisp-code-for-loading-from-fixed (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.94 pgloader/src/parsers/command-copy.lisp

Dependency

command-fixed.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-copy.lisp

Exported Definitions

lisp-code-for-loading-from-copy (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.95 pgloader/src/parsers/command-dbf.lisp

Dependency

command-copy.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-dbf.lisp

Exported Definitions

lisp-code-for-loading-from-dbf (function)

Internal Definitions

lisp-code-for-dbf-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.96 pgloader/src/parsers/command-cast-rules.lisp

Dependency

command-dbf.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-cast-rules.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.97 pgloader/src/parsers/command-materialize-views.lisp

Dependency

command-cast-rules.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-materialize-views.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.98 pgloader/src/parsers/command-alter-table.lisp

Dependency

command-materialize-views.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-alter-table.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.99 pgloader/src/parsers/command-distribute.lisp

Dependency

command-alter-table.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-distribute.lisp

Internal Definitions

create-table-from-dsn-table-name (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.100 pgloader/src/parsers/command-mysql.lisp

Dependency

command-distribute.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-mysql.lisp

Exported Definitions

lisp-code-for-loading-from-mysql (function)

Internal Definitions

lisp-code-for-mysql-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.101 pgloader/src/parsers/command-including-like.lisp

Dependency

command-mysql.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-including-like.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.102 pgloader/src/parsers/command-mssql.lisp

Dependency

command-including-like.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-mssql.lisp

Exported Definitions

lisp-code-for-loading-from-mssql (function)

Internal Definitions

lisp-code-for-mssql-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.103 pgloader/src/parsers/command-sqlite.lisp

Dependency

command-mssql.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-sqlite.lisp

Exported Definitions

lisp-code-for-loading-from-sqlite (function)

Internal Definitions

lisp-code-for-sqlite-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.104 pgloader/src/parsers/command-pgsql.lisp

Dependency

command-sqlite.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-pgsql.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.105 pgloader/src/parsers/command-archive.lisp

Dependency

command-pgsql.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-archive.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.106 pgloader/src/parsers/command-parser.lisp

Dependency

command-archive.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-parser.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.107 pgloader/src/parsers/parse-sqlite-type-name.lisp

Dependency

command-parser.lisp (file)

Parent

parsers (module)

Location

src/parsers/parse-sqlite-type-name.lisp

Exported Definitions

parse-sqlite-type-name (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.108 pgloader/src/parsers/date-format.lisp

Dependency

parse-sqlite-type-name.lisp (file)

Parent

parsers (module)

Location

src/parsers/date-format.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.109 pgloader/src/api.lisp

Dependencies
Parent

src (module)

Location

src/api.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.110 pgloader/src/regress/regress.lisp

Parent

regress (module)

Location

src/regress/regress.lisp

Internal Definitions

Previous: , Up: Lisp files   [Contents][Index]

4.1.111 pgloader/src/main.lisp

Dependencies
Parent

src (module)

Location

src/main.lisp

Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Packages

Packages are listed by definition order.


Next: , Previous: , Up: Packages   [Contents][Index]

5.1 pgloader.params

Source

params.lisp (file)

Use List

common-lisp

Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.2 pgloader.parse-date

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.3 pgloader.source.copy

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.4 pgloader.citus

Source

package.lisp (file)

Use List
Used By List

pgloader.utils

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.5 pgloader

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.6 pgloader.pgsql

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.7 pgloader.parser

Source

package.lisp (file)

Use List
Used By List

pgloader

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.8 pgloader.source.fixed

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.9 pgloader.ini

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.10 pgloader.source.mysql

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.11 pgloader.utils

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.12 pgloader.monitor

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.13 pgloader.syslog

Source

package.lisp (file)

Use List

Next: , Previous: , Up: Packages   [Contents][Index]

5.14 pgloader.transforms

Source

package.lisp (file)

Use List

common-lisp

Used By List

pgloader.pgsql

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.15 pgloader.source.sqlite

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.16 pgloader.connection

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.17 pgloader.source.mssql.index-filter

Source

package.lisp (file)

Use List
Internal Definitions

parse-index-filter-clause (function)


Next: , Previous: , Up: Packages   [Contents][Index]

5.18 pgloader.sources

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.19 pgloader.source.mssql

Source

package.lisp (file)

Use List
Used By List

pgloader.source.mssql.index-filter

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.20 pgloader.source.db3

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.21 pgloader.source.pgsql

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.22 pgloader.pgcopy

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.23 pgloader.source.csv

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.24 pgloader.catalog

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.25 pgloader.sql

Source

package.lisp (file)

Use List

common-lisp

Used By List

pgloader.parser

Exported Definitions

read-queries (function)

Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.26 pgloader.logs

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.27 pgloader.queries

Source

package.lisp (file)

Use List
Used By List

pgloader.utils

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.28 pgloader.source.ixf

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.29 pgloader.load

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

process-catalog (function)


Next: , Previous: , Up: Packages   [Contents][Index]

5.30 pgloader.quoting

Source

package.lisp (file)

Use List
Used By List
Exported Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.31 pgloader.archive

Source

package.lisp (file)

Use List
Used By List

pgloader.connection

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.32 pgloader.state

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Previous: , Up: Packages   [Contents][Index]

5.33 pgloader.user-symbols

Source

transforms.lisp (file)


Next: , Previous: , Up: Top   [Contents][Index]

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

6.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.1 Special variables

Special Variable: *cast-rules*

Specific casting rules added in the command.

Package

pgloader.sources

Source

casting-rules.lisp (file)

Special Variable: *client-min-messages*
Package

pgloader.params

Source

params.lisp (file)

Special Variable: *context*

Alist of (names . values) intialized from the environment at run-time, and from a –context command line argument, then used in the commands when they are using the Mustache templating feature.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *copy-batch-rows*

How many rows to batch per COPY transaction.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *copy-batch-size*

Maximum memory size allowed for a single batch.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *db3-default-cast-rules*

Data Type Casting rules to migrate from DB3 to PostgreSQL

Package

pgloader.source.db3

Source

db3-cast-rules.lisp (file)

Special Variable: *decoding-as*

Special per-table encoding/decoding overloading rules for MySQL.

Package

pgloader.source.mysql

Source

mysql.lisp (file)

Special Variable: *default-cast-rules*

Default casting rules.

Package

pgloader.sources

Source

casting-rules.lisp (file)

Special Variable: *default-tmpdir*

Place where to fetch and expand archives on-disk.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *dry-run*

Set to non-nil to only run checks about the load setup.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *fd-path-root*

Where to load files from, when loading from an archive or expanding regexps.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *identifier-case*

Dealing with source databases casing rules.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *log-filename*

Main pgloader log file

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *log-messengers*

Currently active log-messengers

Package

pgloader.logs

Source

logs.lisp (file)

Special Variable: *log-min-messages*
Package

pgloader.params

Source

params.lisp (file)

Special Variable: *monitoring-kernel*

Internal lparallel kernel to manage the separate monitor thread.

Package

pgloader.monitor

Source

monitor.lisp (file)

Special Variable: *monitoring-queue*

Internal lparallel queue where to send and receive messages from.

Package

pgloader.monitor

Source

monitor.lisp (file)

Special Variable: *mssql-default-cast-rules*

Data Type Casting to migrate from MSSQL to PostgreSQL

Package

pgloader.source.mssql

Source

mssql-cast-rules.lisp (file)

Special Variable: *mssql-settings*

An alist of GUC names and values.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *mysql-default-cast-rules*

Data Type Casting rules to migrate from MySQL to PostgreSQL

Package

pgloader.source.mysql

Source

mysql-cast-rules.lisp (file)

Special Variable: *mysql-settings*

An alist of GUC names and values.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *on-error-stop*

Set to non-nil to for pgloader to refrain from handling errors, quitting instead.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *pg-settings*

An alist of GUC names and values.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *pgsql-default-cast-rules*

Data Type Casting to migrate from PostgtreSQL to PostgreSQL

Package

pgloader.source.pgsql

Source

pgsql-cast-rules.lisp (file)

Special Variable: *pgsql-reserved-keywords*

We need to always quote PostgreSQL reserved keywords

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *prefetch-rows*

How many rows do read in advance in the reader queue.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *preserve-index-names*

Dealing with source databases index naming.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *report-stream*

Stream where to format the output stream.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *root-dir*

Top directory where to store all data logs and reject files.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *rows-per-range*

How many rows to read in each reader’s thread, per SQL query.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *self-upgrade-immutable-systems*

Used for –self-upgrade.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *sqlite-default-cast-rules*

Data Type Casting to migrate from SQLite to PostgreSQL

Package

pgloader.source.sqlite

Source

sqlite-cast-rules.lisp (file)

Special Variable: *summary-pathname*

Pathname where to output the summary.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *version-string*

pgloader version strings, following Emacs versionning model.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error-bad-source+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error-regress+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error-usage+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-success+
Package

pgloader.params

Source

params.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.2 Macros

Macro: log-message CATEGORY DESCRIPTION &rest ARGUMENTS

Protect against evaluating ARGUMENTS in cases where we don’t log at the given CATEGORY log-level.

Package

pgloader.monitor

Source

monitor.lisp (file)

Macro: push-to-end ITEM PLACE
Package

pgloader.catalog

Source

catalog.lisp (file)

Macro: timing &body FORMS

return both how much real time was spend in body and its result

Package

pgloader.monitor

Source

monitor.lisp (file)

Macro: with-connection (VAR CONNECTION &rest ARGS &key &allow-other-keys) &body FORMS

Connect to DB-CONNECTION and handle any condition when doing so, and when connected execute FORMS in a protected way so that we always disconnect at the end.

Package

pgloader.connection

Source

connection.lisp (file)

Macro: with-disabled-triggers (TABLE-NAME &key DISABLE-TRIGGERS) &body FORMS

Run FORMS with PostgreSQL triggers disabled for TABLE-NAME if DISABLE-TRIGGERS is T A PostgreSQL connection must be opened already where this macro is used.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Macro: with-monitor (&key START-LOGGER) &body BODY

Start and stop the monitor around BODY code. The monitor is responsible for processing logs into a central logfile

Package

pgloader.monitor

Source

monitor.lisp (file)

Macro: with-pgsql-connection (PGCONN) &body FORMS

Run FROMS within a PostgreSQL connection to DBNAME. To get the connection spec from the DBNAME, use ‘get-connection-spec’.

Package

pgloader.pgsql

Source

connection.lisp (file)

Macro: with-pgsql-transaction (&key PGCONN DATABASE) &body FORMS

Run FORMS within a PostgreSQL transaction to DBNAME, reusing DATABASE if given.

Package

pgloader.pgsql

Source

connection.lisp (file)

Macro: with-schema (VAR TABLE-NAME) &body BODY

When table-name is a CONS, SET search_path TO its CAR and return its CDR, otherwise just return the TABLE-NAME. A PostgreSQL connection must be established when calling this function.

Package

pgloader.catalog

Source

catalog.lisp (file)

Macro: with-stats-collection (TABLE-NAME &key SECTION DBNAME USE-RESULT-AS-READ USE-RESULT-AS-ROWS) &body FORMS

Measure time spent in running BODY into STATE, accounting the seconds to given DBNAME and TABLE-NAME

Package

pgloader.monitor

Source

monitor.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.3 Functions

Function: add-to-search-path CATALOG &key LABEL SECTION LOG-LEVEL CLIENT-MIN-MESSAGES

Add catalog schemas in the database search_path.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: apply-casting-rules TABLE-NAME COLUMN-NAME DTYPE CTYPE DEFAULT NULLABLE EXTRA &key RULES

Apply the given RULES to the MySQL SOURCE type definition

Package

pgloader.sources

Source

casting-rules.lisp (file)

Function: apply-identifier-case IDENTIFIER

Return given IDENTIFIER with CASE handled to be PostgreSQL compatible.

Package

pgloader.quoting

Source

quoting.lisp (file)

Function: archivep ARCHIVE-FILE

Return non-nil when the ARCHIVE-FILE is something we know how to expand.

Package

pgloader.archive

Source

archive.lisp (file)

Function: build-identifier SEP &rest PARTS

Concatenante PARTS into a PostgreSQL identifier, with SEP in between
parts. That’s useful for creating an index name from a table’s oid and name.

Package

pgloader.quoting

Source

quoting.lisp (file)

Function: camelcase-to-colname STRING

Transform input STRING into a suitable column name. lahmanID lahman_id
playerID player_id
birthYear birth_year

Package

pgloader.quoting

Source

quoting.lisp (file)

Function: catalog-distribution-rules INSTANCE
Function: (setf catalog-distribution-rules) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: catalog-name INSTANCE
Function: (setf catalog-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: catalog-schema-list INSTANCE
Function: (setf catalog-schema-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: catalog-types-without-btree INSTANCE
Function: (setf catalog-types-without-btree) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: citus-backfill-table-p TABLE

Returns non-nil when given TABLE should be backfilled with the distribution key.

Package

pgloader.catalog

Source

citus.lisp (file)

Function: citus-distribute-schema CATALOG DISTRIBUTION-RULES

Distribute a CATALOG with given user provided DISTRIBUTION-RULES. Return the list of rules applied.

Package

pgloader.citus

Source

citus.lisp (file)

Function: citus-distributed-rule-from INSTANCE
Function: (setf citus-distributed-rule-from) VALUE INSTANCE
Package

pgloader.catalog

Source

citus.lisp (file)

Function: citus-distributed-rule-p OBJECT
Package

pgloader.citus

Source

citus.lisp (file)

Function: citus-distributed-rule-table INSTANCE
Function: (setf citus-distributed-rule-table) VALUE INSTANCE
Package

pgloader.catalog

Source

citus.lisp (file)

Function: citus-distributed-rule-using INSTANCE
Function: (setf citus-distributed-rule-using) VALUE INSTANCE
Package

pgloader.catalog

Source

citus.lisp (file)

Function: citus-format-sql-select SOURCE-TABLE TARGET-TABLE

Return the SQL statement to use to fetch data from the COPY context, including backfilling the distribution key in related tables.

Package

pgloader.catalog

Source

citus.lisp (file)

Function: citus-reference-rule-p OBJECT
Package

pgloader.citus

Source

citus.lisp (file)

Function: citus-reference-rule-table INSTANCE
Function: (setf citus-reference-rule-table) VALUE INSTANCE
Package

pgloader.citus

Source

citus.lisp (file)

Function: column-comment INSTANCE
Function: (setf column-comment) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-default INSTANCE
Function: (setf column-default) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-extra INSTANCE
Function: (setf column-extra) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-name INSTANCE
Function: (setf column-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-nullable INSTANCE
Function: (setf column-nullable) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-transform INSTANCE
Function: (setf column-transform) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-transform-default INSTANCE
Function: (setf column-transform-default) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-type-mod INSTANCE
Function: (setf column-type-mod) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-type-name INSTANCE
Function: (setf column-type-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: comment-on-tables-and-columns CATALOG &key LABEL SECTION

Install comments on tables and columns from CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: convert-ini-into-commands FILENAME

Read the INI file at FILENAME and convert each section of it to a command in the new pgloader format.

Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: convert-ini-into-files FILENAME TARGET-DIRECTORY &key WITH-DATA-INLINE INCLUDE-SQL-FILE

Reads the INI file at FILENAME and creates files names <section>.load for each section in the INI file, in TARGET-DIRECTORY.

When WITH-DATA-INLINE is true, read the CSV file listed as the section’s filename and insert its content in the command itself, as inline data.

When INCLUDE-SQL-FILE is :if-exists, try to find a sibling file to the data file, with the same name and with the "sql" type, and use its content in a BEFORE LOAD DO clause.

When INCLUDE-SQL-FILE is t, not finding the SQL file is an error.

Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: copy-rows-from-queue COPY QUEUE &key DISABLE-TRIGGERS ON-ERROR-STOP COLUMNS &aux PGCONN TABLE

Fetch rows from the QUEUE, prepare them in batches and send them down to PostgreSQL, and when that’s done update stats.

Package

pgloader.pgcopy

Source

copy-from-queue.lisp (file)

Function: create-distributed-table DISTRIBUTE-RULES
Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-extensions CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all extensions from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-indexes-again TARGET CATALOG &key MAX-PARALLEL-CREATE-INDEX SECTION DROP-INDEXES

Create the indexes that we dropped previously.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-indexes-in-kernel PGCONN TABLE KERNEL CHANNEL &key LABEL

Create indexes for given table in dbname, using given lparallel KERNEL and CHANNEL so that the index build happen in concurrently with the data copying.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-pgsql-fkeys CATALOG &key SECTION LABEL LOG-LEVEL

Actually create the Foreign Key References that where declared in the MySQL database

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-schemas CATALOG &key INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all schemas from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-sqltypes CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create the needed data types for given CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-state ()
Package

pgloader.state

Source

state.lisp (file)

Function: create-table MAYBE-QUALIFIED-NAME

Create a table instance from the db-uri component, either a string or a cons of two strings: (schema . table).

Package

pgloader.catalog

Source

catalog.lisp (file)

Function: create-tables CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all tables from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-triggers CATALOG &key LABEL SECTION CLIENT-MIN-MESSAGES

Create the catalog objects that come after the data has been loaded.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-views CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all tables from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: distribute LIST-OF-RANGES COUNT

Split a list of ranges into COUNT sublists.

Package

pgloader.utils

Source

utils.lisp (file)

Function: drop-indexes TABLE-OR-CATALOG &key CASCADE LOG-LEVEL

Drop indexes in PGSQL-INDEX-LIST. A PostgreSQL connection must already be active when calling that function.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: drop-pgsql-fkeys CATALOG &key CASCADE LOG-LEVEL

Drop all Foreign Key Definitions given, to prepare for a clean run.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: elapsed-time-since START &optional END

Return how many seconds ticked between START and now

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: ensure-quoted VALUE &optional QUOTE-CHAR
Package

pgloader.quoting

Source

quoting.lisp (file)

Function: ensure-unquoted IDENTIFIER &optional QUOTE-CHAR
Package

pgloader.quoting

Source

quoting.lisp (file)

Function: execute-sql-code-block PGCONN SECTION COMMANDS LABEL

Exceute given SQL commands.

Package

pgloader.parser

Source

command-sql-block.lisp (file)

Function: expand-archive ARCHIVE-FILE &key TMPDIR

Expand given ARCHIVE-FILE in TMPDIR/(pathname-name ARCHIVE-FILE). Return the pathname where we did expand the archive file.

Package

pgloader.archive

Source

archive.lisp (file)

Function: expand-user-homedir-pathname NAMESTRING

Expand NAMESTRING replacing leading ~ with (user-homedir-pathname)

Package

pgloader.utils

Source

utils.lisp (file)

Function: extension-name INSTANCE
Function: (setf extension-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: extension-schema INSTANCE
Function: (setf extension-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fetch-pgsql-catalog DBNAME &key TABLE SOURCE-CATALOG INCLUDING EXCLUDING VARIANT PGVERSION

Fetch PostgreSQL catalogs for the target database. A PostgreSQL connection must be opened.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: finalize-catalogs CATALOG VARIANT

Finalize the target PostgreSQL catalogs, dumbing down datatypes when the target actually is Redshift rather than core PostgreSQL.

Package

pgloader.pgsql

Source

pgsql-finalize-catalogs.lisp (file)

Function: fkey-columns INSTANCE
Function: (setf fkey-columns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-condef INSTANCE
Function: (setf fkey-condef) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-deferrable INSTANCE
Function: (setf fkey-deferrable) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-delete-rule INSTANCE
Function: (setf fkey-delete-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-foreign-columns INSTANCE
Function: (setf fkey-foreign-columns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-foreign-table INSTANCE
Function: (setf fkey-foreign-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-initially-deferred INSTANCE
Function: (setf fkey-initially-deferred) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-match-rule INSTANCE
Function: (setf fkey-match-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-name INSTANCE
Function: (setf fkey-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-oid INSTANCE
Function: (setf fkey-oid) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-table INSTANCE
Function: (setf fkey-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-update-rule INSTANCE
Function: (setf fkey-update-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: flush-summary &key RESET
Package

pgloader.monitor

Source

monitor.lisp (file)

Function: format-vector-row NB-COLS ROW
Package

pgloader.pgsql

Source

copy-format.lisp (file)

Function: get-matching-filenames DIRECTORY REGEX

Apply given REGEXP to the DIRECTORY contents and return the list of matching files.

Package

pgloader.archive

Source

archive.lisp (file)

Function: get-pathname DBNAME TABLE-NAME &key FD-PATH-ROOT

Return a pathname where to read or write the file data

Package

pgloader.sources

Source

files-and-pathnames.lisp (file)

Function: get-state-section STATE SECTION
Package

pgloader.state

Source

state.lisp (file)

Function: getenv-default NAME &optional DEFAULT

Return the value of the NAME variable as found in the environment, or DEFAULT if that variable isn’t set

Package

pgloader.params

Source

params.lisp (file)

Function: guess-csv-params FILENAME-OR-STREAM NB-COLS &key SAMPLE-SIZE

Try a bunch of field separators with LINES and return the first one that returns COLS number of columns

Package

pgloader.source.csv

Source

csv-guess.lisp (file)

Function: http-fetch-file URL &key TMPDIR

Download a file from URL into TMPDIR.

Package

pgloader.archive

Source

archive.lisp (file)

Function: index-columns INSTANCE
Function: (setf index-columns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-condef INSTANCE
Function: (setf index-condef) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-conname INSTANCE
Function: (setf index-conname) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-filter INSTANCE
Function: (setf index-filter) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-fk-deps INSTANCE
Function: (setf index-fk-deps) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-name INSTANCE
Function: (setf index-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-oid INSTANCE
Function: (setf index-oid) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-primary INSTANCE
Function: (setf index-primary) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-schema INSTANCE
Function: (setf index-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-sql INSTANCE
Function: (setf index-sql) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-table INSTANCE
Function: (setf index-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-type INSTANCE
Function: (setf index-type) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-unique INSTANCE
Function: (setf index-unique) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: init-params-from-environment ()

Some of our parameters get their default value from the env. Do that at runtime when using a compiled binary.

Package

pgloader.params

Source

params.lisp (file)

Function: initialize-context FILENAME

Initialize a context from the environment variables and from the given context-filename (might be nil). CONTEXT-FILENAME is an INI file.

Package

pgloader.parser

Source

template.lisp (file)

Function: intern-symbol SYMBOL-NAME &optional OVERRIDES

Return a symbol in either PGLOADER.TRANSFORMS if it exists there
already (it’s a user provided function) or a PGLOADER.USER-SYMBOLS package.

OVERRIDES is an alist of symbol . value, allowing called to force certain values: the classic example is how to parse the "nil" symbol-name. Given OVERRIDES as ’((nil . nil)) the returned symbol will be cl:nil rather than pgloader.user-symbols::nil.

Package

pgloader.transforms

Source

transforms.lisp (file)

Function: lisp-code-for-loading-from-copy COPY-CONN PG-DB-CONN &key ENCODING FIELDS TARGET-TABLE-NAME COLUMNS GUCS BEFORE AFTER OPTIONS &aux WORKER-COUNT CONCURRENCY
Package

pgloader.parser

Source

command-copy.lisp (file)

Function: lisp-code-for-loading-from-csv CSV-CONN PG-DB-CONN &key ENCODING FIELDS TARGET-TABLE-NAME COLUMNS GUCS BEFORE AFTER OPTIONS &allow-other-keys &aux WORKER-COUNT CONCURRENCY
Package

pgloader.parser

Source

command-csv.lisp (file)

Function: lisp-code-for-loading-from-dbf DBF-DB-CONN PG-DB-CONN &key TARGET-TABLE-NAME ENCODING GUCS CASTS OPTIONS BEFORE AFTER-SCHEMA AFTER &allow-other-keys
Package

pgloader.parser

Source

command-dbf.lisp (file)

Function: lisp-code-for-loading-from-fixed FIXED-CONN PG-DB-CONN &key ENCODING FIELDS TARGET-TABLE-NAME COLUMNS GUCS BEFORE AFTER OPTIONS &allow-other-keys &aux WORKER-COUNT CONCURRENCY
Package

pgloader.parser

Source

command-fixed.lisp (file)

Function: lisp-code-for-loading-from-ixf IXF-DB-CONN PG-DB-CONN &key TARGET-TABLE-NAME GUCS OPTIONS BEFORE AFTER-SCHEMA AFTER &allow-other-keys
Package

pgloader.parser

Source

command-ixf.lisp (file)

Function: lisp-code-for-loading-from-mssql MS-DB-CONN PG-DB-CONN &key GUCS MSSQL-GUCS CASTS BEFORE AFTER AFTER-SCHEMA OPTIONS DISTRIBUTE VIEWS ALTER-SCHEMA ALTER-TABLE INCLUDING EXCLUDING &allow-other-keys
Package

pgloader.parser

Source

command-mssql.lisp (file)

Function: lisp-code-for-loading-from-mysql MY-DB-CONN PG-DB-CONN &key GUCS MYSQL-GUCS CASTS OPTIONS VIEWS BEFORE AFTER AFTER-SCHEMA ALTER-TABLE ALTER-SCHEMA DISTRIBUTE (INCLUDING INCL) (EXCLUDING EXCL) (DECODING DECODING-AS) &allow-other-keys
Package

pgloader.parser

Source

command-mysql.lisp (file)

Function: lisp-code-for-loading-from-sqlite SQLITE-DB-CONN PG-DB-CONN &key GUCS CASTS OPTIONS BEFORE AFTER-SCHEMA AFTER ALTER-TABLE ALTER-SCHEMA (INCLUDING INCL) (EXCLUDING EXCL) &allow-other-keys
Package

pgloader.parser

Source

command-sqlite.lisp (file)

Function: list-all-columns CATALOG &key TABLE-TYPE INCLUDING EXCLUDING &aux TABLE-TYPE-NAME

Get the list of PostgreSQL column names per table.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-all-fkeys CATALOG &key INCLUDING EXCLUDING

Get the list of PostgreSQL index definitions per table.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-all-indexes CATALOG &key INCLUDING EXCLUDING PGVERSION

Get the list of PostgreSQL index definitions per table.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-all-sqltypes CATALOG &key INCLUDING EXCLUDING

Set the catalog’s schema extension list and sqltype list

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-encodings-and-aliases ()

Return an alist of encoding names supported by the current implementation, associated with a list of encoding name aliases for each of them.

Package

pgloader.utils

Source

charsets.lisp (file)

Function: list-missing-fk-deps CATALOG

Add in the CATALOG the foreign keys we don’t have to deal with directly but that the primary keys we are going to DROP then CREATE again depend on: we need to take care of those first.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-reserved-keywords PGCONN

Connect to PostgreSQL DBNAME and fetch reserved keywords.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: list-schemas ()

Return the list of PostgreSQL schemas in the already established PostgreSQL connection.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-table-oids CATALOG &key VARIANT

Return an hash table mapping TABLE-NAME to its OID for all table in the TABLE-NAMES list. A PostgreSQL connection must be established already.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-typenames-without-btree-support ()

Fetch PostgresQL data types without btree support, so that it’s possible to later CREATE INDEX ... ON ... USING gist(...), or even something else than gist.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: load-data &key (FROM SOURCE) (INTO TARGET) ENCODING FIELDS TARGET-TABLE-NAME OPTIONS GUCS CASTS BEFORE AFTER START-LOGGER FLUSH-SUMMARY

Load data from SOURCE into TARGET.

Package

pgloader

Source

api.lisp (file)

Function: make-catalog &key (NAME NAME) (SCHEMA-LIST SCHEMA-LIST) (TYPES-WITHOUT-BTREE TYPES-WITHOUT-BTREE) (DISTRIBUTION-RULES DISTRIBUTION-RULES)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-citus-distributed-rule &key (TABLE TABLE) (USING USING) (FROM FROM)
Package

pgloader.catalog

Source

citus.lisp (file)

Function: make-citus-reference-rule &key (TABLE TABLE)
Package

pgloader.catalog

Source

citus.lisp (file)

Function: make-column &key (TABLE TABLE) (NAME NAME) (TYPE-NAME TYPE-NAME) (TYPE-MOD TYPE-MOD) (NULLABLE NULLABLE) (DEFAULT DEFAULT) (COMMENT COMMENT) (TRANSFORM TRANSFORM) (EXTRA EXTRA) (TRANSFORM-DEFAULT TRANSFORM-DEFAULT)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-external-format NAME

Return an object suitable as an external format in the current implementation.

Package

pgloader.utils

Source

charsets.lisp (file)

Function: make-fkey &key (NAME NAME) (OID OID) (TABLE TABLE) (COLUMNS COLUMNS) (PKEY PKEY) (FOREIGN-TABLE FOREIGN-TABLE) (FOREIGN-COLUMNS FOREIGN-COLUMNS) (CONDEF CONDEF) (UPDATE-RULE UPDATE-RULE) (DELETE-RULE DELETE-RULE) (MATCH-RULE MATCH-RULE) (DEFERRABLE DEFERRABLE) (INITIALLY-DEFERRED INITIALLY-DEFERRED)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-including-expr-from-catalog CATALOG

Return an expression suitable to be used as an :including parameter.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: make-including-expr-from-view-names VIEW-NAMES

Turn MATERIALIZING VIEWs list of view names into an INCLUDING parameter.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: make-index &key (NAME NAME) (OID OID) (SCHEMA SCHEMA) (TABLE TABLE) (TYPE TYPE) (PRIMARY PRIMARY) (UNIQUE UNIQUE) (COLUMNS COLUMNS) (SQL SQL) (CONNAME CONNAME) (CONDEF CONDEF) (FILTER FILTER) (FK-DEPS FK-DEPS)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-kernel WORKER-COUNT &key BINDINGS

Wrapper around lparallel:make-kernel that sets our usual bindings.

Package

pgloader.utils

Source

threads.lisp (file)

Function: make-match-rule &key (RULE RULE) (SCHEMA SCHEMA) (ACTION ACTION) (ARGS ARGS)
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: make-matview &key (SOURCE-NAME SOURCE-NAME) (NAME NAME) (SCHEMA SCHEMA) (DEFINITION DEFINITION)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-pgstate &key (TABLES TABLES) (TABNAMES TABNAMES) (READ READ) (ROWS ROWS) (ERRS ERRS) (SECS SECS) (RS RS) (WS WS) (BYTES BYTES)
Package

pgloader.state

Source

state.lisp (file)

Function: make-procedure &key (SCHEMA SCHEMA) (NAME NAME) (RETURNS RETURNS) (LANGUAGE LANGUAGE) (BODY BODY)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-regex-match-rule &key (TARGET TARGET)
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: make-schema &key (SOURCE-NAME SOURCE-NAME) (NAME NAME) (CATALOG CATALOG) (IN-SEARCH-PATH IN-SEARCH-PATH) (TABLE-LIST TABLE-LIST) (VIEW-LIST VIEW-LIST) (MATVIEW-LIST MATVIEW-LIST) (EXTENSION-LIST EXTENSION-LIST) (SQLTYPE-LIST SQLTYPE-LIST)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-sqltype &key (NAME NAME) (SCHEMA SCHEMA) (TYPE TYPE) (SOURCE-DEF SOURCE-DEF) (EXTRA EXTRA) (EXTENSION EXTENSION)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-state &key (PRELOAD PRELOAD) (DATA DATA) (POSTLOAD POSTLOAD) (BYTES BYTES) (SECS SECS)
Package

pgloader.state

Source

state.lisp (file)

Function: make-string-match-rule &key (TARGET TARGET)
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: make-table &key (SOURCE-NAME SOURCE-NAME) (NAME NAME) (SCHEMA SCHEMA) (OID OID) (COMMENT COMMENT) (STORAGE-PARAMETER-LIST STORAGE-PARAMETER-LIST) (TABLESPACE TABLESPACE) (FIELD-LIST FIELD-LIST) (COLUMN-LIST COLUMN-LIST) (INDEX-LIST INDEX-LIST) (FKEY-LIST FKEY-LIST) (TRIGGER-LIST TRIGGER-LIST) (CITUS-RULE CITUS-RULE)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-trigger &key (NAME NAME) (TABLE TABLE) (ACTION ACTION) (PROCEDURE PROCEDURE)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: match-rule-action INSTANCE
Function: (setf match-rule-action) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: match-rule-args INSTANCE
Function: (setf match-rule-args) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: match-rule-rule INSTANCE
Function: (setf match-rule-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: match-rule-schema INSTANCE
Function: (setf match-rule-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: matview-definition INSTANCE
Function: (setf matview-definition) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: matview-name INSTANCE
Function: (setf matview-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: matview-schema INSTANCE
Function: (setf matview-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: matview-source-name INSTANCE
Function: (setf matview-source-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: maybe-drop-indexes CATALOG &key DROP-INDEXES

Drop the indexes for TABLE-NAME on TARGET PostgreSQL connection, and returns a list of indexes to create again. A PostgreSQL connection must already be active when calling that function.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: merge-catalogs SOURCE-CATALOG TARGET-CATALOG

In order for the data loading to be as fast as possible, we DROP the constraints and indexes on the target table. Once the data is loaded we want to install the same constraints as found pre-existing in the TARGET-CATALOG rather than the one we casted from the SOURCE-CATALOG.

Also, we want to recheck the cast situation and the selected transformation functions of each column.

Package

pgloader.pgsql

Source

merge-catalogs.lisp (file)

Function: new-label SECTION LABEL &optional DBNAME

Send an event to create a new LABEL for registering a shared state under SECTION.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: parse-cli-casts CASTS

Parse additional CAST rules when we get them from the CLI.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-encoding ENCODING

Parse the –encoding option

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-fields TYPE FIELDS

Parse the –fields option.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-gucs GUCS

Parse PostgreSQL GUCs as per the SET clause when we get them from the CLI.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-options TYPE OPTIONS

Parse options as per the WITH clause when we get them from the CLI.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-type TYPE

Parse the –type option

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-column-typemod DATA-TYPE COLUMN-TYPE

Given int(7), returns the number 7.

Beware that some data-type are using a typmod looking definition for things that are not typmods at all: enum.

Package

pgloader.transforms

Source

transforms.lisp (file)

Function: parse-commands COMMANDS-TEMPLATE &key START END JUNK-ALLOWED

Parse a command and return a LAMBDA form that takes no parameter.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-commands-from-file MAYBE-RELATIVE-FILENAME &aux FILENAME

The command could be using from :inline, in which case we want to parse as much as possible then use the command against an already opened stream where we moved at the beginning of the data.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-date-format FORMAT-STRING

Parse a given format string and return a format specification for parse-date-string

Package

pgloader.parse-date

Source

date-format.lisp (file)

Function: parse-date-string DATE-STRING &optional FORMAT

Apply this function when input date in like ’20041002152952’

Package

pgloader.parse-date

Source

date-format.lisp (file)

Function: parse-ini-file FILENAME

Parse an old-style INI file into a list of PARAMS structures

Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: parse-source-string SOURCE-STRING
Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-source-string-for-type TYPE SOURCE-STRING

use the parse rules as per xxx-source rules

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-sql-file FILENAME

Parse FILENAME for SQL statements

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-sqlite-type-name TYPE-NAME
Package

pgloader.parser

Source

parse-sqlite-type-name.lisp (file)

Function: parse-target-pg-db-uri COMMAND-FILE

Partially parse COMMAND-FILE and return its target connection string.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-target-string TARGET-STRING
Package

pgloader.parser

Source

command-parser.lisp (file)

Function: pgsql-connect-and-execute-with-timing PGCONN SECTION LABEL SQL

Run pgsql-execute-with-timing within a newly establised connection.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: pgsql-execute SQL &key LOG-LEVEL CLIENT-MIN-MESSAGES ON-ERROR-STOP

Execute given SQL list of statements in current transaction.

When ON-ERROR-STOP is non-nil (the default), we stop at the first sql statement that fails. That’s because this facility is meant for DDL. With ON_ERROR_STOP nil, log the problem and continue thanks to PostgreSQL savepoints.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: pgsql-execute-with-timing SECTION LABEL SQL-LIST &key LOG-LEVEL ON-ERROR-STOP CLIENT-MIN-MESSAGES

Execute given SQL and resgister its timing into STATE.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: pgstate-decf PGSTATE NAME &key READ ROWS ERRS SECS RS WS BYTES
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-errs INSTANCE
Function: (setf pgstate-errs) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-get-label PGSTATE NAME
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-incf PGSTATE NAME &key READ ROWS ERRS SECS RS WS BYTES
Package

pgloader.state

Source

state.lisp (file)

<