The shorty Reference Manual

Table of Contents

The shorty Reference Manual

This is the shorty Reference Manual, version 1.0.0, generated automatically by Declt version 2.4 "Will Decker" on Wed Jun 20 12:34:15 2018 GMT+0.

1 Introduction

shorty

A toy URL datastore.

Overview

This is a toy project which can be used to store any amount of data in the URL of a URL-shortening service. It exploits the fact that most URL shortening services do not check that a URL points to a real website, and as such, any data can be URL-shortened.

How it works:

• Takes a file pathname as input.

• Compresses the file with the DEFLATE compression scheme.

• Encodes the compressed file using the Base64 encoding scheme.

• Splits the data into chunks of typically a few kilobytes long, depending on the URL shortening service used.

• Creates a short URL for each chunk.

• Concatenates the path components of all short URLs into 1 long string, separated by a forward slash /.

• Again, creates a short URL using the concatenated path components as input.

• The result is a single short URL that has an arbitrary amount of data stored in the URL.

Install

(ql:quickload :shorty)

Usage

To store a file in a URL:

(shorty:shorten-file "/path/to/in-file")

To write a file to disk given a URL:

(shorty:unshorten-file "http://short-url" :file "/path/to/out-file")

Known Limitations

There are 2 main issues that will likely never be fixed, as this is just meant to be a proof of concept toy.

• Local limit: Shortening and unshortening are done completely in memory, so you need enough heap space.

• Remote limit: During the concatenation of URL path components to create the single short URL at the end of the shortening process, the resulting string must also fit into a chunk size supported by the specified URL-shortening service. The path component length depends on the service. Do not expect to store more than a few megabytes of data at best.

License

Copyright © 2017-2018 Michael Fiano.

Licensed under the MIT License.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 shorty

Maintainer

Michael Fiano <mail@michaelfiano.com>

Author

Michael Fiano <mail@michaelfiano.com>

Home Page

https://www.michaelfiano.com/projects/shorty

Source Control

(:git "git@github.com:mfiano/shorty.git")

Bug Tracker

https://github.com/mfiano/shorty/issues

License

MIT

Description

Shorten URLs using third-party services.

Long Description

# shorty

A toy URL datastore.


## Overview

This is a toy project which can be used to store any amount of data in the URL of a URL-shortening service. It exploits the fact that most URL shortening services do not check that a URL points to a real website, and as such, any data can be URL-shortened.

How it works:

- Takes a file pathname as input.

- Compresses the file with the DEFLATE compression scheme.

- Encodes the compressed file using the Base64 encoding scheme.

- Splits the data into chunks of typically a few kilobytes long, depending on
the URL shortening service used.

- Creates a short URL for each chunk.

- Concatenates the path components of all short URLs into 1 long string,
separated by a forward slash ‘/‘.

- Again, creates a short URL using the concatenated path components as input.

- The result is a single short URL that has an arbitrary amount of data stored
in the URL.

## Install

“‘lisp
(ql:quickload :shorty)
“‘

## Usage

To store a file in a URL:

“‘lisp
(shorty:shorten-file "/path/to/in-file")
“‘

To write a file to disk given a URL:

“‘lisp
(shorty:unshorten-file "http://short-url" :file "/path/to/out-file")
“‘

## Known Limitations

There are 2 main issues that will likely never be fixed, as this is just meant to be a proof of concept toy.

- Local limit: Shortening and unshortening are done completely in memory, so you need enough heap space.

- Remote limit: During the concatenation of URL path components to create the single short URL at the end of the shortening process, the resulting string must also fit into a chunk size supported by the specified URL-shortening service. The path component length depends on the service. Do not expect to store more than a few megabytes of data at best.

## License

Copyright © 2017-2018 [Michael Fiano](mailto:mail@michaelfiano.com).

Licensed under the MIT License.

Version

1.0.0

Dependencies

• dexador
• quri
• split-sequence
• cl-base64
• babel
• salza2
• chipz
• fast-io

Source

shorty.asd (file)

Components

• package.lisp (file)
• services.lisp (file)
• shorty.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 shorty.asd

Location

/home/quickref/quicklisp/dists/quicklisp/software/shorty-20180228-git/shorty.asd

Systems

shorty (system)

3.1.2 shorty/package.lisp

Parent

shorty (system)

Location

package.lisp

Packages

shorty

3.1.3 shorty/services.lisp

Dependency

package.lisp (file)

Parent

shorty (system)

Location

services.lisp

Internal Definitions

• service-properties (method)
• service-properties (method)

3.1.4 shorty/shorty.lisp

Dependency

services.lisp (file)

Parent

shorty (system)

Location

shorty.lisp

Exported Definitions

• shorten-file (function)
• unshorten-file (function)

Internal Definitions

• *input-buffer* (special variable)
• build-query (function)
• compress-data (function)
• decode-data (function)
• encode-data (function)
• finalize-shorten (function)
• generate-chunk-size (function)
• generate-host (generic function)
• generate-host (method)
• generate-host (method)
• get-path-component (function)
• get-redirect (function)
• get-redirect-path (function)
• read-chunk (function)
• rejoin-chunks (function)
• resolve-chunk-urls (function)
• shorten-chunks (function)
• shorten-data (function)
• uncompress-data (function)
• unshorten-chunks (function)

4 Packages

Packages are listed by definition order.

4.1 shorty

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions

• shorten-file (function)
• unshorten-file (function)

Internal Definitions

• *input-buffer* (special variable)
• build-query (function)
• compress-data (function)
• decode-data (function)
• encode-data (function)
• finalize-shorten (function)
• generate-chunk-size (function)
• generate-host (generic function)
• generate-host (method)
• generate-host (method)
• get-path-component (function)
• get-redirect (function)
• get-redirect-path (function)
• read-chunk (function)
• rejoin-chunks (function)
• resolve-chunk-urls (function)
• service-properties (generic function)
• service-properties (method)
• service-properties (method)
• shorten-chunks (function)
• shorten-data (function)
• uncompress-data (function)
• unshorten-chunks (function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Functions

Function: shorten-file FILE &key SERVICE

Shorten the file corresponding to the pathname FILE, producing a single short URL which encodes all of its data.

Package

shorty

Source

shorty.lisp (file)

Function: unshorten-file URL &key FILE

Unshorten the file stored in URL, storing the result to disk in the pathname corresponding to FILE.

Package

shorty

Source

shorty.lisp (file)

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *input-buffer*

Package

shorty

Source

shorty.lisp (file)

5.2.2 Functions

Function: build-query SERVICE DATA

Build a query URL string for SERVICE and the encoded DATA.

Package

shorty

Source

shorty.lisp (file)

Function: compress-data DATA

Compress DATA using the DEFLATE compression scheme.

Package

shorty

Source

shorty.lisp (file)

Function: decode-data DATA

Decode DATA encoded in the Base64 encoding scheme.

Package

shorty

Source

shorty.lisp (file)

Function: encode-data DATA

Encode DATA using the Base64 encoding scheme.

Package

shorty

Source

shorty.lisp (file)

Function: finalize-shorten SERVICE CHUNKS

Shortens all shortened URL CHUNKS into a single short URL for the specified SERVICE.

Package

shorty

Source

shorty.lisp (file)

Function: generate-chunk-size SERVICE

Generate a random chunk size within 25% of the maximum chunk size allowed for SERVICE.

Package

shorty

Source

shorty.lisp (file)

Function: get-path-component URL

Get the path component of URL.

Package

shorty

Source

shorty.lisp (file)

Function: get-redirect URL

Get the 301 redirection target from the header of URL.

Package

shorty

Source

shorty.lisp (file)

Function: get-redirect-path URL

Get the path component of the 301 redirection target of URL.

Package

shorty

Source

shorty.lisp (file)

Function: read-chunk SERVICE DATA

Read a chunk of DATA as a string, with a size up to the maximum size supported by SERVICE.

Package

shorty

Source

shorty.lisp (file)

Function: rejoin-chunks URL

Rejoin all chunks encoded in the short URL URL, returning a DEFLATE-compressed, Base64-encoded string.

Package

shorty

Source

shorty.lisp (file)

Function: resolve-chunk-urls URL

Resolve all the chunks contained in the short URL URL, returning a list of more short URL’s.

Package

shorty

Source

shorty.lisp (file)

Function: shorten-chunks SERVICE DATA

Split DATA into chunks, collecting a list of shortened URL’s of each chunk, for the specified SERVICE.

Package

shorty

Source

shorty.lisp (file)

Function: shorten-data SERVICE DATA

Query the specified SERVICE for a short URL that redirects to a long URL storing the encoded DATA.

Package

shorty

Source

shorty.lisp (file)

Function: uncompress-data DATA

Uncompress DATA encoded in the DEFLATE compression scheme.

Package

shorty

Source

shorty.lisp (file)

Function: unshorten-chunks URL

Unshorten the data stored in URL, returning an octet vector.

Package

shorty

Source

shorty.lisp (file)

5.2.3 Generic functions

Generic Function: generate-host TYPE

Package

shorty

Source

shorty.lisp (file)

Methods

Method: generate-host (TYPE (eql google))

Generate a URL prefix using Google’s domain name. This is a workaround for bugs present in both drakma and dexador HTTP client libraries.

Method: generate-host (TYPE (eql random))

Generate a URL prefix with a random 4-character domain name under the .com TLD, and a random 4-character sub-domain, which will prefix the encoded data in a URL shortening service query, in order to mimic an real website link.

Generic Function: service-properties SERVICE

Package

shorty

Methods

Method: service-properties (SERVICE (eql tinyurl))

Source

services.lisp (file)

Method: service-properties (SERVICE (eql isgd))

Source

services.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types