The xcat Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The xcat Reference Manual

This is the xcat Reference Manual, version 0.0.1, generated automatically by Declt version 3.0 "Montgomery Scott" on Wed Nov 04 15:45:50 2020 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

xcat

xcat is a simple mechanism for mass distributing big files on LANs. It was developed for simultaneously imaging multiple embedded board's SSDs while on a burnin rack in a way that allows them to cooperate amongst themselves to ease the IT department burden. Hundreds of of boards being turned on all at once where each one of them requests the same disk image file at the same time can easily saturate network uplinks and NICs. As much as IT departments love to buy and reside over more fancy equipment, this can be solved without budget hikes via a one-line change to production shell scripts.

From something like:

tar xfzv some/nfs/dir/image.tar.gz

to

xcat some/nfs/dir/image.tar.gz | tar xfzv -

xcat clients begin by broadcasting a pathname and file offset on udp port 19023. From that point, an xcatd server will initiate a TCP connection back to the client and send it the requested 16MByte chunk + checksum, if it has it. If another xcat client happens to have that chunk in RAM, it will also attempt to connect to the broadcasting client. The original client accepts one (and only one) incoming connection and then closes the listening port. In this way, whomever is the fastest to react and establish the connection first wins and the rest have their connections harmlessly refused. It is expected xcat clients keep at least one of the previous 16MByte chunks in memory as it attempts to download the next. In the case of a lot of boards being turned on en-masse and simultaneously attempting to fetch the same file, the origin xcatd server and/or network pipe eventually gets overloaded and a situation develops naturally for peers on closer ethernet switches to beat the origin xcatd server in establishing the first response TCP connection.

Note that the 16Mbyte TCP transfers are intended to be sent from RAM to RAM, i.e. not streamed from server filesystem to client filesystem or block device. The server only initiates a connection after it has completely read the chunk and calculated the checksum. The client only starts broadcasting requests when it has a 16MByte block of memory ready and waiting.

The embedded xcat client is a simple single portable xcat.c that only holds onto and serves helpouts on the previous 16mbyte chunk it has downloaded. This keeps memory usage compatible with embedded boards with limited RAM, while still allowing reasonable probabilty of being able to contribute and helpout peers during simultaneous download. To maximize scalability in light of this finite sliding window of transiently cached chunks, it is in the systems interest for all boards to start at the same moment and proceed at approximately the same pace. If a single downloader gets too far behind or too far ahead of its peers, the network loses the benefit of swarm caching. A typical way to accomplish this is to first synchronize time via NTP, then have all boards pause before download and synchronize continuation to the 0 second of the next minute. Since most boards write their SSDs at the same rate, it is likely boards remain in lock step throughout while consuming the stdout stream.

Usage

The XCAT server is started from the function (xcat:xcatd "/u/x/var/ts-production/images") If no directory argument is used the default is the Lisp home directory as returned by (user-homedir-pathname) All XCAT requests are attempted served from the file tree named by this argument. If a broadcasted request is received for a file that is not present, the server simply ignores the request.

;; Start the xcatd server and listen for broadcasted requests for files 
;; rooted from the user homedir:
(xcat:xcatd)

;; ...similar, but from a named root directory
(xcat:xcatd :root "/u/x/var/ts-production/images")

To use the client mode and broadcast for downloading a requested file, there are three methods:

;; Form 1, Output to a stream
(xcat:xcat "ts7000/flash.dd" *standard-output*)

;; Form 2, Output to a file/pathname
(xcat:xcat "ts7000/flash.tgz" #P"/tmp/flash.tgz")

;; Form 3, Output via callbacks
(xcat:xcat "ts7000/flash.gz" #'some-decompressor-callback-fn)

The function callback gets sent simple-array (unsigned-byte 8) (*) as its single argument.

The broadcast IP defaults to 255.255.255.255 but can be changed by setting the global xcat:*xcat-broadcast-ip*

UNIX command line interface

Although this system establishes Lisp functions for using XCAT, it does not provide mechanism for using the functions within a shell script or the UNIX command line. For that, another quicklisp system scriptl is recommended. Use this system to establish a background running, daemonized, Lisp instance that is then communicated with by a portable C binary over Unix domain socket IPC. Both the xcat and xcatd functions maintain RAM caches and continually serve helpouts in the background.

Sample xcatd background server image

To create a standalone image to be run on a server, the following Lisp script may serve as an example in building one. This creates a standalone daemonizing executable /tmp/xcatd for root directory /u/x/var/ts-production/images that logs to /tmp/log.txt and also starts background threads serving scriptl and zyredir (rooted at /u/x/zyre)

(ql:quickload '(:scriptl :zyre/zyredir :xcat :daemon))

(defparameter zyre::*zyredir* #p"/u/x/zyre/")

(defparameter uiop:*image-entry-point* (lambda ()
  (daemon:daemonize :exit-parent t)
  (log4cl:remove-all-appenders log4cl:*root-logger*)
  (log:config :daily "/tmp/log.txt" :backup nil) 
  (bt:make-thread #'zyre::zyredir :name "zyredir")
  (xcat:xcatd :root #p"/u/x/var/ts-production/images/" :background t)
  (bt:join-thread (scriptl:start))
  (daemon:exit)
))

(uiop/image:dump-image #p"/tmp/xcatd" :executable t)

Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 xcat

Author

Jesse Off <jesseoff@me.com>

License

MIT

Description

XCAT mass LAN big file distributor

Version

0.0.1

Dependencies
Source

xcat.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 xcat.asd

Location

xcat.asd

Systems

xcat (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 xcat/package.lisp

Parent

xcat (system)

Location

package.lisp

Packages

xcat


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.3 xcat/fifo.lisp

Dependency

package.lisp (file)

Parent

xcat (system)

Location

fifo.lisp

Internal Definitions

Previous: , Up: Lisp files   [Contents][Index]

3.1.4 xcat/xcatd.lisp

Dependency

fifo.lisp (file)

Parent

xcat (system)

Location

xcatd.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 xcat

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Special variables

Special Variable: *xcat-broadcast-ip*
Package

xcat

Source

xcatd.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Functions

Function: xcatd &key ROOT BACKGROUND
Package

xcat

Source

xcatd.lisp (file)


Previous: , Up: Exported definitions   [Contents][Index]

5.1.3 Generic functions

Generic Function: xcat PATH OUTPUT
Package

xcat

Source

xcatd.lisp (file)

Methods
Method: xcat PATH (OUT-FILE pathname)
Method: xcat PATH (OUT-STREAM stream)
Method: xcat PATH (OUT-FUNCTION function)

Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Constants

Constant: +max-array-size+
Package

xcat

Source

xcatd.lisp (file)

Constant: +xcatd-max-xfrs+
Package

xcat

Source

xcatd.lisp (file)

Constant: +xcatd-xfr-timeout-sec+
Package

xcat

Source

xcatd.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.2 Special variables

Special Variable: *xcat-chunks*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcat-helpout-thread*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcat-helpout-xfr-thread*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcat-lock*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcat-prefetcher-thread*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcat-prev-chunks*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcatd-chunks*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcatd-directory*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcatd-lock*
Package

xcat

Source

xcatd.lisp (file)

Special Variable: *xcatd-remotes*
Package

xcat

Source

xcatd.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.3 Macros

Macro: fifo-get! F

Removes and returns object from fifo. If fifo becomes empty, struct is destroyed, i.e. setf nil

Package

xcat

Source

fifo.lisp (file)

Macro: fifo-put! F OBJ

Adds obj to fifo, instantiating new fifo struct if necessary. Returns new count of fifo

Package

xcat

Source

fifo.lisp (file)

Macro: log-but-retry-errors-after-delay &body BODY
Package

xcat

Source

xcatd.lisp (file)

Macro: log-errors &body BODY
Package

xcat

Source

xcatd.lisp (file)

Macro: return-errors &body BODY
Package

xcat

Source

xcatd.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.4 Functions

Function: bufs-length BUFS
Package

xcat

Source

xcatd.lisp (file)

Function: copy-fifo INSTANCE
Package

xcat

Source

fifo.lisp (file)

Function: ensure-cached CHUNK &optional MEGABYTES
Package

xcat

Source

xcatd.lisp (file)

Function: fifo-count F
Package

xcat

Source

fifo.lisp (file)

Function: fifo-data INSTANCE
Function: (setf fifo-data) VALUE INSTANCE
Package

xcat

Source

fifo.lisp (file)

Function: fifo-endp F
Package

xcat

Source

fifo.lisp (file)

Function: fifo-head F
Package

xcat

Source

fifo.lisp (file)

Function: fifo-n INSTANCE
Function: (setf fifo-n) VALUE INSTANCE
Package

xcat

Source

fifo.lisp (file)

Function: fifo-p OBJECT
Package

xcat

Source

fifo.lisp (file)

Function: fifo-tail INSTANCE
Function: (setf fifo-tail) VALUE INSTANCE
Package

xcat

Source

fifo.lisp (file)

Function: file-read-chunk XCAT-REQ-STRING

Reads 16Mbyte chunk. Stores result in a weak hash table for cacheing. Returns a list of octet vectors– first is the reply header w/checksum, next are the file buf(s).

Package

xcat

Source

xcatd.lisp (file)

Function: make-bufs NBYTES

Makes list of arrays for when nbytes is greater than +max-array-size+

Package

xcat

Source

xcatd.lisp (file)

Function: make-fifo &key (DATA DATA) (TAIL TAIL) (N N)
Package

xcat

Source

fifo.lisp (file)

Function: net-read-chunk XCAT-REQ-STRING

Broadcasts requests for files via XCAT. Returns cons of two octet vectors– car is the resp header, cdr is the verified datablock(s).

Package

xcat

Source

xcatd.lisp (file)

Function: read-into BUFS STREAM

read-sequence into a list of bufs, returning multiple values of filled bufs and total size

Package

xcat

Source

xcatd.lisp (file)

Function: sum-bufs BUFS
Package

xcat

Source

xcatd.lisp (file)

Function: xcat-broadcast-handler BUF
Package

xcat

Source

xcatd.lisp (file)

Function: xcat-prefetch XCAT-REQ-STRING

Starts a nonblocking background request for the xcat-req-string chunk.

Package

xcat

Source

xcatd.lisp (file)

Function: xcat-req XCAT-REQ-STRING

Requests XCAT chunk, possibly from a prefetch or from the chunk cache hash table.

Package

xcat

Source

xcatd.lisp (file)

Function: xcatd-broadcast-handler BUF
Package

xcat

Source

xcatd.lisp (file)


Previous: , Up: Internal definitions   [Contents][Index]

5.2.5 Structures

Structure: fifo ()
Package

xcat

Source

fifo.lisp (file)

Direct superclasses

structure-object (structure)

Direct slots
Slot: data
Type

list

Readers

fifo-data (function)

Writers

(setf fifo-data) (function)

Slot: tail
Type

list

Readers

fifo-tail (function)

Writers

(setf fifo-tail) (function)

Slot: n
Type

fixnum

Initform

0

Readers

fifo-n (function)

Writers

(setf fifo-n) (function)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L   X  
Index Entry  Section

F
File, Lisp, xcat.asd: The xcat․asd file
File, Lisp, xcat/fifo.lisp: The xcat/fifo․lisp file
File, Lisp, xcat/package.lisp: The xcat/package․lisp file
File, Lisp, xcat/xcatd.lisp: The xcat/xcatd․lisp file

L
Lisp File, xcat.asd: The xcat․asd file
Lisp File, xcat/fifo.lisp: The xcat/fifo․lisp file
Lisp File, xcat/package.lisp: The xcat/package․lisp file
Lisp File, xcat/xcatd.lisp: The xcat/xcatd․lisp file

X
xcat.asd: The xcat․asd file
xcat/fifo.lisp: The xcat/fifo․lisp file
xcat/package.lisp: The xcat/package․lisp file
xcat/xcatd.lisp: The xcat/xcatd․lisp file

Jump to:   F   L   X  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (  
B   C   E   F   G   L   M   N   R   S   X  
Index Entry  Section

(
(setf fifo-data): Internal functions
(setf fifo-n): Internal functions
(setf fifo-tail): Internal functions

B
bufs-length: Internal functions

C
copy-fifo: Internal functions

E
ensure-cached: Internal functions

F
fifo-count: Internal functions
fifo-data: Internal functions
fifo-endp: Internal functions
fifo-get!: Internal macros
fifo-head: Internal functions
fifo-n: Internal functions
fifo-p: Internal functions
fifo-put!: Internal macros
fifo-tail: Internal functions
file-read-chunk: Internal functions
Function, (setf fifo-data): Internal functions
Function, (setf fifo-n): Internal functions
Function, (setf fifo-tail): Internal functions
Function, bufs-length: Internal functions
Function, copy-fifo: Internal functions
Function, ensure-cached: Internal functions
Function, fifo-count: Internal functions
Function, fifo-data: Internal functions
Function, fifo-endp: Internal functions
Function, fifo-head: Internal functions
Function, fifo-n: Internal functions
Function, fifo-p: Internal functions
Function, fifo-tail: Internal functions
Function, file-read-chunk: Internal functions
Function, make-bufs: Internal functions
Function, make-fifo: Internal functions
Function, net-read-chunk: Internal functions
Function, read-into: Internal functions
Function, sum-bufs: Internal functions
Function, xcat-broadcast-handler: Internal functions
Function, xcat-prefetch: Internal functions
Function, xcat-req: Internal functions
Function, xcatd: Exported functions
Function, xcatd-broadcast-handler: Internal functions

G
Generic Function, xcat: Exported generic functions

L
log-but-retry-errors-after-delay: Internal macros
log-errors: Internal macros

M
Macro, fifo-get!: Internal macros
Macro, fifo-put!: Internal macros
Macro, log-but-retry-errors-after-delay: Internal macros
Macro, log-errors: Internal macros
Macro, return-errors: Internal macros
make-bufs: Internal functions
make-fifo: Internal functions
Method, xcat: Exported generic functions
Method, xcat: Exported generic functions
Method, xcat: Exported generic functions

N
net-read-chunk: Internal functions

R
read-into: Internal functions
return-errors: Internal macros

S
sum-bufs: Internal functions

X
xcat: Exported generic functions
xcat: Exported generic functions
xcat: Exported generic functions
xcat: Exported generic functions
xcat-broadcast-handler: Internal functions
xcat-prefetch: Internal functions
xcat-req: Internal functions
xcatd: Exported functions
xcatd-broadcast-handler: Internal functions

Jump to:   (  
B   C   E   F   G   L   M   N   R   S   X  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   *   +  
C   D   N   S   T  
Index Entry  Section

*
*xcat-broadcast-ip*: Exported special variables
*xcat-chunks*: Internal special variables
*xcat-helpout-thread*: Internal special variables
*xcat-helpout-xfr-thread*: Internal special variables
*xcat-lock*: Internal special variables
*xcat-prefetcher-thread*: Internal special variables
*xcat-prev-chunks*: Internal special variables
*xcatd-chunks*: Internal special variables
*xcatd-directory*: Internal special variables
*xcatd-lock*: Internal special variables
*xcatd-remotes*: Internal special variables

+
+max-array-size+: Internal constants
+xcatd-max-xfrs+: Internal constants
+xcatd-xfr-timeout-sec+: Internal constants

C
Constant, +max-array-size+: Internal constants
Constant, +xcatd-max-xfrs+: Internal constants
Constant, +xcatd-xfr-timeout-sec+: Internal constants

D
data: Internal structures

N
n: Internal structures

S
Slot, data: Internal structures
Slot, n: Internal structures
Slot, tail: Internal structures
Special Variable, *xcat-broadcast-ip*: Exported special variables
Special Variable, *xcat-chunks*: Internal special variables
Special Variable, *xcat-helpout-thread*: Internal special variables
Special Variable, *xcat-helpout-xfr-thread*: Internal special variables
Special Variable, *xcat-lock*: Internal special variables
Special Variable, *xcat-prefetcher-thread*: Internal special variables
Special Variable, *xcat-prev-chunks*: Internal special variables
Special Variable, *xcatd-chunks*: Internal special variables
Special Variable, *xcatd-directory*: Internal special variables
Special Variable, *xcatd-lock*: Internal special variables
Special Variable, *xcatd-remotes*: Internal special variables

T
tail: Internal structures

Jump to:   *   +  
C   D   N   S   T  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   F   P   S   X  
Index Entry  Section

F
fifo: Internal structures

P
Package, xcat: The xcat package

S
Structure, fifo: Internal structures
System, xcat: The xcat system

X
xcat: The xcat system
xcat: The xcat package

Jump to:   F   P   S   X