5. Parts of Arla

Caution: This text just tries to give a general picture. For real info read the code. If you have any questions, mail arla-drinkers@stacken.kth.se.

How does arla work
The relation between Arlad and XFS
5.1 The life of a file
Tools and libs
5.4 The files in arlad/
pioctl and kafs

How does arla work

Arla consists of two parts, a userland process (arlad) and the kernel-module (xfs).

Arlad is written in user-space for simpler debugging (and less rebooting). As a uset space program arlad does not have the same limitations as if it would be written in the kernel. To avoid performance loss as much as possible, xfs is caching data.

xfs and arlad communicate with each other via a char-device-driver. There is a rpc-protocol currenly used specially written for this (`arlad/message.c')

xfs is written to be as simple as possible. Theoretically, xfs could be used by other user-space daemons to implement a file system. Some parts, such as syscalls, are arla-specific. These parts are designed to be as general as possible.

For example, xfs does not recognize which pioctl the user-level program calls, it just passes this information on to arlad.

The relation between Arlad and XFS

Userland --------- Edit file | Arlad | ------> Network | --------- ----|-----------------|[1]---- ------- ------- Kernel | VFS | <--[2]--> | XFS | ------- -------

[1] A char device (/dev/xfs0)

[2] Xfs provides a filesystem for the vfs-layer in

the operating system.

5.1 The life of a file

Step by step description of what happens during the creation of a file. The names are inspired of BSD-style VFS-layer but the idea is the same in most operating systems.

• The user decides to open a file.

• open(2) syscall is issued.

• The vfslayer sends a VOP_LOOKUP to xfs that is forwarded to arlad with a getnode() (seq-num 1).

• arlad tries to find the requested file and then, if found, sends an install_node to xfs by writing to the xfs character device.

• xfs inserts the node into the cache and returns from the device write.

• arlad sends a wakeup rpc message (seq-num 1) to xfs. If the return value is zero xfs tries to find the node in the cache, if not found it might have been flushed out of the cache and the whole thing is repeated.

• If a none-zero return value is returned, this value is sent as reply to the user. This way arla can decide what error message is returned, without xfs having support for each error.

• xfs now checks if it has the valid attributes. If the attributes are invalid, xfs will send a rpc message to arlad to refresh it.

• Since the user wanted to open the file, a getdata rpc message is sent from xfs to arlad. Now arlad fetches the files from the afs file server.

• Arlad stores the file in the file cache. All vnode operations will be done on this file. Now arlad sends a installdata to xfs.

• When xfs recives the installdata it looks up the node in the cache, and then it does a VOP_LOOKUP to find a vnode to the cachefile (and store it to keep it for future use).

• The same thing is done when the file is a directory, except that the directory is converted from the afs directory format to an operating system dependent format and stored in a file. xfs reads this file instead.

• If the directory is modified locally, write operations are done on the file obtained from the afs-server, and when done the newly changed file is converted and reinstalled.

• Now the user wants to read a file.

• read(2) system call is issued.

• A VOP_READ is sent to the from the vfs-layer to xfs.

• xfs checks if it has valid attributes/and data (and updates if needed). Now VOP_READ is simply performed on the stored vnode of the cachefile.

Tools and libs

What other tools does the arla suite consists of

libutil: util/libutil.a - A library for the most often used

modules like hashtable, double-linked list, logging functions, date-parsing, etc

rx: rx/librx.a - The library for the rx protocol

(see section 5.2 Rx protocol).

lwp: lwp/liblwp.a - The library for the lwp thread-package

(see section 5.3 LWP).

ydr: ydr/ydr - A stub generator that replaces rxgen.

rxkad: rxkad/librxkad.a - The rx Kerberos authentication package.

roken: lib/roken/libroken.a - The library that will unbreak

things that are missing or broken.

ko: lib/ko/libko.a - A library of functions that are arlad-core

related but also are useful for programs like vos, pts, fs, etc.

arlalib: appl/lib/libarlalib.a - A broken library that does all

the hard work with connections etc.

fs: appl/fs/fs - The fs util, extra feature

(amongst others): getfid.

vos: appl/vos/vos - The vos util.

pts: appl/pts/pts - The pts util, extra feature: dump.

udebug: appl/udebug/udebug - Debug your ubik server.

5.2 Rx protocol

Rx is run over UDP.

One of rxgen or ydr is used to generate stub-files, ydr is better since it generates prototypes, too.

The current implemetation of rx it not that beautiful.

5.3 LWP

LWP is a preepmtive thread package. It does it's context-switching by creating a private stack for each thread. The heart of the package is select(2).

The stack is checked for overruns in context-switches, but that is often too late. It might be an idea to add a red zone at the top of the stack to be able to detect overruns.

5.4 The files in arlad/

This is a short describtion of the files to bring new deveplopers up to speed.

5.4.1 The core of arlad

`adir.c' - contains all functions needed to to operations

on afs-directory files.

`afsdir_check.c' - check if an AFS-directory looks sane.

`arla.c' - The startup and the standalone (-t) code.

`arladeb.c' - The logging code specific to arla, like aliases

for debugging masks.

`cmcb.c' - The callback-server that is contacted by the

server when a callback expires or a server wants to send an InitCallBackState.

`conn.c' - The connection cache, responsible for caching connection

based on pag and security index. It will also create new connection when needed.

`cred.c' - Keep track of all credentials that all users have

inserted. Indexed on pag.

`fbuf.c' - An interface between rx and filedescriptors. It is also

used to mmap files. Used by `adir.c'.

`fcache.c' - Responsible for keeping track of files in the cache.

Also fetches files from the afs-server.

`fprio.c' - Tries to give files priority. These files are

therefore not garbarge-collected as fast as they would be otherwise. If you wonder what this is for, think of the disconnected mode.

`inter.c' - An interface to hide all junk in fcache, just give

the items a VenusFid and you can access them this way.

`kernel.c' - The interface between arlad and the char-device.

`messages.c' - The rpc interface between arlad and xfs.

`volcache.c' - Cache for all volumes.

5.4.2 Operating system specific files

These are the files that contain operating specific functions. Today it's just conv_dir().

`aix-subr.c' - AIX

`bsd-subr.c' - FreeBSD 2.2.6, OpenBSD 2.2, 2.3, NetBSD 1.3.x

`hpux-subr.c' - HPUX

`irix-subr.c' - Irix

`linux-subr.c' - Linux 2.0.x, 2.1.x, 2.2

`solaris-subr.c' - Solaris 2.5.x, 2.6, 7

`sunos-subr.c' - SunOS

`unknown-subr.c' - Stub used when compiled on a unknown OS.

pioctl and kafs

The pioctl interface is the only part of xfs that is afs related.

pioctl is a ioctl but called with a path instead of a filedescriptor. When you probe if there is a live afsclient you first run k_hasafs() that probes if there is an afsclient around. It also sets up some static variables in the library. So if you start to do pioctl() w/o running k_hasafs(), you're up to funny errors, and/or get a corefile.

k_hasafs() does an AFSCALL_PIOCTL with opcode VIOCSETTOK and insize == 0, ie you try to set a token (ticket) that is 0 bytes long. This is cleary invalid and kafs expects to find an EINVAL returned from syscall(2).

The pioctl is used more then just for AFSCALL_PIOCTL, an other use is AFSCALL_SETPAG (setting pag). It has also been in use for setting xfs debugging levels.

When xfs discovers that a path is given in the pioctl() it does a VOP_LOOKUP on the path and if the returned value is a vnode that resides in afs then it extracts the xfs-handle for that node (that just happens to be the VenusFid) and passes that on to arlad.

The only ugly thing about the current implentation is that the syscall code assumes that the arlad on "xfs-fd" is the arlad that should get this syscall.

An example of using pioctl():

int fs_getfilecellname(char *path, char *cell, size_t len) { struct ViceIoctl a_params; a_params.in_size=0; a_params.out_size=len; a_params.in=NULL; a_params.out=cell; if (k_pioctl(path,VIOC_FILE_CELL_NAME,&a_params,1) == -1) return errno; return 0; } int main (int argc, char **argv) { char cell[100]; if (!k_hasafs()) errx (1, "there is no afs"); if (fs_getfilecellname (".", cell, sizeof(cell))) errx (1, "fs_getfilecellname failed"); printf ("cell for `.' is %s", cell); return 0; }