|
|
The lightweight storage library |
libosl, the object storage layer, is a library for storing small to medium-sized data in relational tables. It is easy to use, lightweight, fast and portable. libosl is suitable for applications that need only a small fraction of the features a full database management system provides.
Libosl is simple:
For libosl, all data stored in the osl tables are blobs, libosl will never try to interpret the table contents.
There is no fancy query language but only a relatively simple C-API.
There’s no support for network-access and libosl only uses a very rudimentary table locking mechanism.
There are no dependencies on external libraries. An augmented version of the red-black tree implementation of the linux kernel is included in the libosl package.
Libosl is fast:
It uses sha1 hashes for content-based addressing.
Red-black trees are used to make lookups fast.
Libosl is portable:
It is known to compile and run on Linux, FreeBSD, NetBSD. It might work on other Unix systems as well.
Content and metadata is stored in an endian-agnostic way.
Libosl is open source:
Apart from the library itself, the package also contains simple examples and an fsck program, oslfsck, which can be used to recover from corrupted tables due to system crashes or bugs in the application or the library.
Resources:
Comments and bug reports are welcome. Please provide enough info such as the version of osl you are using and relevant parts of the logs.
Besides a working C compiler, gnu m4, gnu make and the lopsub library must be installed to to build and install this package. Download and install lopsub with
git clone git://git.tuebingen.mpg.de/lopsub
cd lopsub && make && sudo make install
If everything mentioned above is available on your system, type
make
to build libosl. Then type
sudo make install
to install. Finally, update the library cache by running
ldconfig
The default installation prefix is /usr/local. Use
make install PREFIX=/somewhere/else
to install in /somewhere/else instead.
The make targets in the web/ directory are used for web page generation
and are not needed to compile, install or use libosl. In order to
make these targets the following additional tools are needed:
The osl logo was created with Adobe Illustrator by Sebastian Schultheiss.
This document describes the steps which have to be performed in order
to create and use an osl table. The code sniplets in this section
are taken from the file osltar.c in the source distribution.
The complete API reference is contained in the file osl.h.
Define an enum that assigns descriptive names to the columns of your table. Example:
enum osltar_columns {
OTC_NAME,
OTC_DATA,
NUM_OT_COLUMNS
};
The last element is is useful because the number of columns of your table must be specified later, see below.
Define an array of struct osl_column_description, one array member per column:
struct osl_column_description tar_table_cols[] = {
[OTC_NAME] = {
.storage_type = OSL_MAPPED_STORAGE,
.storage_flags = OSL_RBTREE | OSL_UNIQUE,
.name = "filename",
.compare_function = string_compare,
},
[OTC_DATA] = {
.storage_type = OSL_MAPPED_STORAGE,
.name = "data",
},
};
Three different storage types are available which may be selected
on a per-column basis: OSL_DISK_STORAGE, OSL_MAPPED_STORAGE, and
OSL_NO_STORAGE.
For columns of type OSL_MAPPED_STORAGE and OSL_NO_STORAGE an
optional rbtree is maintained by the osl library which allows to
quickly lookup rows by cell content. Whether or not an rbtree should
be used must be specified in the storage_flags field which should
contain the bitwise or of suitable osl_storage_flags.
If a column has an associated rbtree, i.e. if the OSL_RBTREE flag
is set in the storage flags for the column, the compare_function
field must be initialized to point to a function of type
osl_compare_func. In this example, string_compare() is used,
which is just a wrapper for strcmp() that interprets osl objects
as C-strings and calls strcmp() on the object data.
Define a struct osl_table_description and initialize it with
the number of columns of your table and the column descriptions:
struct osl_table_description tar_table_desc = {
.name = "tar_table",
.num_columns = NUM_OT_COLUMNS,
.column_descriptions = tar_table_cols,
.dir = "/tmp/osltest"
};
Create the table by calling osl_create_table():
ret = osl_create_table(&tar_table_desc);
Open the newly created table by calling osl_open_table():
struct osl_table *table;
ret = osl_open_table(&tar_table_desc, &table);
To add a new row to the table, you must define an array of struct
osl_object of length NUM_OT_COLUMNS which holds the contents of
the new row. Note that an osl object is just a blob: It consists of
a data pointer and a size value. Once the array has been initialized,
pass it to osl_add_row() together with the table handle obtained from
osl_open_table():
struct osl_object objs[NUM_OT_COLUMNS];
/* ...init the array... */
ret = osl_add_row(table, objs);
Close the table with osl_close_table().
osl_close_table(table, OSL_MARK_CLEAN);
The storage type of both columns of the table in this example is
OSL_MAPPED_STORAGE, so you can later open the table again and
retrieve its contents:
ret = osl_get_row(table, OTC_NAME, &obj, &row);
if (ret < 0) {
fprintf(stderr, "osl_get_row(%s): %s\n", name,
osl_strerror(-ret));
return ret;
}
ret = osl_get_object(table, row, OTC_DATA, &obj);
The call to osl_get_row() uses the rbtree of the OTC_NAME
column to find the row whose object in the OTC_NAME column
matches the given object obj. If a row was found, it is
passed to osl_get_object() which returns the object of
the OTC_DATA column of this row.
This concludes the quick start document. Of course, libosl contains
many more public functions than those used above. For details on the
C-API, look at the file osl.h which contains the declarations of
all public functions and the complete documentation of the public
part of the library.
The “examples” subdirectory of the source distribution
contains the full code of the above example and another
small program which illustrates the use of columns of type
OSL_NO_STORAGE. Larger applications using libosl are paraslash,
a network audio streaming system, and adu, the advanced
disk usage tool.
oslfsck - check and repair the tables of an osl database
oslfsck [--help] [--detailed-help] [--version] [--loglevel=<severity>] [--database-dir=<path>] [--dump-dir=<path>] [--no-fsck] [--force] [--dry-run] [--] [table]...
oslfsck tries to recover the tables of an osl database after a crash which potentially leaves the database in an inconsistent state.
-h, --help
print help and exit
--detailed-help
print help, including all details, and exit
-V, --version
print version and exit
-l, --loglevel=<severity>
set loglevel (0-6)
default: 3
Log messages are always written to stderr while normal output goes to stdout. Lower values mean more verbose logging.
-d, --database-dir=<path>
full path to the database directory
Unless non-option arguments are given, all subdirectories of <path> are considered osl tables which oslfsck will try to fix.
-D, --dump-dir=<path>
enable dump mode
If path is non-empty, oslfsck will write a dump of all given tables to the specified path.
-n, --no-fsck
disable fsck mode
This is mainly useful in conjunction with the --dump-dir option.
-f, --force
enable force mode
Ignore the dirty bit when opening osl tables.
--dry-run
only report problems, don’t try to fix them