“Doesn’t it sound like a mechanical engineering subject: The design of the hard disk?” questioned Shweta. “Yes, it does. But understanding it gives us an insight into its programming aspect,” reasoned Pugs, while waiting for the commencement of the seminar on storage systems.

The seminar started with a few hard disks in the presenter’s hand and then a dive into her system, showing the output of fdisk -l (Figure 1).

Figure 1: Partition listing by fdisk

The first line shows the hard disk size in human-friendly format and in bytes. The second line mentions the number of logical heads, logical sectors per track, and the actual number of cylinders on the disk — together known as the geometry of the disk.

The 255 heads indicate the number of platters or disks, as one read-write head is needed per disk. Let’s number them, say D1, D2, … D255. Now, each disk would have the same number of concentric circular tracks, starting from the outside to the inside. In the above case, there are 60,801 such tracks per disk. Let’s number them, say T1, T2, … T60801. And a particular track number from all the disks forms a cylinder of the same number. For example, tracks T2 from D1, D2, … D255 will together form the cylinder C2. Now, each track has the same number of logical sectors — 63 in our case, say S1, S2, … S63. And each sector is typically 512 bytes. Given this data, one can actually compute the total usable hard disk size, using the following formula:

Usable hard disk size in bytes = (Number of heads or disks) * (Number of tracks per disk) * (Number of sectors per track) * (Number of bytes per sector, i.e. sector size)

For the disk under consideration, it would be: 255 * 60801 * 63 * 512 bytes = 500105249280 bytes.

Note that this number may be slightly less than the actual hard disk (500107862016 bytes, in our case). The reason is that the formula doesn’t consider the bytes in the last partial or incomplete cylinder. The primary reason for that is the difference between today’s technology of organising the actual physical disk geometry and the traditional geometry representation using heads, cylinders and sectors.

Note that in the fdisk output, we referred to the heads and sectors per track as logical not physical. One may ask that if today’s disks don’t have such physical geometry concepts, then why still maintain them and represent them in a logical form? The main reason is to be able to continue with the same concepts of partitioning, and be able to maintain the same partition table formats, especially for the most prevalent DOS-type partition tables, which heavily depend on this simplistic geometry. Note the computation of cylinder size (255 heads * 63 sectors / track * 512 bytes / sector = 8225280 bytes) in the third line and then the demarcation of partitions in units of complete cylinders.

DOS-type partition tables

This brings us to the next important topic: understanding DOS-type partition tables. But first, what is a partition, and why should we partition? A hard disk can be divided into one or more logical disks, each of which is called a partition. This is useful for organising different types of data separately, for example, different operating system data, user data, temporary data, etc.

So, partitions are basically logical divisions and need to be maintained by metadata, which is the partition table. A DOS-type partition table contains four partition entries, each a 16-byte entry. Each of these entries can be depicted by the following ‘C’ structure:

typedef struct
{
    unsigned char boot_type; // 0x00 - Inactive; 0x80 - Active (Bootable)
    unsigned char start_head;
    unsigned char start_sec:6;
    unsigned char start_cyl_hi:2;
    unsigned char start_cyl;
    unsigned char part_type;
    unsigned char end_head;
    unsigned char end_sec:6;
    unsigned char end_cyl_hi:2;
    unsigned char end_cyl;
    unsigned long abs_start_sec;
    unsigned long sec_in_part;
} PartEntry;

This partition table, followed by the two-byte signature 0xAA55, resides at the end of the disk’s first sector, commonly known as the Master Boot Record (MBR). Hence, the starting offset of this partition table within the MBR is 512 - (4 * 16 + 2) = 446. Also, a 4-byte disk signature is placed at offset 440.

The remaining top 440 bytes of the MBR are typically used to place the first piece of boot code, that is loaded by the BIOS to boot the system from the disk. The part_info.c listing contains these various definitions, along with code for parsing and printing a formatted output of the partition table.

From the partition table entry structure, it could be noted that the start and end cylinder fields are only 10 bits long, thus allowing a maximum of 1023 cylinders only. However, for today’s huge hard disks, this is in no way sufficient. Hence, in overflow cases, the corresponding <head, cylinder, sector> triplet in the partition table entry is set to the maximum value, and the actual value is computed using the last two fields: the absolute start sector number (abs_start_sec) and the number of sectors in this partition (sec_in_part).

The code for this too is in part_info.c:

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define SECTOR_SIZE 512
#define MBR_SIZE SECTOR_SIZE
#define MBR_DISK_SIGNATURE_OFFSET 440
#define MBR_DISK_SIGNATURE_SIZE 4
#define PARTITION_TABLE_OFFSET 446
#define PARTITION_ENTRY_SIZE 16 // sizeof(PartEntry)
#define PARTITION_TABLE_SIZE 64 // sizeof(PartTable)
#define MBR_SIGNATURE_OFFSET 510
#define MBR_SIGNATURE_SIZE 2
#define MBR_SIGNATURE 0xAA55
#define BR_SIZE SECTOR_SIZE
#define BR_SIGNATURE_OFFSET 510
#define BR_SIGNATURE_SIZE 2
#define BR_SIGNATURE 0xAA55

typedef struct {
    unsigned char boot_type; // 0x00 - Inactive; 0x80 - Active (Bootable)
    unsigned char start_head;
    unsigned char start_sec:6;
    unsigned char start_cyl_hi:2;
    unsigned char start_cyl;
    unsigned char part_type;
    unsigned char end_head;
    unsigned char end_sec:6;
    unsigned char end_cyl_hi:2;
    unsigned char end_cyl;
    unsigned long abs_start_sec;
    unsigned long sec_in_part;
} PartEntry;

typedef struct {
    unsigned char boot_code[MBR_DISK_SIGNATURE_OFFSET];
    unsigned long disk_signature;
    unsigned short pad;
    unsigned char pt[PARTITION_TABLE_SIZE];
    unsigned short signature;
} MBR;

void print_computed(unsigned long sector) {
    unsigned long heads, cyls, tracks, sectors;

    sectors = sector % 63 + 1 /* As indexed from 1 */;
    tracks = sector / 63;
    cyls = tracks / 255 + 1 /* As indexed from 1 */;
    heads = tracks % 255;
    printf("(%3d/%5d/%1d)", heads, cyls, sectors);
}

int main(int argc, char *argv[]) {
    char *dev_file = "/dev/sda";
    int fd, i, rd_val;
    MBR m;
    PartEntry *p = (PartEntry *)(m.pt);

    if (argc == 2) {
        dev_file = argv[1];
    }
    if ((fd = open(dev_file, O_RDONLY)) == -1) {
        fprintf(stderr, "Failed opening %s: ", dev_file);
        perror("");
        return 1;
    }
    if ((rd_val = read(fd, &m, sizeof(m))) != sizeof(m)) {
        fprintf(stderr, "Failed reading %s: ", dev_file);
        perror("");
        close(fd);
        return 2;
    }
    close(fd);
    printf("\nDOS type Partition Table of %s:\n", dev_file);
    printf("  B Start (H/C/S)   End (H/C/S) Type  StartSec    TotSec\n");
    for (i = 0; i < 4; i++) {
        printf("%d:%d (%3d/%4d/%2d) (%3d/%4d/%2d)  %02X %10d %9d\n",
            i + 1, !!(p[i].boot_type & 0x80),
            p[i].start_head,
            1 + ((p[i].start_cyl_hi << 8) | p[i].start_cyl),
            p[i].start_sec,
            p[i].end_head,
            1 + ((p[i].end_cyl_hi << 8) | p[i].end_cyl),
            p[i].end_sec,
            p[i].part_type,
            p[i].abs_start_sec, p[i].sec_in_part);
    }
    printf("\nRe-computed Partition Table of %s:\n", dev_file);
    printf("  B Start (H/C/S)   End (H/C/S) Type  StartSec    TotSec\n");
    for (i = 0; i < 4; i++) {
        printf("%d:%d ", i + 1, !!(p[i].boot_type & 0x80));
        print_computed(p[i].abs_start_sec);
        printf(" ");
        print_computed(p[i].abs_start_sec + p[i].sec_in_part - 1);
        printf(" %02X %10d %9d\n", p[i].part_type,
            p[i].abs_start_sec, p[i].sec_in_part);
    }
    printf("\n");
    return 0;
}

As the above is an application, compile it with gcc part_info.c -o part_info, and then run ./part_info /dev/sda to check out your primary partitioning information on /dev/sda. Figure 2 shows the output of ./part_info on the presenter’s system. Compare it with the fdisk output in Figure 1.

Figure 2: Output of ./part_info

Partition types and boot records

Now, as this partition table is hard-coded to have four entries, that’s the maximum number of partitions you can have. These are called primary partitions, each having an associated type in the corresponding partition table entry. These types are typically coined by various OS vendors, and hence sort of map to various OSs like DOS, Minix, Linux, Solaris, BSD, FreeBSD, QNX, W95, Novell Netware, etc., to be used for/with the particular OS. However, this is more a formality than a real requirement.

Besides this, one of the four primary partitions can be labelled as something called an extended partition, which has a special significance. As the name suggests, it is used to further extend hard disk division, i.e., to have more partitions. These are called logical partitions and are created within the extended partition. The metadata of these is maintained in a linked-list format, allowing an unlimited number of logical partitions (at least theoretically).

For that, the first sector of the extended partition, commonly called the Boot Record (BR), is used like the MBR to store (the linked-list head of) the partition table for the logical partitions. Subsequent linked-list nodes are stored in the first sector of the subsequent logical partitions, referred to as the Logical Boot Record (LBR). Each linked-list node is a complete 4-entry partition table, though only the first two entries are used — the first for the linked-list data, namely, information about the immediate logical partition, and the second as the linked list’s next pointer, pointing to the list of remaining logical partitions.

To compare and understand the primary partitioning details on your system’s hard disk, follow the steps (as the root user — hence with care) given below:

./part_info /dev/sda ## Displays the partition table on /dev/sda
fdisk -l /dev/sda ## To display and compare the partition table entries with the above

In case you have multiple hard disks (/dev/sdb, …), hard disk device files with other names (/dev/hda, ……), or an extended partition, you may try ./part_info <device_file_name> on them as well. Trying on an extended partition would give you the information about the starting partition table of the logical partitions.

Right now, we have carefully and selectively played (read-only) with the system’s hard disk. Why carefully? Since otherwise, we may render our system non-bootable. But no learning is complete without a total exploration. Hence, in our next session, we will create a dummy disk in RAM and do destructive exploration on it.

In rare moments of self-reflection, when I allow myself to doubt my skills as a Lisp evangelist, I sometimes wonder if I have left behind some of my fellow programmers who favour the object-oriented style of programming. Just because I have been focusing on Lisp as a functional programming paradigm, it doesn’t mean we don’t have a role for you in our plans of world domination. Read on to know where you fit in.

Functional vs object-oriented (OO) programming

With an OO approach, programmers write code that describes in exacting detail the steps that the computer must take to accomplish the goal. They focus on how to perform tasks, and how to track changes in state. They would use loops, conditions and method calls as their primary flow control, and instances of structures or classes as primary manipulation units. OO tries to control state behind object interfaces.

In contrast, functional programming (FP) involves composing the problem as a set of functions to be executed. FP programmers focus on what information is desired and what transformations are required, by carefully defining the input to each function and what each function returns. They would use function calls, including recursion, as their primary flow control, functions as first-class objects and data collections as primary manipulation units. FP tries to minimise state by using pure functions as much as possible.

According to Michael Feathers: “OO makes code understandable by encapsulating moving parts. FP makes code understandable by minimising moving parts.”

Conard Barski points out that the critics of the OO programming style may complain that object-oriented techniques force data to be hidden away in a lot of disparate places by requiring them to live inside many different objects. Having data located in disparate places can make programs difficult to understand, especially if that data changes over time.

Therefore, many Lispers prefer to use functional techniques over object-oriented techniques, though the two can often be used together — with some care. Nonetheless, there are still many domains in which object-oriented techniques are invaluable, such as in user interface programming or simulation programming.

On the other hand, James Hague, in his assessment of functional programming argues that, “100 per cent pure functional programming doesn’t work. Even 98 per cent pure functional programming doesn’t work. But if the slider between functional purity and 1980s BASIC-style imperative messiness is kicked down a few notches — say to 85 per cent — then it really does work. You get all the advantages of functional programming, but without the extreme mental effort and un-maintainability that increases as you get closer and closer to perfectly pure.”

CLOS

If OO is what gets you going, Common Lisp offers the most sophisticated object-oriented programming framework of any major programming language. It’s called Common Lisp Object System (CLOS). It is customisable at a fundamental level, using the Meta-Object Protocol (MOP). It has been claimed that there’s really nothing like it anywhere else in programming. It lets you control incredibly complex software without losing control over the code.

Let the tears of joy flow…

Object-oriented programming in Common Lisp

What is CLOS?

#1: It is a layered system designed for flexibility.

One of the design goals of CLOS is to provide a set of layers that separate different programming language concerns from one another. The first level of the Object System provides a programmatic interface to object-oriented programming. This level is designed to meet the needs of most serious users, and to provide a syntax that is crisp and understandable.

The second level provides a functional interface into the heart of the Object System. This level is intended for programmers who are writing very complex software or a programming environment. The first level is written in terms of this second level.

The third level provides the tools for programmers who are writing their own object-oriented language. It allows access to the primitive objects and operators of the Object System. It is this level at which the implementation of the Object System itself is based.

The layered design of CLOS is founded on the meta-object protocol, a protocol that is used to define the characteristics of an object-oriented system. Using the meta-object protocol, other functional or programmatic interfaces to the Object System, as well as other object systems, can be written.

#2: It is based on the concept of generic functions rather than on message-passing.

This choice is made for two reasons:

1. there are some problems with message-passing in operations of more than one argument;
2. the concept of generic functions is a generalisation of the concept of ordinary Lisp functions.

A key concept in object-oriented systems is that given an operation and a tuple of objects on which to apply the operation, the code that is most appropriate to perform the operation is selected, based on the classes of the objects.

In most message-passing systems, operations are essentially properties of classes, and this selection is made by packaging a message that specifies the operation and the objects to which it applies, before sending that message to a suitable object. That object then takes responsibility for selecting the appropriate piece of code. These pieces of code are called methods.

#3: It is a multiple inheritance system.

Another key concept in object-oriented programming is the definition of structure and behaviour on the basis of the class of an object. Classes thus impose a type system — the code that is used to execute operations on objects depends on the classes of the objects. The sub-class mechanism allows classes to be defined that share the structure and the behaviour of other classes. This sub-classing is a tool for modularisation of programs.

#4: It provides a powerful method combination facility.

Method combination is used to define how the methods that are applicable to a set of arguments can be combined to provide the values of a generic function. In many object-oriented systems, the most specific applicable method is invoked, and that method may invoke other, less specific methods.

When this happens, there is often a combination strategy at work, but that strategy is distributed throughout the methods as local control structure. Method combination brings the notion of a combination strategy to the surface, and provides a mechanism for expressing that strategy.

#5: The primary entities of the system are all first-class objects.

In the Common Lisp Object System, generic functions and classes are first-class objects with no intrinsic names. It is possible and useful to create and manipulate anonymous generic functions and classes. The concept of “first-class” is important in Lisp-like languages. A first-class object is one that can be explicitly made and manipulated; it can be stored in any location that can hold general objects.

What CLOS is not

It does not make for a great pickup conversation at the bar. I tried. It did not work!

It also does not attempt to solve problems of encapsulation. The inherited structure of a class depends on the names of the internal parts of the classes from which it inherits. CLOS does not support subtractive inheritance. Within Common Lisp, there is a primitive module system that can be used to help create separate internal namespaces.

Classes

The defclass macro is used to define a new class. The definition of a class consists of its name, a list of its direct super-classes, a set of slot specifiers and a set of class options. The direct super-classes of a class are those from which the new class inherits structure and behaviour. When a class is defined, the order in which its direct super-classes are mentioned in the defclass form defines a local precedence order on the class and those super-classes. The local precedence order is represented as a list consisting of the class, followed by its direct super-classes, in the order mentioned in the defclass form. The following two classes define a representation of a point in space. The x-y-position class is a sub-class of the position class:

> (defclass position () ())

> (defclass x-y-position (position)
      ((x :initform 0)
       (y :initform 0))
     (:accessor-prefix position-))

The position class is useful if we want to create other sorts of representations for spatial positions. The x and y coordinates are initialised to 0 in all instances, unless explicit values are supplied for them. To refer to the x coordinate of an instance of x-y-position, you would write:

> (position-x position)

To alter the x coordinate of that instance, you would write:

(setf (position-x position) new-x)

The macro defclass is part of the Object System programmatic interface and, as such, is on the first of the three levels of the Object System.

Generic functions

The class-specific operations of the Common Lisp Object System are provided by generic functions and methods. A generic function is one whose behaviour depends on the classes or identities of the arguments supplied to it. The methods associated with the generic function define the class-specific operations of the generic function.

Like an ordinary Lisp function, a generic function takes arguments, performs a series of operations and returns values. An ordinary function has a single body of code that is always executed when the function is called. A generic function is able to perform different series of operations and to combine the results of the operations in different ways, depending on the class or identity of one or more of its arguments.

Generic functions are defined by means of the defgeneric-options and defmethod macros. The defgeneric-options macro is designed to allow for the specification of properties that pertain to the generic function as a whole, and not just to individual methods. The defmethod form is used to define a method. If there is no generic function of the given name, however, it automatically creates a generic function with default values for the argument precedence order (left-to-right, as defined by the lambda-list), the generic function class (the class standard-generic-function), the method class (the class standard-method) and the method combination type (standard-method combination).

Methods

The class-specific operations provided by generic functions are themselves defined and implemented by methods. The class or identity of each argument to the generic function indicates which method or methods are eligible to be invoked.

A method object contains a method function, an ordered set of parameter specialisers that specify when the given method is applicable, and an ordered set of qualifiers that are used by the method combination facility to distinguish between methods.

The defmethod macro is used to create a method object. A defmethod form contains the code that is to be run when the arguments to the generic function cause the method that it defines, to be selected. If a defmethod form is evaluated, and a method object corresponding to the given generic function name, parameter specialisers and qualifiers already exists, then the new definition replaces the old.

Generic functions can be used to implement a layer of abstraction on top of a set of classes. For example, the x-y-position class can be viewed as containing information in polar coordinates.

Two methods have been defined — position-rho and position-theta, that calculate the ρ and Θ coordinates given an instance of x-y-position:

> (defmethod position-rho ((pos x-y-position))
      (let ((x (position-x pos))
            (y (position-y pos)))
         (sqrt (+ (* x x) (* y y)))))

> (defmethod position-theta ((pos x-y-position))
     (atan (position-y pos) (position-x pos)))

It is also possible to write methods that update the “virtual slots” position-rho and position-theta:

> (defmethod-setf position-rho ((pos x-y-position)) (rho)
      (let* ((r (position-rho pos))
           (ratio (/ rho r)))
        (setf (position-x pos) (* ratio (position-x pos)))
        (setf (position-y pos) (* ratio (position-y pos)))))

> (defmethod-setf position-theta ((pos x-y-position)) (theta)
      (let ((rho (position-rho pos)))
       (setf (position-x pos) (* rho (cos theta)))
       (setf (position-y pos) (* rho (sin theta)))))

To update the ρ-coordinate you may write:

> (setf (position-rho pos) new-rho)

This is precisely the same syntax that would be used if the positions were explicitly stored as polar coordinates.

Class redefinition

The Common Lisp Object System provides a powerful class-redefinition facility.

When a defclass form is evaluated, and a class with the given name already exists, the existing class is redefined. Redefining a class modifies the existing class object to reflect the new class definition.

You may define methods on the generic function class-changed to control the class redefinition process. This generic function is invoked automatically by the system after defclass has been used to redefine an existing class; for example, suppose it becomes apparent that the application that requires representing positions uses polar coordinates more than it uses rectangular coordinates. It might make sense to define a sub-class of position that uses polar coordinates:

> (defclass rho-theta-position (position)
      ((rho :initform 0)
      (theta :initform 0))
    (:accessor-prefix position-))

The instances of x-y-position can be automatically updated by defining a class-changed method:

> (defmethod class-changed ((old x-y-position)
                          (new rho-theta-position))
;; Copy the position information from old to new to make new
;; be a rho-theta-position at the same position as old.
     (let ((x (position-x old))
           (y (position-y old)))
        (setf (position-rho new) (sqrt (+ (* x x) (* y y)))
              (position-theta new) (atan y x))))

At this point, we can change an instance of the class x-y-position, p1, to be an instance of rho-theta-position by using change-class:

> (change-class p1 'rho-theta-position)

Inheritance

Inheritance is the key to program modularity within CLOS. A typical object-oriented program consists of several classes, each of which defines some aspect of behaviour. New classes are defined by including the appropriate classes as super-classes, thus gathering the desired aspects of behaviour into one class.

In general, slot descriptions are inherited by sub-classes. That is, slots defined by a class are usually slots implicitly defined by any sub-class of that class, unless the sub-class explicitly shadows the slot definition. A class can also shadow some of the slot options declared in the defclass form of one of its super-classes by providing its own description for that slot.

A sub-class inherits methods in the sense that any method applicable to an instance of a class is also applicable to instances of any sub-class of that class (all other arguments to the method being the same).

The inheritance of methods acts the same way regardless of whether the method was created by using defmethod or by using one of the defclass options that cause methods to be generated automatically.

I hope with this article I have managed to convince OO programmers that Lisp is generous enough to cater to your style of thinking. Stick with me, and I promise that you won’t be disappointed. So far we’ve seen how to fit the nuts and bolts into the engine. Next month, we’ll learn how to paint it a nice shiny red… I am referring to Graphical Programming in Lisp!

References

• Let Over Lambda, Doug Hoyte
• CLOS: Integrating Object-Oriented and Functional programming, Richard P. Gabriel, Jon L White, Daniel G. Bobrow

There was a time when all desktop applications were developed from scratch. Then came the concept of code reuse. Static and shared libraries were created for use in application development, but developers didn’t stop at that; they came up with software frameworks. Though libraries and frameworks seem to functionally be the same, there are some major differences between them, which should be understood:

• Libraries offer reusability of functionality, whereas frameworks offer reusability of behaviour. For example, a library may provide classes for TCP and UDP sockets, while a framework will provide a class for an abstract socket.
• A library may provide functions used for signals/events, but the framework function is how they interact with the system and other components.
• Libraries are called from application code, but the framework calls application code — or, you can say, provides services to the code.

Desktop applications

Desktop applications are very platform-specific. An application compiled for Linux cannot be directly executed on another OS without some sort of emulation, due to differences in system calls and libraries between OSs. A big question was how to write platform-independent applications.

One method was to develop libraries for each platform, keeping application code the same and recompiling for each target platform. This does make life easier for developers — but then everything changed with the revolutionary concept of virtual machines from Sun Microsystems, which gave the world the platform-independent Java programming language. Java applications run on the Java Virtual Machine (JVM). Code developed once can be deployed to every platform that has a JVM for it.

But everything comes with a price. In Java’s case, the price was application performance. The performance of Java applications is not as good as of C/C++ applications compiled to platform-native code. Another problem is the large memory footprint. No doubt Java is still a leading language, but these days we have some really big data applications (for example, in biotech) with very high performance requirements. So again, the need is to develop applications compilable to native code. Application development using C is time-consuming, while C++ is a better option. We have some frameworks that support C++ for application development. The Qt framework is one of them.

Introducing Qt

Qt is a cross-platform application framework developed by Trolltech and presently owned by Nokia. Its APIs are for C++. Qt has been extensively used by application developers to develop cross-platform applications.

Qt can help with graphical application development, network applications, database and multimedia applications, handling XML and 3D, painting, drawing and Web access. As far as platform support is concerned, it supports Linux, Mac OS, Windows, Meego, Embedded Linux and Symbian.

Qt architecture

In Figure 1, you can see a minimal architecture diagram.

Figure 1: Qt architecture

The top layer is C++ program code. Below that are Qt classes for GUI, WebKit, databases, etc., and then an OS-specific support layer. Earlier, Qt also supported Java; its Java-based version was called Jambie. As Qt development progressed, it was getting difficult to support both C++ and Java, so the decision was made to support only C++.

Qt installation

Installation methods differ by OS. On Ubuntu 10.04 LTS, I installed Qt via the Synaptic package manager — install the qtcreator package; dependency resolution will install Qt Assistant, Qt Designer, Qt Linguist and Qt Creator.

If you want to try out the latest Qt release, you can download the offline installer from their official website and install it. Just chmod the installer file to make it executable and run it, and then follow the prompts.

A ‘Hello World’ program (non-GUI)

Open a terminal. Create a directory (first) and create a simple “Hello World” program with the following code:

#include<QtCore>
int main(){
    qDebug() << "Hello world\n";
}

The included header file QtCore contains declarations for classes that do not use a GUI. We will look at these in detail in later articles. For now, just use the qDebug class, which outputs debugging messages to the console and is equivalent to cout in traditional C++ programming.

So how does one compile it? The Trolltech people came out with an easy solution to support cross-platform compilation. First, a project file should be created, and then a Makefile is created using it. Then, just run make to compile the program. When you install QT Creator, a utility called qmake is also installed. This is a cross-platform Makefile generator for Qt. Check out its man pages for more. Run qmake -project to create a project file (.pro extension, with its name that of the containing directory, i.e., first).

$ cat first.pro
######################################################################
# Automatically generated by qmake (2.01a) Mon Nov 28 05:49:29 2011
######################################################################

TEMPLATE = app
TARGET =
DEPENDPATH += .
INCLUDEPATH += .

# Input
SOURCES += main.cpp

You can create both applications and libraries; the value of TEMPLATE is app, indicating this is an application. I will cover library development in later articles. The SOURCES entry lists source files in the project, about which  more details appear later in this article series.

Now, let us generate the Makefile with qmake. It’s a long file — it may contain up to 200 lines. Run make to create the executable file:

$ make
g++ -c -pipe -O2 -Wall -W -D_REENTRANT -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++ -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtGui -I/usr/include/qt4 -I. -I. -o main.o main.cpp
make: Circular all <- first dependency dropped.
g++ -Wl,-O1 -o first main.o    -L/usr/lib -lQtGui -lQtCore -lpthread

In the last line, you can see that our sample program uses the POSIX thread library too. Run the file with ./first to see the output “Hello world”. It’s done! We have successfully compiled our first program.

Sample GUI program

Qt has an option to create UI files using a drag-and-drop method, which we will explore in the next article. For now, let us hand-code a sample simple GUI program:

#include<QApplication>
#include<QLabel>

int main(int argc, char *argv[]){
    QApplication a(argc, argv);
    QLabel    label;
    label.setText("Hello World");
    label.show();
    a.exec();
}

In the first example, there was no GUI. The program terminates when main returns. However, in GUI programs, we can’t do that, or the application will not be usable. We want the GUI to run until the user closes the window. To achieve this, run your program in a loop till this happens, so you use the event-loop-based class QApplication. When you create an object of this class and call its exec() function, main never returns. The application keeps on waiting for user input events.

The second header file contains the class QLabel, a simple widget used to display text. Instantiate it and set its text to “Hello World”. When a widget’s show() function is called, then it becomes a window. So the widget label will be seen like a window. In Figure 2, you can see the resultant label window with title bar. You can resize the output window simply with the help of the mouse.

Figure 2: Widget label

In the next articles, we will cover the core classes of Qt. In the meanwhile, I suggest you go through the official documentation.

MTD (Memory Technology Devices) are NAND/NOR-based flash memory chips used for storing non-volatile data like boot images and configurations. Readers are cautioned not to get confused with USB sticks, SD cards, etc., which are also called flash devices, but are not MTD devices. The latter are generally found on development boards, used to store boot loaders, an OS, etc.

Even though MTD devices are for data storage, they differ from hard disks and RAM in several aspects. The biggest difference is that while hard disk sectors are rewritable, MTD device sectors must be erased before rewriting — which is why they are more commonly called erase-blocks. Second, hard disk sectors can be rewritten several times without wearing out the hardware, but MTD device sectors have a limited life and are not usable after about 10^3-10^5 erase operations. The worn out erase-blocks are called bad blocks and the software must take care not to use such blocks.

Like hard disks, MTD devices can be partitioned and can therefore act as independent devices. On a system with one or more MTD devices, device and partition information can be obtained from the /proc/mtd file. A typical /proc/mtd file is as follows:

cat /proc/mtd
dev:  size    erasesize name
mtd0: 000a0000 00020000 "misc"
mtd1: 00420000 00020000 "recovery"
mtd2: 002c0000 00020000 "boot"
mtd3: 0fa00000 00020000 "system"
mtd4: 02800000 00020000 "cache"
mtd5: 0af20000 00020000 "userdata"

A partitioned MTD device can be depicted as in Figure 1, which shows the relation between an MTD device, a partition and a sector.

Figure 1: An MTD device

As already said, MTD write operations are different from usual storage devices. Therefore, before we move further, let’s understand how write operations take place on MTD devices. Figure 2 shows a typical write case.

Figure 2: An MTD write operation

The left-most part shows a sector that has some data at the end. The rest of the sector has not been written since the last erase. A user wants to write “new data 1″ to this sector at offset 0. Since this part of the sector has already been erased, it is ready to be written and so “new data 1″ can be directly written to the sector. Later, the user may want to write “new data 2″, again at offset 0. To do this, the sector must be erased. Since the sector needs to be erased in entirety, the “old data” must be backed up in a temporary buffer. After erasing the complete sector, the “new data 2″ and “old data” must be written at appropriate offsets.

This procedure is the reason there are specific file systems for MTD devices, like JFFS2 and YAFFFS, and flash translation layers (FTL) like NFTL, INFTL, etc. These FTLs and file systems take special care of MTD device properties to hide complexity from the user.

In the first section that follows, we will look at how to access, read/write and erase MTD devices from Linux applications. The second section describes the same things in kernel space, so that this article can be useful to both application as well as kernel developers.

Accessing MTDs from applications

The user must know the device partition to work upon, which can be found from /proc/mtd as shown earlier. Assuming users want to work on the “userdata” partition, they must use the /dev/mtd5 device.

The first thing to do is to get information about the MTD device. Use the MEMGETINFO ioctl command, as follows:

#include <stdio.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <mtd/mtd-user.h>

int main()
{
    mtd_info_t mtd_info;
    int fd = open("/dev/mtd5", O_RDWR);
ioctl(fd, MEMGETINFO, &mtd_info);

    printf("MTD type: %u\n", mtd_info.type);
    printf("MTD total size : %u bytes\n", mtd_info.size);
    printf("MTD erase size : %u bytes\n", mtd_info.erasesize);

    return 0;
}

Error handling has been omitted for brevity. The mtd_info_t structure is used with the MEMGETINFO command. The MTD type can be MTD_ABSENT, MTD_RAM, MTD_ROM, MTD_NAND, MTD_NOR, etc., which are defined in the mtd/mtd-abi.h header file. The mtd_info.size indicates the size of the whole device (i.e., the partition, in this case). Finally, mtd_info.erasesize indicates the sector size. During an erase operation, this is the minimum size that can be erased, as we’ll see later.

Reading MTD devices is similar to ordinary devices:

/* read something from last sector */
unsigned char buf[64];
lseek(fd, -mtd_info.erasesize, SEEK_END);
read(fd, buf, sizeof(buf));

A write operation can be performed in the same way, provided the sector has been erased previously. Finally, we come to the erase operation. Here is an example of erasing a partition, sector by sector:

void erase_partition(mtd_info_t *mtd_info, int fd) {
    erase_info_t ei;
    ei.length = mtd_info->erasesize;
 
    for(ei.start = 0; ei.start < mtd_info->size; ei.start += mtd_info->erasesize) {
        ioctl(fd, MEMUNLOCK, &ei);
        ioctl(fd, MEMERASE, &ei);
    }
}

All sectors of the device are writeable after this erase operation. Notice the use of MEMUNLOCK before MEMERASE, which is essential to allow the erase operation.

Accessing MTDs from kernel space

This section will repeat the functions explained in the previous section, but in kernel space. This needs a separate section since the erase operation is more complex here  –  the erase operation may sleep and therefore the kernel programmer has to wait until the operation is completed. This is the case for applications too, but the sleep is transparently taken care of by the scheduler.

As explained earlier, the first MTD information is the mtd_info structure. This is retrieved by iterating through all registered MTD devices:

#include <linux/kernel.h>
#include <linux/mtd/mtd.h>
#include <linux/err.h>

static struct mtd_info *mtd_info = NULL;
 
int init_module(void) {
    int num;
    for(num = 0; num < 64; num++) {
        mtd_info = get_mtd_device(NULL, num);
        if(IS_ERR(mtd_info)) {
            printk("No device for num %d\n", num);
            continue;
        }
        if(mtd_info->type == MTD_ABSENT) {
            put_mtd_device(mtd_info);
            continue;
        }
        if(strcmp(mtd_info->name, "userdata")) {
            put_mtd_device(mtd_info);
            continue;
        }
        printk("MTD type: %u\n", mtd_info->type);
        printk("MTD total size : %u bytes\n", mtd_info->size);
        printk("MTD erase size : %u bytes\n", mtd_info->erasesize);
        return 0;
    }
    mtd_info = NULL;
    return 0;
}
 
void cleanup_module(void)

{

if(mtd_info)
        put_mtd_device(mtd_info);
}

The above kernel module searches for the “userdata” partition. The function get_mtd_device(), when invoked with the first argument NULL, returns the MTD device associated with the minor number specified in the second argument. On a successful search, it increments the reference count of the device. That’s why, before exiting, a call to put_mtd_device() must be made to release (decrement) the reference count.

Additionally, the module uses the flag MTD_ABSENT (which is available to applications too). This check is required to function correctly with some probing device drivers used to allocate placeholder MTD devices on systems that have socketed or removable media.

Having retrieved the mtd_info structure, reading is relatively simple:

/* read something from last sector */
u_char buf[64];

mtd_info->read(mtd_info, mtd_info.size-mtd_info.erasesize, sizeof(buf), buf);

The second argument of the read function specifies the read offset, and the third the length to read. Note that the read operation too may sleep and, therefore, it must not be performed in an interrupt context. The write operation can be performed as follows (assuming the sector has been previously erased):

/* write something to last sector */
mtd_info->write(mtd_info, mtd_info.size-mtd_info.erasesize, sizeof(buf), buf);

As mentioned before, the read, write and erase operations may sleep. Therefore, kernel code must wait for the operation to finish. Here is an example of erasing the partition and waiting to finish the operation:

#include <linux/sched.h>

void erase_partition(struct mtd_info *mtd_info) {
    unsigned int start;
    for(start = 0; start < mtd_info->size; start += mtd_info->erasesize)
        erase_sector(mtd_info, start, mtd_info->erasesize);
}
 
void erase_sector(struct mtd_info *mtd_info, unsigned int start, unsigned int len) 

{
    int ret;
    struct erase_info ei = {0};
    wait_queue_head_t waitq;
    DECLARE_WAITQUEUE(wait, current);
    
    init_waitqueue_head(&waitq);
    ei.addr = start;
    ei.len = mtd_info->erasesize;
    ei.mtd = mtd_info;
    ei.callback = erase_callback;
    ei.priv = (unsigned long)&waitq;
    ret = mtd_info->erase(mtd_info, &ei);
    if(!ret)     {
        set_current_state(TASK_UNINTERRUPTIBLE);
        add_wait_queue(&waitq, &wait);
        if (ei.state != MTD_ERASE_DONE && ei.state != MTD_ERASE_FAILED)
            schedule();
        remove_wait_queue(&waitq, &wait);
        set_current_state(TASK_RUNNING);
 
        ret = (ei.state == MTD_ERASE_FAILED)?-EIO:0;
    }
}
 
void erase_callback (struct erase_info *instr) {
    wake_up((wait_queue_head_t *)instr->priv);
}

The erase_partition() function iterates over all sectors, and erases them with erase_sector(). At the core of erase_sector() is the mtd_info->erase call, which (as mentioned previously) may sleep. Therefore, erase_sector() prepares a wait queue and a wait queue head.

After a call to mtd_info->erase, the function prepares itself to relinquish the CPU (presuming that mtd_info->erase will sleep) by changing task state to TASK_UNINTERRUPTIBLE and adding itself to the wait queue head. Before relinquishing the CPU, it checks if erase is done, through the ei.state flag. If erase is done successfully, this flag will be set to MTD_ERASE_DONE.

If the erase operation is not complete, the task relinquishes the CPU by calling schedule(). Later, when the erase operation is complete, the driver calls the callback function provided in ei.callback. Here the task wakes up to itself, then removes itself from the wait queue, changes the task state to TASK_RUNNING and finally, the erase_sector() function returns.

MTD devices have many more features that can be used by application programmers. ECC (error correction codes) and OOB (out of band) data are some of them. The MTD framework is integrated into the Linux kernel — therefore it makes working with MTD devices very simple, as we have seen in this article.

Modern life is incomplete without gadgets, smartphones, automated appliances, et al. These electronic devices aide us in our daily grind, making our otherwise mundane/hectic life a bit easier. So what controls these devices? In layman’s language, it’s a small circuit with preprogrammed human logic, called an embedded system. Listed below are some useful definitions.

• Embedded system (ES): An embedded system is some combination of computer hardware and software, either fixed in capability or programmable, that is specifically designed for a particular function. Industrial machines, automobiles, medical equipment, cameras, household appliances, air-planes, vending machines and toys (as well as the more obvious cellular phone and PDA) are among the myriad possible hosts of an embedded system.
• Image processing: In electrical engineering and computer science, image processing is any form of signal processing for which the input is an image, such as a photograph or video frame. The output of image processing may be either an image, or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal, and applying standard signal-processing techniques to it. In other words, it is basically the transformation of data from a still or video camera into either a decision or a new representation. All such transformations are done to achieve some particular goal. The input data may be a live video feed, the decision may be that a face has been detected, and a new representation may be conversion of a colour image into a greyscale image.
• Embedded vision: Embedded vision is the merging of two technologies — embedded systems and image-processing/computer vision (also sometimes referred to as machine vision). Due to the emergence of very powerful, low-cost and energy-efficient processors, it has become possible to incorporate vision capabilities into a wide range of embedded systems. One successful example is the Microsoft Kinect video game controller, which uses embedded vision to track the movements of Xbox 360 users, avoiding the need for handheld controllers. Microsoft sold eight million Kinect units in the first two months after its introduction.

Moving on, the five basic components required to build an embedded system using Python are discussed below, in brief.

The operating system (OS)

The OS is the heart of an embedded vision system. Many different dependencies that are required to run software are provided by the OS, which is the interface between the hardware and the software. Of the many OSs in the market (such as Windows, Linux, etc.) there are reasons why Linux is preferable:

• It is open source, i.e., it is free of cost and free, as in “freedom”.
• It is virtually virus-free.
• It has pretty good support — bug trackers, documentation, forums, mailing lists, IRC, etc.
• It is easy on systems resources — that is, more resources can be dedicated to applications.

There are various Linux distributions like Ubuntu, Fedora, etc., to choose from. I personally prefer Ubuntu, because of its easy learning curve. Now, size is the main constraint for an embedded system. Using a PC OS won’t do. A much more portable solution is required, such as the PandaBoardshown in Figure 1.

Figure 1: PandaBoard

Check out its specs:

• OMAP4 (Cortex-A9) CPU-based open development platform
• OMAP4430 Application processor
• 1 GB low-power DDR2 RAM
• Display HDMI v1.3 Connector (Type A) to drive HD displays, DVI-D Connector
• 8.89cm audio in/out and HDMI Audio out
• Full-size SD/MMC card
• Built-in 802.11 and Bluetooth v2.1+EDR
• Onboard 10/100 Ethernet
• Expansion: 1xUSB OTG, 2xUSB HS host ports, general-purpose expansion header

PandaBoard is such a powerful mobile computing platform that one can port various OSs like Android and Ubuntu 10.04 to it!

Image-Processing (IP) software

There are many software and libraries available for image processing, like MATLAB, OpenCV, etc. Licensing costs for MATLAB are very high, but OpenCV is preferable since:

• It is free.
• It is fast.
• Has good documentation, tutorials, user groups, forums, etc.
• There are a lot of prebuilt functions and algorithms to get a head start.
• There is active development on interfaces for other languages like Ruby, Python, MATLAB, etc.

OpenCV grew out of an Intel Research initiative to advance CPU-intensive applications. The intent behind OpenCV was to provide a platform that a student could readily use for developing applications, instead of reinventing basic functions from scratch. Figure 2 is a block diagram showing the different modules in OpenCV. Check out the documentation here.

Figure 2: OpenCV modules

Arduino

To make any changes in the external environment, we need a hardware circuit with a “brain”. This role can be filled by any microcontroller or microprocessor. Various companies such as Texas Instruments, NXP, Maxim, Atmel, etc., make microcontrollers.

A microcontroller cannot be directly used in a circuit, as is. A full development circuit board has to be made, providing access to different pins of the microcontroller. This process is not easy, as it involves designing the circuit, soldering, etc. All changes are permanent. Also, a program loader has to be made to load the code from the computer into the microcontroller. This can be quite cumbersome, so Arduino is used, due to the following reasons:

• It is open source, so all schematics are available and you can design the same board on your own.
• It is based on the concept of breadboard prototyping, which encourages the use of hook-up wires and breadboard, to make a circuit which can be easily modified later. This avoids the hassles of soldering.
• There are a number of tutorials, tons of code, forums, etc., to help beginners.
• The design of the board and IDE is so simple that one does not have to be an engineer to use it. Even a middle school student can use it.
• It is pretty cheap compared to other available development boards.

There are various versions of Arduino to choose from, such as Uno, Mega, Lilypad, etc. I personally prefer the Arduino Mega (Figure 3), which is a microcontroller board based on the ATmega1280.

Figure 3: Arduino Mega

It has 54 digital input/output pins, of which 14 can be used as PWM (Pulse-width modulation) outputs, 16 analogue inputs, 4 UARTs (hardware serial ports), a 16 MHz crystal oscillator, a USB connection, a power jack, an ICSP header, and a reset button. It contains everything needed to support the microcontroller; simply connect it to a computer with a USB cable, or power it with an AC-to-DC adaptor or battery to get started.

Pyserial

This is a library that provides Python support for serial connections (RS-232) over a variety of different devices: old-style serial ports, Bluetooth dongles, infra-red ports, and so on. It also supports remote serial ports via RFC 2217 (since V2.5). We use this for communication with the external hardware (Arduino). This library is easy to use, and has good documentation.

Python

Python is a general-purpose, high-level programming language whose design philosophy emphasises code readability. Python claims to combine remarkable power with very clear syntax, and its standard library is large and comprehensive.

OpenCV installation

Let us now proceed to OpenCV installation. All commands are meant to be run in a terminal.

1. Install all prerequisites as shown below:

   sudo apt-get install build-essential
   sudo apt-get install cmake
   sudo apt-get install pkg-config
   sudo apt-get install libpng12-0 libpng12-dev libpng++-dev libpng3
   sudo apt-get install libpnglite-dev libpngwriter0-dev libpngwriter0c2
   sudo apt-get install zlib1g-dbg zlib1g zlib1g-dev
   sudo apt-get install libjasper-dev libjasper-runtime libjasper1
   sudo apt-get install pngtools libtiff4-dev libtiff4 libtiffxx0c2 libtiff-tools
   sudo apt-get install libjpeg8 libjpeg8-dev libjpeg8-dbg libjpeg-prog
   sudo apt-get install ffmpeg libavcodec-dev libavcodec52 libavformat52 libavformat-dev
   sudo apt-get install libgstreamer0.10-0-dbg libgstreamer0.10-0 libgstreamer0.10-dev
   sudo apt-get install libxine1-ffmpeg libxine-dev libxine1-bin
   sudo apt-get install libunicap2 libunicap2-dev
   sudo apt-get install libdc1394-22-dev libdc1394-22 libdc1394-utils
   sudo apt-get install swig
   sudo apt-get install libv4l-0 libv4l-dev
   sudo apt-get install python-numpy
   sudo apt-get install build-essential libgtk2.0-dev libjpeg62-dev libtiff4-dev libjasper-dev \
   libopenexr-dev cmake python-dev python-numpy libtbb-dev libeigen2-dev yasm libfaac-dev \
   libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev

2. Install Python development headers with sudo apt-get install python-dev.
3. Download the OpenCV source code. It is recommended that you move the downloaded OpenCV package to your /home/<user> directory. With that as your current directory, extract the archive, as follows:

   tar -xvf OpenCV-2.3.1a.tar.bz2
   cd OpenCV-2.3.1/

4. Make a new directory called build and cd into it (mkdir build; cd build).
5. Run Cmake:

   cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=OFF-D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON ..

6. Follow this with make and sudo make install (to install the library).
7. Next, configure the system to use the new OpenCV shared libraries. Edit a configuration file with sudo gedit /etc/ld.so.conf.d/opencv.conf. Add the following line at the end of the file (it may be an empty file, which is okay) and then save it:

   /usr/local/lib

8. Close the file and run sudo ldconfig.
9. Open your system bashrc file (for me, that is sudo gedit /etc/bash.bashrc) and add the following code at the end of the file:

   PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
   export PKG_CONFIG_PATH

10. Save and close the file, then log out and log in again, or reboot the system.

In the second part of this series, I plan to cover the basics and functions of OpenCV, and then how to develop image-processing programs based on it. In the third (and final) part, I will cover Arduino and Pyserial programming, as well as how to integrate all five components to finally build an embedded system for image processing.

Everywhere we look in today’s data management landscape, the volume of information is soaring. According to one estimate, the data created in 2010 is about 1,200 exabytes, and will grow to nearly 8,000 exabytes by 2015, with the Internet/Web being the primary data driver and consumer.

This growth is outpacing the growth of storage capacity, leading to the emergence of information management systems where data is stored in a distributed way, but accessed and analysed as if it resides on a single machine (Figure 1).

Figure 1: Distributed data storage

Besides resolving the data size problem, these solutions also need to cater to massive performance requirements to ensure timeliness of data processing. Unfortunately, an increase in processing power does not directly translate to faster data access, triggering a rethink on existing database systems.

To understand the enormity of data volumes, let’s look at a couple of figures:  Facebook needs to store 135 billion messages a month. Twitter has the problem of storing 7 TB of data per day, with the prospect of this requirement doubling multiple times per year. Criticality of data and continuity in data availability has become more important than ever. We expect data to be available 24×7 and from everywhere.

This brings in the third dimension of high availability and durability. So we have to factor in high availability without any single point of failure, which has traditionally been a telecom forte. These challenges have created a new wave in database processing solutions, which manage data in both structured
and unstructured ways.

Legacy information management systems are characterised by monolithic relational databases, disk/tape-based stores (as memory in huge quantities is scarce or expensive), vendor lock-in and limited enterprise-level scalability. With the advent of the cloud, new data management solutions are emerging to handle distributed (relational/non-relational) content on open platforms at the speed of a mouse-click.

NoSQL

One of the key advances in resolving the “big-data” problem has been the emergence of NoSQL as an alternative database technology. NoSQL (sometimes expanded to “not only SQL”) is a broad class of DBMS that differ significantly from the classic RDBMS model. These data stores may not require fixed table schemas, usually avoid join operations and typically scale horizontally.

NoSQL solutions have carved a niche market for themselves as key-value stores, document databases, graph databases and big-tables. Use cases for these solutions in Web applications are a dime a dozen. They are used everywhere — whether it is in Web-based social interactions like online gaming/networking, or revenue maximising decisions like ad offerings, and not to mention basic operations like searching the Internet. Figure 2 shows the architecture of a famous NoSQL database (CouchDB) written in Erlang.

Figure 2: CouchDB architecture

But, is NoSQL the answer to all problems? In spite of the technological acumen provided by NoSQL solutions, the RDBMS users in enterprises are reluctant to switch to it. Why?

The greatest deployment of traditional RDBMS is primarily in enterprises, for everything the enterprise does, whether storing customer information, internal financials, employee information or anything else. Is such data growing? Yes, so big-data issues are relevant to them. So why not NoSQL?

Even though there are varieties of NoSQL offerings, they are typically characterised by lack of SQL support, and non-adherence to ACID (Atomicity, Consistency, Isolation and Durability) properties. (Here, we will not go into the technical reasons for this.)

So while NoSQL could help enterprises manage large distributed data, enterprises cannot afford to lose the ACID properties — which were the key reasons for choosing an RDBMS in the first place. Also, since NoSQL solutions don’t provide SQL support, which most current enterprise applications require, this pushes enterprises away from NoSQL.

Hence, a new set of data-management solutions are emerging to address large data OLTP concerns, without sacrificing ACID and SQL interfaces.

Why is traditional OLTP insufficient?

We have already discussed the fact that there is an explosion in the OLTP data space. For example, more widely used social network sites (like Facebook and Linkedin) lead to larger OLTP requirements. Each user of such sites requires credentials and user profile information to be stored, generally in some OLTP database, even though the actual user data is stored in a NoSQL data-store. Facebook’s 750 million users require a very large OLTP database.

While growing OLTP data is a direct contributor, business requirements also force non-OLTP data to be managed as OLTP. Let’s look at the example of how analytics (which is the quintessential OLAP application) has led to the scalability requirements of OLTP.

Case study: Real-time analytics

In the earlier days, analytics were performed on historic data with specialised data warehouse solutions, mostly using an ETL (Extract, Transform and Load) approach. Data extracted from OLTP systems was fed to data warehouse systems capable of handling voluminous data. Real-time analytics is an approach that enables business users to get up-to-the-minute data by directly accessing business operational systems or feeding real-time transactions to analytics systems.

For example, previously Google Analytics analysed past performance — but recently, Google launched Google Analytics: Real Time, which shows a set of new reports on what’s happening on their site, as it happens.

Even though traditional OLTP systems provide ACID, they are not well equipped to handle the volume of data seen because of new innovative business scenarios like real-time analytics. A combination of traditional OLTP systems and analytics systems might under-utilise the analytics systems, due to scalability limitations and the performance of traditional OLTP systems.

Welcome NewSQL

To address big-data OLTP business scenarios that neither traditional OLTP systems nor NoSQL systems address, alternative database systems have evolved, collectively named NewSQL systems. This term was coined by the 451 Group, in their now famous report, “NoSQL, NewSQL and Beyond”.

The term NewSQL was used to categorise these new alternative database systems. 451 Group’s senior analyst, Matthew Aslett, clarified the meaning of the term NewSQL (in his blog) as follows: “NewSQL is our shorthand for the various new scalable/high-performance SQL database vendors. We have previously referred to these products as “ScalableSQL” to differentiate them from the incumbent relational database products. Since this implies horizontal scalability, which is not necessarily a feature of all products, we adopted the term NewSQL in the new report. And to clarify, like NoSQL, NewSQL is not to be taken too literally: the new thing about the NewSQL vendors is the vendor, not the SQL. NewSQL is a set of various new scalable/high-performance SQL database vendors (or databases). These vendors have designed solutions to bring the benefits of the relational model to the distributed architecture, and improve the performance of relational databases to an extent that the scalability is no longer an issue.”

Figure 3 is an architectural example of one of the NewSQL solutions (dbShards).

Figure 3: dbShards

Technical characteristics of NewSQL solutions

1. SQL as the primary mechanism for application interaction.
2. ACID support for transactions.
3. A non-locking concurrency control mechanism so real-time reads will not conflict with writes, and thus cause them to stall.
4. An architecture providing much higher per-node performance than available from traditional RDBMS solutions.
5. A scale-out, shared-nothing architecture, capable of running on a large number of nodes without suffering bottlenecks.

The expectation is that NewSQL systems are about 50 times faster than traditional OLTP RDBMS.

Similar to NoSQL, there are many categories of NewSQL solutions. Now we can look at these categorisations and their characteristics.

NewSQL categorisation

Categorisation is based on the different approaches adopted by vendors to preserve the SQL interface, and address the scalability and performance concerns of traditional OLTP solutions.

1. New databases: These NewSQL systems are newly designed from scratch to achieve scalability and performance. Of course, some (hopefully minor) changes to the code will be required and data migration is still needed. One of the key considerations in improving the performance is making non-disk (memory) or new kinds of disks (flash/SSD) the primary data store. Solutions can be software-only (VoltDB, NuoDB and Drizzle) or supported as an appliance (Clustrix, Translattice). Examples of offerings are Clustrix, NuoDB and Translattice (commercial); and VoltDB, Drizzle, etc., (open source).
2. New MySQL storage engines: MySQL is part of the LAMP stack and is used extensively in OLTP. To overcome MySQL’s scalability problems, a set of storage engines are developed, which include Xeround, Akiban, MySQL NDB cluster, GenieDB, Tokutek, etc. The good part is the usage of the MySQL interface, but the downside is data migration from other databases (including old MySQL) is not supported. Examples of offerings are Xeround, GenieDB and TokuTek (commercial); and Akiban, MySQL NDB Cluster and others in open source.
3. Transparent clustering: These solutions retain the OLTP databases in their original format, but provide a pluggable feature to cluster transparently, to ensure scalability. Another approach is to provide transparent sharding to improve scalability. Schooner MySQL, Continuent Tungsten and ScalArc follow the former approach, whereas ScaleBase and dbShards follow the latter approach. Both approaches allow reuse of existing skillsets and ecosystem, and avoid the need to rewrite code or perform any data migration. Examples of offerings are  ScalArc, Schooner MySQL, dbShards and ScaleBase (commercial); and Continuent Tungsten (open source).

Summing up

Given the continuing trend of data growth in OLTP systems, a new generation of solutions is required to cater to them. DBMSs have to be rearchitected from scratch to meet this demand. However, unlike NoSQL, these DBMSs have to cater for applications already written for an earlier generation of RDBMS. Hence, drastic interface changes like throwing out SQL or extensive changes to data schema is out of the question.

A new generation of information management systems, termed NewSQL systems, caters to this trend and these constraints. NewSQL is apt for businesses that are planning to:

1. migrate existing applications to adapt to new trends of data growth,
2. develop new applications on highly scalable OLTP systems, and
3. rely on existing knowledge of OLTP usage.

We look forward to your comments on this introductory article, and we will also plan for a series of expert in-depth articles on the NewSQL database technology.

Gartner has predicted in-memory computing to be one of the top 10 strategic technologies of 2012. In-memory computing is expected to have a disruptive impact on the data warehousing domain in the coming two years. There have been a series of such products released in the last two years, one of the most famous being SAP’s HANA. Real-time analytics and sub-second response times for enterprise applications require high-performance data management systems. This, in turn, has led to a surge in the importance of IMDBs.

Network bandwidth has increased dramatically and multi-core processors are available even in mobile phones. However, disk I/O speed has not been increasing at the same rate, which has crippled traditional databases. The first step towards high-performance databases is IMDBs. With ever-growing RAM size and the ability to address more RAM (with 64-bit address spaces), IMDBs are in vogue.

What is an in-memory database?

Wikipedia defines an in-memory database as, “An in-memory database (IMDB; also, main memory database system or MMDB) is a database management system that primarily relies on main memory for computer data storage. It is contrasted with database management systems that employ a disk storage mechanism.”

Margaret H Eich has given a very simple definition of an IMDB/MMDB: a database whose primary data store is main memory.

IMDBs are architected and designed differently from traditional RDBMSs. They have simplified algorithms and mechanisms, which are built with the awareness that all data is going to be in RAM. Figure 1 is a simple illustration of the differences.

Figure 1: IMDB vs Disk DB

IMDBs are also different from simplified storage algorithms like hash-tables or trees, in that IMDBs are “databases” or RDBMSs. Most IMDBs are ACID-compliant relational databases offering SQL. They have all the properties of a traditional RDBMS, but are tuned for data to be in-memory.

Getting deeper into the IMDB

IMDBs are not a new phenomenon. The great DB scientist Jim Gray conceptualised this technology 30 years ago and also predicted that in the 2000s these technologies would become widely accepted. Almost from the early 1980s, IMDBs have existed in the telecom and defense domains. However, these were built and maintained as internal components. IBM built one of the first in-memory engines (IMS/VS FastPath) way back in 1978.

The first significant commercial IMDB offering was TimesTen (later acquired by Oracle). Since then, almost all top DB vendors boast an IMDB product — like IBM’s SolidDB, Sybase’s ASE, etc.

Myths about IMDBs

Consider the following assertions:

• Given the same amount of RAM, disk DBs can perform at the same speed as IMDBs (by using caching technology).
• If a RAM disk is created and a traditional disk DB is deployed on it, it delivers the same performance as an in-memory database.
• If the system crashes, all the data stored in IMDBs will be lost.
• SSDs and flash storage are getting better and better. Using these technologies along with traditional disk DBs yields the same performance as an in-memory database.
• Since RAM size is limited, sizes of IMDBs are also limited.
• Finally, IMDBs are not special; they are traditional disk DBs made to run on RAM.

Naturally, all of these are myths. It is close to impossible to create an in-memory DB from a traditional disk database by just changing the OS or hardware environment.

So, internally, how different is an IMDB from a traditional disk DB?

IMDBs are architected and designed keeping in mind the fact that all data is in memory. This actually leads to much simpler design as compared to disk DBs. There are six areas of difference:

1. Query optimisation: In disk DBs, the I/O cost factor dominates the optimisation. However, in IMDBs there is no such clear factor, which makes query optimisation very tricky. This is generally solved by taking constants and falling back on rule-based optimisation.
2. Indexing: More memory-friendly data structures and algorithms are used for indexing. While most disk DBs use B-Tree as a primary indexing data structure/algorithm, IMDBs tend to use T-Tree as a primary indexing data structure/algorithm.
3. Internal data representation: Compactness of representation dominates concerns for IMDBs. With all data being in memory, IMDBs tend to use direct memory pointers heavily. This is very typical of the IMDB memory page, index data or relation representations.
4. Durability and recovery: Contrary to popular belief, IMDBs are durable. They use algorithms similar to disk DBs for persistence. However, the buffer management, which is the biggest performance bottleneck for disk DBs, is eliminated. During database loading, IMDBs tend to take a bit more time as they have to load the complete data into memory. Hence, recovery is a bit slower.
5. Access methodology: Generally, disk DBs offer client server over sockets as a primary access method. However, with no disk I/O, if IMDBs only offer sockets for access, this will become a bottleneck. Hence, most IMDBs tend to offer shared-memory access as a primary method. In a few cases, JDBC/ODBC interfaces are also supported.
6. Concurrency control: Due to inherent speed in processing, IMDBs can take coarser locks and also do less to persist them. However, disk DBs take finer locks and take elaborate measures to persist them.

Figure 2 is a typical architecture for an in-memory database.

Figure 2: IMDB architecture

Typical applications of IMDBs

IMDBs are applicable in all domains that require real-time performance and very low latency. Four domains typically use IMDBs: telecom, financial segments, enterprises, and e-commerce and Web applications. In these spaces, IMDBs are used in a variety of applications (refer to Figure 3).

Figure 3: IMDB usecases

Key offerings in the IMDB space

The IMDB space is dominated by a lot of commercial players. Some of the most important ones are Oracle TimesTen, IBM SolidDB, Sybase ASE, ENEA Polyhedra and McObject ExtremeDB. There are also some typical open source solutions. Let us take a look at two of the FOSS IMDBs — CSQL and MonetDB.

CSQ

CSQL is an open source main-memory high-performance RDBMS developed in India. It is one of the fastest open source IMDBs. It is designed to provide high performance on simple SQL queries and DML statements that involve only one table. It supports only limited features, which are used by most real-time applications, like INSERT, UPDATE, DELETE on a single table, and SELECT with local predicates on a single table.

It provides multiple interfaces such as JDBC, ODBC and other SQL APIs. CSQL offers atomicity, consistency and isolation. It is typically recommended for use as a cache for existing disk-based commercial databases.

MonetDB

MonetDB is an open source high-performance DBMS developed at the National Research Institute for Mathematics and Computer Science in the Netherlands. It was designed to provide high performance on complex queries against large databases, e.g., combining tables with hundreds of columns and multi-million rows.

MonetDB is one of the first database systems to focus its query optimisation effort on exploiting CPU caches. Development of MonetDB started in 1979 and it became an open source project in 2003. MonetDB has been successfully applied in high-performance applications for data mining, OLAP, GIS, XML Query, and text and multimedia retrieval.

How do IMDBs change the rules of the game? While existing applications and schemas can directly benefit from IMDB due to performance improvement, if some things are not carefully handled, the full benefit of IMDB is not achieved. Areas that require significant changes at both the conceptual as well as implementation levels are application design, database schema design and data design (partitioning of data).

Application design

Taking advantage of performance benefits offered by an IMDB means redesigning applications to take advantage of the specific strengths of in-memory technology. The main way to achieve this is to push work that is currently done in the application layer down to the database.

This not only allows developers to take advantage of special operations offered by the DBMS, but also reduces the amount of data that must be transferred between application layers. This can lead to substantial performance improvements and can open up new application areas.

Database schema design

IMDB is beneficial if the data fits into a single database. This requires efforts to conserve space, so more elaborate normalisation procedures are not suitable. Also, using very precise and apt data types enhances storage space. Reduced redundancy, carefully formed columns, precise index creation and efficient data management will help IMDB yield better performance.

Data design

One of the key changes when moving from traditional disk DBs to in-memory DBs is the space available, so data partitioning and storage assumes great significance. IMDB requires as much related data as possible in a single process space. Too much distribution will introduce network I/O in the processing, thereby degrading performance.

Summing up

IMDBs, combined with various current hardware trends, have the ability to change the performance of enterprise and other applications drastically. This, in turn, will result in tremendous value generation to businesses. Such enhanced performance will also foster the evolution of innovative applications and services.

Do send in your feedback and queries, which can be addressed in our forthcoming articles in the IMDB series.

A cluster is a collection of commodity components that provide scalability and availability at a low cost to the consumer. A database cluster is a collection of database servers that store and process data using commodity hardware, satisfying the need for more data storage, higher throughput and providing high availability. Postgres-XC is such a database cluster system; it is based on PostgreSQL, and follows the same open source model.

The Postgres-XC project began in 2009, through a collaboration between NTT and EnterpriseDB. The goal was to build an open source clustering solution based on PostgreSQL with 100 per cent compatible client APIs. Having PostgreSQL-compatible APIs allows existing PostgreSQL applications to use Postgres-XC with little (or no) change. The licensing terms of this project are the same as that of PostgreSQL.

Postgres-XC architecture

Postgres-XC is a write-scalable, synchronous multi-master, transparent PostgreSQL clustering solution based on shared-nothing architecture. It is a collection of tightly coupled database components, which can be installed on one or more physical or virtual machines. The components do not share any resources such as disk, cache or memory.

• Write-scalability means that Postgres-XC can be configured with as many database servers as needed; Postgres-XC is able to handle more writes than a single PostgreSQL server.
• Multi-master implies that clients can connect to multiple database servers, and that each database server provides a single, consistent, cluster-wide view of the database.
• Synchronous means that a write from any of the masters is immediately visible to other transactions running on other masters.
• Transparent means that applications do not have to worry about how the data is stored in multiple database servers, internally.

Figure 1: Postgres-XC architecture

Figure 1 gives the architectural overview of Postgres-XC with its three main components:

1. Global Transaction Manager (GTM) gathers and manages information about transactional activities in Postgres-XC, issues global transaction identifiers to transactions (to maintain a consistent view of the database on all nodes), and  provides ACID properties. It provides support for other global data, such as sequences and timestamps. It stores no user data, except control information.
2. Coordinators (masters) provide a point of contact for the application/client. They are responsible for parsing and executing queries from the clients, and returning the results (if needed). They do not store any user data themselves, but gather the data from datanodes, with the help of SQL queries fired through a PostgreSQL-native interface. The coordinators also process the data if required and even manage the two-phase commit. Although coordinators do not store user data, they use the catalogue data to parse queries, resolve symbols, plan queries, locate data, etc.
3. Datanodes store user data and catalogues. The datanodes execute the queries received from the coordinator and return results to the coordinator.

Distribution of data and scalability

Postgres-XC allows two ways of storing the tables on the datanodes:

1. Distributed tables: A table is distributed on a given set of datanodes using strategies like hash, round-robin, or modulo partitioning. Every row of a distributed table resides on a single datanode. Multiple rows can be modified or written in parallel to various datanodes; we can also read the rows from various datanodes in parallel. Performance is greatly improved by parallel writes and reads from different datanodes.
2. Replicated tables: A table is replicated on a given set of datanodes using statement-level replication, which performs better than log-based replication, since the size of the logs that must be shipped is much greater than the size of the statement. In the case of a replicated table, a row in the table resides on each datanode on which the table is replicated. Any modifications to the row must be duplicated to each replicated copy. Since all the data in the table is available on a single datanode, the coordinator can gather all the data from a single node and in some cases, act as a proxy between the client application and the datanode. This allows multiple queries on the same table to be directed to different datanodes, thus balancing the load and increasing the read throughput.

Figures 2 and 3 depict the read and write concepts for distributed and replicated tables, respectively.

Figure 2: Distributed tables

Figure 3: Replicated tables

High availability

To achieve high availability, one needs data redundancy, component redundancy and automatic failover. In Postgres-XC, data redundancy can be achieved by using the PostgreSQL native replication with Hot Standby for datanodes. Since each coordinator is a master (capable of writing data) and is capable of reading writes performed by any other coordinator instantaneously, every coordinator is capable of replacing any other, should that coordinator fail. GTM-standby acts as a redundant component for GTM. However, third-party tools are required for automatic failover of all the three types of components.

Performance evaluation

Initial transaction throughput measurements carried out using the DBT-1 benchmark have shown significant throughput scalability, as shown in Figure 4.

Figure 4: Performance evaluation

The figure plots the Scaling Factor versus the Number of Servers in Postgres-XC. The Scaling Factor is the ratio of the number of transactions completed per unit time by Postgres-XC, to that completed by PostgreSQL. A Server comprised of a coordinator and a datanode run on single machine. This benchmark demonstrated an improvement in throughput of approximately 6 times, when using 10 servers.

Release management & development processes

The Postgres-XC project is hosted on SourceForge. The Postgres-XC team tries to release a minor version of Postgres-XC every three to four months to ensure that the latest Postgres-XC features are available to users. The team also tries to make the latest PostgreSQL features available in Postgres-XC by doing frequent merges with the latest stable release of PostgreSQL.

The last release (0.9.6) of Postgres-XC supports most of the SQL syntax and features of PostgreSQL 9.1. The team is currently working on the first major release of Postgres-XC 1.0, due in 2012, with maximum PostgreSQL compliance. Some of the other features like the dynamic addition and removal of components, global deadlock detection, global constraint support, etc, will be targeted for major release after 1.0.

The development team follows the open source development model, where the issues, features or any other development related items are discussed on the public mailing list: postgres-xc-developers@lists.sourceforge.net. The postgres-xc-general@lists.sourceforge.net mailing list is used to discuss other Postgres-XC matters and to solicit help about Postgres-XC.

Wish to contribute?

The Postgres-XC team needs help with feature development, bug fixing, creating installers and distribution packages, testing, and evaluation of Postgres-XC on real applications. To be part of the Postgres-XC community, feel free to contact the Postgres-XC team at the appropriate mailing list.

Currently, PostgreSQL supports range and list partitioning via table inheritance. Basically, you have to create each partition as a child table of the master table. Generally, if you want to split data into specific ranges, then use range partitioning.

Date and timestamp values are good examples for this. For example, you may want to split yearly data into quarterly partitions. Similarly, if you want to split data based on a specific list of values, then use list partitioning. For this, you can consider creating partitions for each state of a country. Generally, in range partitioning, you would be adding more partitions over time, whereas in list partitioning the data will be continuously growing within each partition. This design decision is very important from the partition maintenance point of view, which most people ignore.

Although the syntax for creating partitioned tables (just like the one in a commercial RDBMS) is not available in PostgreSQL, you may wonder exactly how to set up partitioning in PostgreSQL. You will be glad to know that PostgreSQL uses the concept of inheritance, triggers and constraint exclusion to support partitioning. Additionally, you can also use rules instead of triggers.

Now, inheritance and constraint exclusion come with their own pros and cons. There are various scenarios that you need to take care of. Sharing details on them is the main goal of this article. Further, once that has been addressed, the next goal should be to automate partition maintenance because you would most certainly like to automate the creation and maintenance part of partitioning. As you will realise by the end of this article, partition maintenance is another important task, apart from setting up the partitions.

Partitioning using inheritance

Before we begin, you should note that there are various ways to set up partitioning. The scheme that I am going to show here is just one of many. You would need to tweak it as per the actual requirements.

Before we start, make sure that the constraint_exclusion configuration option is set to either ON or PARTITION in the postgresql.conf file. This is absolutely necessary to get the partitioning scheme working.

First, let us create an empty master table without any primary key. If you are wondering about the missing primary key, then you should know that this is an empty master table, and that we need to set up a primary key on each of the partitions separately (more on this in the latter part of the article):

CREATE TABLE orders (
     id            INT NOT NULL,
     address       TEXT NOT NULL,
     order_date    TIMESTAMP NOT NULL
);

Now, create some partitions using non-overlapping CHECK constraints:

CREATE TABLE orders_part_2011 (
    CHECK (order_date >= DATE '2011-01-01'
                    AND order_date < DATE '2012-01-01')

) INHERITS (orders);

CREATE TABLE orders_part_2010 (
    CHECK (order_date < DATE '2011-01-01')
) INHERITS (orders);

Here, the partition orders_part_2011 will hold all data from 2011 onwards, whereas orders_part_2010 will hold all previous data. Note that I am not using any primary key clause, as order_date cannot be a primary key in this particular setup. Further, always make sure that you match the data-type of values used with that of the column referred to in the CHECK constraints.

Optionally, you may create indexes on the partitioning key columns for each of the partitions:

CREATE INDEX orders_part_2011_idx ON orders_part_2011(order_date);
CREATE INDEX orders_part_2010_idx ON orders_part_2010(order_date);

The partitioning scheme is not yet ready. What I really want is that the data inserted into the master table should be redirected to the appropriate partition. This is where I need a trigger or rule. Triggers and rules have their own pros and cons. Here I use triggers, with trigger functions.

Again, a trigger function can be written in many ways, depending on the requirements. I will show two ways of writing one, along with their advantages and disadvantages. Let’s create a trigger function as shown below:

CREATE OR REPLACE FUNCTION orders_insert_simple() RETURNS TRIGGER AS $$
DECLARE
vsql Text;
BEGIN
   vsql :=   'INSERT INTO orders_part_'|| to_char(NEW.order_date, 'YYYY' )||
   ' VALUES ('||NEW.id||','||quote_litera(NEW.address)||','||quote_literal(NEW.order_date)||')';
   RETURN NULL;
END;
$$ LANGUAGE plpgsql;

Now we need to attach this trigger function to a trigger, to handle the INSERT on the master table:

CREATE TRIGGER orders_insert_trigger
BEFORE INSERT ON orders
FOR EACH ROW EXECUTE PROCEDURE orders_insert_simple();

There are quite a few interesting things used in the trigger function that you need to closely look at. It is important to note that the partition name is computed on the fly. This works for our particular case, as all the required partitions were created beforehand. This function would not require any updates, even if you add new partitions for the next year, and so on.

Looks easy? Then take a close look again, at the part where I have used quote_literal to quote the input values. Essentially, you need to quote each and every possible value that is going to be inserted into the partitions.

Similarly, you would need to take care of NULL values. This is the most cumbersome part of using this easy-to-write trigger function, so let’s try out another way of writing the trigger function:

CREATE OR REPLACE FUNCTION orders_insert() RETURNS TRIGGER AS
$$
BEGIN
    IF (NEW.order_date >= DATE '2011-01-01' AND NEW.order_date < DATE '2012-01-01') THEN
       INSERT INTO orders_part_2011 VALUES (NEW.*);
   ELSIF (NEW.order_date < DATE '2011-01-01') THEN
       INSERT INTO orders_part_2010 VALUES (NEW.*);
   ELSE
       RAISE EXCEPTION 'Date out of range. check orders_insert() function!';
   END IF;
   RETURN NULL;
END;
$$
LANGUAGE plpgsql;

CREATE TRIGGER orders_insert_trigger
BEFORE INSERT ON orders
FOR EACH ROW EXECUTE PROCEDURE orders_insert();

As you can see, we are now explicitly checking order_date to redirect data into the appropriate partitions. This doesn’t need handling of quote or NULL values — at least, for non-partition-key values. I have simply used NEW.* instead of referring to individual column values.

In case you are not able to create all partitions beforehand, you would need to update the above trigger function with the logic to create partitions on-the-fly. Generally, we use separate triggers for partition creation and data redirection. Obviously, the partition creation trigger should always be fired before the redirection trigger. You should take care to avoid race conditions between those two triggers.

Now that the trigger is in place, let’s insert some records and verify the setup:

pg=# INSERT INTO orders VALUES(1, 'pune', '2011-08-22');
INSERT 0 0
pg=# INSERT INTO orders VALUES(2, 'pune', '2010-02-22');
INSERT 0 0
pg=# UPDATE orders SET address = 'bengaluru' WHERE id = 2;
UPDATE 1

Do not panic on seeing the INSERT statement messages — they simply convey that zero records were inserted in the base (master) table. Actually, the records were transparently inserted into the related partitions.

Another thing to note about the UPDATE statement is that you are not supposed to update the partitioning key. Basically, any change to the partitioning key value might result in the movement of that record to another partition, which is termed as the row movement.

A simple way to handle queries that do update the partitioning key is to capture the UPDATE query, delete the related record from the partition, and then fire an INSERT on the base table. Then the partitioning mechanism will kick in and redirect the record to the appropriate partition. This is not handled in the current setup, as I am updating the address value, which is a non-partition-key column.

Let’s now check where the records are stored:

SELECT * FROM orders;
 id |  address   |     order_date
----+--------------+---------------------------
  1 | pune         | 22-AUG-11 00:00:00
  2 | bengaluru | 22-FEB-10 00:00:00
(2 rows)

PostgreSQL knows about the child tables of the orders table, so it assumes that the user wants all the data, from the parent as well as all the children. The partitions are normal tables, and you can query them as usual:

SELECT * FROM orders_part_2011;
 id | address |     order_date
----+---------+------------------------
  1 | pune     | 22-AUG-11 00:00:00
(1 row)

SELECT * FROM orders_part_2010;
 id |  address  |     order_date
----+-----------+--------------------
  2 | bengaluru | 22-FEB-10 00:00:00
(1 row)

You can also check if the master table is really empty, by using the ONLY clause, which restricts the lookup to only the table specified in the statement:

SELECT * FROM ONLY orders;
 id | address | order_date
----+---------+------------
(0 rows)

Querying over partitions
Use the EXPLAIN feature to check the plan for querying over partitions:

EXPLAIN SELECT * FROM orders WHERE order_date = '02-JAN-11';
                                       QUERY PLAN
-----------------------------------------------------------------
 Result  (cost=0.00..26.01 rows=7 width=40)
   ->  Append  (cost=0.00..26.01 rows=7 width=40)
         ->  Seq Scan on orders  (cost=0.00..23.75 rows=6 width=44)
               Filter: (order_date = '02-JAN-11 00:00:00'::timestamp without time zone)
         ->  Seq Scan on orders_part_2011 orders  (cost=0.00..2.26 rows=1 width=18)
               Filter: (order_date = '02-JAN-11 00:00:00'::timestamp without time zone)
(6 rows)

In the above output, you see that only one partition was scanned, based on the WHERE clause conditions. Let’s look at another example:

EXPLAIN SELECT * FROM orders WHERE order_date = now();
                                     QUERY PLAN
-----------------------------------------------------------------
 Result  (cost=0.00..30.03 rows=8 width=41)
   ->  Append  (cost=0.00..30.03 rows=8 width=41)
         ->  Seq Scan on orders  (cost=0.00..26.50 rows=6 width=44)
               Filter: (order_date = now())
         ->  Seq Scan on orders_part_2011 orders  (cost=0.00..2.51 rows=1 width=18)
               Filter: (order_date = now())
         ->  Seq Scan on orders_part_2010 orders  (cost=0.00..1.01 rows=1 width=44)
               Filter: (order_date = now())
(8 rows)

Here all the partitions are scanned — definitely not what we wanted! You should be aware of the fact that the planner analyses the query before the values from the parameters or stored procedures are substituted. As the planner does not know the exact value of now() during the planning phase, it cannot prune partitions, and so scans all the partitions. You need to look out for such cases where constant values are expected. In case you are planning to use functions in the WHERE clause, do make sure to understand the various types of functions that can be created in PostgreSQL.

Let us now go through the finer details of the three features used in PostgreSQL partitioning.

Constraint exclusion

Constraint exclusion works with only range or equality check constraints. It might not work for constraints like the following:

ALTER TABLE product_items_j ADD CONSTRAINT chk_item_name CHECK (item_name LIKE 'P%');

More importantly, the WHERE condition should be similar to the CHECK constraints. For example, if you have the following CHECK constraint:

ALTER TABLE product_items_j ADD CONSTRAINT chk_item_name CHECK (item_name BETWEEN 'P' AND 'PB'');

Then constraint exclusion would not help for the following query, since the WHERE condition is not similar to the CHECK constraint:

SELECT item_name FROM product_items WHERE item_name LIKE 'Q%' ;

To get the constraint exclusion working, you need this form of query:

SELECT item_name FROM product_items WHERE item_name = 'Pen';

From the above examples, it should be clear that the user needs to take extra effort while handling the WHERE clause predicates, in order to hit the CHECK constraints. The other major problem with this setup is that there is no automatic way to verify if all the CHECK constraints are mutually exclusive. You need to be extra careful while setting them up.

The EXPLAIN feature comes in very handy to tackle these issues. Basically, any change to the query or table, even the slightest one, should be rigorously followed by looking at the EXPLAIN output. If you do not see the expected plan, then either the WHERE clause or the CHECK constraints need to be looked into.

Inheritance

The important rule that you should always be aware of is that child tables inherit column DEFAULT values, not NULL and CHECK constraints only. Any other constraints, like UNIQUE, PRIMARY and FOREIGN key, will not be inherited. Also, indexes, ownership and permissions will not be inherited.

One has to set up the non-inherited constraints on all the child tables. However, there are various ways of setting up these constraints, which we will discuss in the very next section. The child tables cannot rename inherited columns, but can add new columns — this is the biggest advantage of using inheritance.

You should also note that one can enable or disable inheritance on child tables by using the ALTER TABLE command. This is, in fact, very useful when you want to remove or move partitions from the base table.

Using inheritance, you can have multiple levels of partitioning. You can always create a great-great-grandchild of a partition, but you might experience a performance loss with this kind of setup, since more base tables are involved, directly or indirectly. So generally, 1 or 2 levels for partitions and sub-partitions should be good. If you are going any deeper, then most probably you need to rethink the partitioning scheme employed.

Uniqueness

PostgreSQL uses indexes for supporting uniqueness. However, there are no multi-table indexes, which means that an index cannot span over more than one table. This is the reason for not having any primary key in our setup. In general, you might want to concentrate on indexing the partitioning key columns. For this, create non-overlapping CHECK constraints and then create a unique index on partitioning key columns over each partition. This is almost like having unique values over all partitions.

For indexing the non-partitioning-key columns, you create unique indexes over each partition. However, for verifying uniqueness, you need custom functions to scan all partitions, which would hopefully use the related indexes.

As you must have realised, this scanning of all partitions was not what we wanted, to begin with. Also, due to the lack of multi-table indexes, maintaining a primary key or a foreign key is technically not possible.

Consider that you created a primary key on each of the order_id columns of all child tables. Now it may happen that both orders_part_2011 and order_part_2010 have an order_id value of 1. Even though the uniqueness constraint within each child table is upheld, the base table orders now has duplicate order_id values.

To overcome this, you can create an additional lookup table, with primary keys from the base as well as child tables. Basically, you need to use triggers to keep the lookup table updated as you insert, update or delete records from the related partitioned table. Now all the primary and foreign key constraints can be mapped to this lookup table.

That’s it! These are basic setup issues that you must be able to get through now. You should now be ready to face the next challenge — performance tuning for partitions.

What’s around the corner?

The PostgreSQL community does have several discussions on this feature. For more details, you can visit this wiki entry on partitioning.

Macros — can’t get enough of them!

You already know from my previous articles that macros are Lisp programs that generate other Lisp programs. The generated Lisp code has fully parenthesised notation, and so does the macro that generates the code. In the simplest case, a macro substitutes forms within a template, clearly establishing a visual correspondence between the generating code and the generated code.

Complex macros can use the full power of the Lisp language to generate code according to the macro parameters; often, a template form is wrapped in code that constructs appropriate sub-forms, but even this approach is just a typical pattern of use, and not a requirement (or restriction) of the Lisp macro facility.

To refresh our memories, let’s examine the mechanism by which the Lisp system translates code generated by a macro. You define a macro with a defmacro form; defmacro is like defun, but instead of returning values, the body of the defmacro returns a Lisp form. Your program ‘calls’ a macro the same way it calls a function, but the behaviour is quite different.

First, none of the macro’s parameters are evaluated…ever. Macro parameters are bound literally to the corresponding arguments in the macro definition. If you pass (* 7 (+ 3 2)) to a macro, the argument in the body of the macro definition is bound to the literal list (* 7 (+ 3 2)) and not the value 35.

Next, the macro expander is invoked, receiving all of the actual parameters bound to their corresponding arguments as named by the defmacro form. The macro expander is merely the body of the defmacro form, which is just Lisp code; the only catch is that the Lisp system expects the macro expander to return a Lisp form. The Lisp system then evaluates whatever form the macro expander returns.

A Lisp implementation may expand macros at different times. A macro could be expanded just once, when your program is compiled. Or it could be expanded on first use as your program runs, and the expansion could be cached for subsequent reuse. A properly written macro will behave the same under all of these implementations.

Let’s explore a real-world example of using a macro to extend Lisp into the problem domain. In addition to providing a macro expander, our new macro will automatically generate an environment that will be referenced by the expander. Our example will show how to move computations from run-time to compile-time, and how to share information computed at compile-time.

Let’s say you’re working on an interactive game that makes heavy use of the trigonometric function sin r in computing player motion and interaction. You’ve already determined that calling the Lisp function sin is too time-consuming; you also know that your program will work just fine with approximate results for the computation of sin r. You’d like to define a lookup-sin macro to do the table lookup at runtime and also hide the details of table generation, which would just clutter your program’s source code.

Your macro will be invoked as (lookup-sin radians divisions), where radians is always in the range of zero to one-half pi, and divisions is the number of discrete values available as the result of lookup-sin. At runtime, the macro expander will just compute the index into a lookup table, and return the value from the table. The table will be generated at compile-time (on most Lisp systems). Furthermore, only one table will ever be generated for a given value of divisions in the macro call.

;; This is where we cache all of the sine tables generated during compilation.
;; The tables stay around at runtime so they can be used for lookups.
(defvar *sin-tables* (make-hash-table)
    "A has table of tables of sine values. The hash is keyed by the number of entries in each sine table.")

;; This is a helper function for the lookup-sin macro. It is used only at compile
;; time.
(defun get-sin-table-and-increment (divisions)
    "Returns a sine lookup table and the number of radians quantised by each entry in the table. Tables of a given size are reused. A table covers angles from zero to pi/2 radians."
(let ((table (gethash divisions *sin-tables* :none))
      (increment (/ pi 2 divisions)))
    (when (eq table :none)
        ;; Uncomment the next line to see when a table gets created.

        ;; (print `|Making new table|)
        (setq table
            (setf (gethash divisions *sin-tables*)
                  (make-array
                    (1+ divisions)
                    :initial-element 1.0)))
        (dotimes (i divisions)
            (setf (aref table i)
                  (sin (* increment i)))))
     (values table increment)))

;; Macro calls the helper at compile time, and returns an AREF form to do the
;; lookup at runtime.
(defmacro lookup-sin (radians divisions)
    "Return a sine value via table lookup."
    (multiple-value-bind (table increment)
                         (get-sin-table-and-increment divisions)
        `(aref, table (round, radians, increment))))

Let us examine what is happening here. When this program runs, it executes just aref  (and associated round) to look up the sin r value.

> (pprint (macroexpand-1 `(lookup-sin (/ pi 4) 50)))

(AREF #(0.0D0 0.03141075907812829D0 0.06279051952931338D0 0.09410831331851433D0 0.12533323356430426D0 0.15643446504023087D0 0.18738131458572463D0 0.21814324139654257D0 0.2486898871648548D0 0.2789911060392293D0
        0.3090169943749474D0 0.3387379202452914D0 0.368124552684678D0 0.3971478906347806D0 0.4257792915650727D0 0.4539904997395468D0 0.4817536741017153D0 0.5090414157503713D0 0.5358267949789967D0 0.5620833778521306D0
        0.5877852522924731D0 0.6129070536529765D0 0.6374239897486898D0 0.6613118653236518D0 0.6845471059286887D0 0.7071067811865476D0 0.7289686274214116D0 0.7501110696304596D0 0.7705132427757893D0 0.7901550123756904D0
        0.8090169943749475D0 0.8270805742745618D0 0.8443279255020151D0 0.8607420270039436D0 0.8763066800438637D0 0.8910065241883678D0 0.9048270524660196D0 0.9177546256839811D0 0.9297764858882515D0 0.9408807689542256D0
        ...)
      (ROUND (/ PI 4) 0.031415926535897934D0))

Note that the macro call makes no mention of a lookup table. Tables are generated as needed by (and for) the compiler.

> (lookup-sin (/ pi 4) 50)
0.7071067811865476D0

In the macro expansion, the #(...) is the printed representation of the lookup table for 50 divisions of the quarter circle. This table is stored in the *sin-tables* hash table, where it is shared by every macro call to (lookup-sin angle 50). We don’t even have to do a hash lookup at runtime, because the macro expander has captured the free variable table from the multiple-value-bind form in lookup-sin.

Macros that define macros

Macros that define macros are used infrequently, partly because it’s hard to think of a good use for this technique, and partly because it’s difficult to get right. The following macro, based on an example in Paul Graham’s book On Lisp, can be used to define synonyms for the names of Lisp functions, macros, and special forms.

> (defmacro defsynonym (old-name new name)
"Define OLD-NAME to be equivalent to NEW-NAME when used in the first position of a Lisp form."
 `(defmacro, new-name (&rest args)
                    `(,`,old-name, @args)))
DEFSYNONYM

> (defsynonym make-pair cons)
MAKE-PAIR

> (make-pair `a `b)
(A . B)

Macros are always a little bit dangerous because code containing a macro call does not automatically get updated if you change the definition of the macro. You can always establish your own convention to help you remember that you need to recompile certain code after you change a macro definition. But there’s always the possibility that you’ll forget, or make a mistake.

Ultimately, the likelihood that you’ll inadvertently end up with code that was compiled with an old version of a macro is directly proportional to how often you’re likely to change the macro. A macro like defsynonym practically begs to be used again and again as you generate new code. If you change your mind about the old name to associate with a given new name, all of your previously compiled code will still refer to the old name that you had decided upon earlier.

I’ll leave you here to have fun with your macros.

In the next article, we’ll take a peek into Common Lisp Object System (CLOS) which allows you to build very sophisticated object-based systems. If you care to code with a strongly object-oriented mindset, you will probably find all the OOP language functionality you need in Common Lisp. CLOS has many advanced object-oriented features that you won’t find in many other places. Because of this, CLOS has often been used as a research tool for studying OOP ideas.