Category Archives: OODB

260 000 images on a Raspberry Pi

In the previous post the op_lite object oriented database library for C++ was introduced.  I have been testing this library on Windows and Kubuntu using images from the Raspberry Pi1 Model B weather camera. The camera captures a JPEG image of size 1296×972 every minute, which means that each day there are 1440 additional images to put in the weather camera database. The database now has a viewer written in C++ based on op_lite and wxWidgets, It works fine on both Windows and Linux.

The PI weather camera has been  running steady for just about 6 months now, it has so far accumulated just over 260 000 images (database size is > 20GB) showing the daily weather plus stars at night. Today I wanted to try op_lite and the viewer on another Pi1 Model B, so I compiled the database viewer application there. This is straightforward as Raspbian is a debian derivative, just like Kubuntu.

The the 20GB database was copied from Windows across the LAN network to the PI which has a 320GB USB hard drive connected, formatted as linux ext4. The copy took a few minutes and the compilation of the software took longer, but it worked!  It shows that the database is compatible and can be freely copied around Windows/Kubuntu/Raspberry PI.

260 000 images captures on a PI1, viewed on another. It doesn’t work as fast as on a desktop, but it is certainly usable. I have a new PI2 Model B coming soon, and  it will be interesting to see how things performs there.  As the PI2 is said to be about ~6 times faster than the PI1, it should be good!

image

op_lite – OODB library for C++

This post announces the existence of a class library for C++ called op_lite.  It is the result of something I have been wanting to do for quite some time, so this is a kind of major milestone. The library is a portable, lightweight object persistence library for C++ for Windows (MSVC2010) and Linux (g++), it makes it easy to write C++ applications with in-process persistent objects, i.e. objects that live within a database file.

Download version V1.0-00:  source code and white paper.   If you download and try the library, I would appreciate your comments below this post.

Update 16-Oct-2015: Version V1.0-02 source code available.

image

Introduction

In the world of C++, a large number of excellent open source libraries exist for almost any conceivable purpose. In addition, most of these libraries are cross platform, i.e. they may be used under several different operating systems.  Many database libraries also exist, but In this author’s opinion, it is a problem that the word ‘database’ for many people is synonym with a traditional relational database based on some form of SQL. Although these databases are very powerful, the programming model they impose is not supporting object oriented programming.

What is missing is an open source and portable database library supporting object orientation, allowing the developer to use native C++ classes with persistent instances living naturally in the database. Such systems do exist, but there are unfortunately few open source libraries in this ‘pure’ category. op_lite’s objective is therefore to provide a single process object oriented database library for C++ applications, with support for persistent containers and polymorphic pointers.

The name op_lite stands for Object Persistence – Lightweight. It is a C++ library that offers automatic in-process object persistence of C++ objects, the application code never explicitly reads or writes to the database – it all happens behind the scenes, given that certain programming patterns are followed. This is similar to most other “real” OO databases. The effect is that the objects are perceived to “live in the database”.

Design and implementation

op_lite is implemented as a small C++ class library. The library provides helper classes for managing databases, base classes for deriving user defined persistent classes, and classes for declaring persistent member data.  From the figure below it is clear that op_lite relies on SQLite for low level implementation. Several libraries exist that encapsulate relational databases, but op_lite tries to take a different approach than most of these. In op_lite, the use of SQLite is mostly considered an implementation detail, however a very useful one. Reading the source code of an application using op_lite will not show many signs of SQLite being used, it is mostly hidden from view.

image

All in all, the reasons for using SQLite as back end are
– it is a zero-configuration, in-process engine
– it is proven technology, extensively tested
– it is very efficient
– it is open source
– it is very well documented
– it is portable (both source code and databases)
– it supports virtually unlimited size databases
– it allows using standard SQLite tools for special purpose operations
– it relieves the author of op_lite of developing a competing back end :-)

Using the library

For an in-depth description of the library, please see the white paper mentioned early in this post.  Here,  we just give a small taste for what it looks like. Assuming the application needs a 2-dimensional Point class, containing x- and y- floating point coordinates, declaring it as a persistent class using op_lite may look like something this:

image

An op_lite persistent class needs to inherit from op_object, or from another class derived from op_object. It also needs to have a default constructor, plus override the pure virtual function op_layout declared in op_object. The persistent data members are declared using op_double, one of the supported persistent types:

image

Looking at the Point.cpp implementation, we find more characteristics of persistent classes. Notice that persistent members must be initialised using op_construct taking the member variable as a parameter or op_construct_v1 which also takes an initialisation value of the corresponding transient type.

image

Furthermore, we notice the use of op_bind  in the op_layout overload. Here, each member variable is “bound” so that it will appear in the database.  This is all that is required to read and write data!  Once we have a persistent class like Point, we can create persistent objects in a database this way:

image

In this tiny example, several important aspects are illustrated. One is the use of op_mgr() to create or access databases. In this case we create a new database file with internal logical name “poly_shapes” , stored in the given file path “db_path”.

The next thing that happens is that an op_transaction is declared as a stack object. This starts a database transaction, plus it is a clean way of making sure that the Point instances do not cause memory leak at the termination of the if scope. The transaction causes the objects created within the scope to be committed to the database, plus the transient cache objects are automatically removed, while the persistent objects remain in the database. Finally, the database file is closed.

For more details, including how to restore persistent objects from the database, see the poly_shapes example code in the test example folder that is found in the source code download.

Other features

Sometimes, a persistent object contains a pointer to another persistent object, possibly of a different type.  This is done using the op_ptr<T> template. For example, a persistent pointer to a Point is declared as op_ptr<Point> .  When  op_ptr<Point> is stored in the database, it is represented as text:  “0 Point 123”. The first value (zero) indicates the format, the second value (class name) indicates the concrete type of the object, and the third value is the object’s persistent identifier.

All you have to do to get at this object is to dereference the op_ptr<Point>  variable using the –> operator, just like normal pointers. When you do that, op_lite will create a cached Point instance on your behalf and initiate the member variables with whatever is store in the database. However, this means that op_lite must know which C++ class to instantiate when it sees a text like “Point”. This is achieved via the “type factory”, where an application declares the persistent classes it is using. Such code must be executed at each startup of an op_lite application.

image

Another interesting capability is to be able to use C++ container classes as member variables, as in the example below. In this case, the ShapeCollection  is a persistent class that has a persistent vector of persistent polymorphic pointers as a member variable. Implementation of persistent containers is achieved using of the MessagePack library, which is provided as part of the source code.

image

The following “cheat sheet” provides an overview of the various classes provided in op_lite. For a more complete description, see the white paper and the provided example source code.

image

Building op_lite from source code

There are 2 ways op_lite can built, either using the Code::Blocks project file, or via the provided makefiles generated from the Code::Blocks project file. Using the makefiles is the easiest option as they have been prepared to have few dependencies. Before building op_lite you must download and build boost and MessagePack .

Building with ‘Makefile.msvc’ on Windows

The file ‘Makefile.msvc’ builds op_lite on Windows using MS Visual Studio 2010 (Express or full edition). To use the makefile, open the “Visual Studio Command Prompt (2010)” from the Windows start menu, navigate to the op_lite source directory. Edit the file Makefile.msvc and adjust the two lines on the top, so they point to where boost and MessagePack have been installed and built (replace the bold parts below)

MSGPACK_INCLUDE = E:\\cpde3\\zdep\\3rdparty\\msgpack\\msgpack-c\\include
BOOST_INCLUDE = E:\\cpde3\\zdep\\3rdparty\\boost\\boost_1_55_0

Then run the makefile

$ nmake -f Makefile.msvc

The generated op_lite.lib and op_lite.dll files are found in the .cmp\msvc\bin\Release subfolder.

Building with ‘Makefile’ on Linux

The file ‘Makefile’ builds op_lite under Linux using g++. To use the makefile, open a terminal window in the op_lite source directory. Edit the file Makefile and adjust the two lines on the top, so they point to where boost and MessagePack have been installed and built.

MSGPACK_INCLUDE = /usr/local
BOOST_INCLUDE = /home/ca/home_work/cpde_root/zdep/3rdparty/boost/boost_1_55_0/

Then run the makefile

$ make

The generated libop_lite.so shared object library is found in the .cmp\gcc\bin\Release subfolder.

Feedback wanted

I would like to have your feedback on this library, please comment below. If the interest is sufficient, the library could move to github or similar places. Right now I publish it here as a beta for review. I hope you enjoy it :-)