op_lite – OODB library for C++

This post announces the existence of a class library for C++ called op_lite.  It is the result of something I have been wanting to do for quite some time, so this is a kind of major milestone. The library is a portable, lightweight object persistence library for C++ for Windows (MSVC2010) and Linux (g++), it makes it easy to write C++ applications with in-process persistent objects, i.e. objects that live within a database file.

Download version V1.0-00:  source code and white paper.   If you download and try the library, I would appreciate your comments below this post.

Update 16-Oct-2015: Version V1.0-02 source code available.

image

Introduction

In the world of C++, a large number of excellent open source libraries exist for almost any conceivable purpose. In addition, most of these libraries are cross platform, i.e. they may be used under several different operating systems.  Many database libraries also exist, but In this author’s opinion, it is a problem that the word ‘database’ for many people is synonym with a traditional relational database based on some form of SQL. Although these databases are very powerful, the programming model they impose is not supporting object oriented programming.

What is missing is an open source and portable database library supporting object orientation, allowing the developer to use native C++ classes with persistent instances living naturally in the database. Such systems do exist, but there are unfortunately few open source libraries in this ‘pure’ category. op_lite’s objective is therefore to provide a single process object oriented database library for C++ applications, with support for persistent containers and polymorphic pointers.

The name op_lite stands for Object Persistence – Lightweight. It is a C++ library that offers automatic in-process object persistence of C++ objects, the application code never explicitly reads or writes to the database – it all happens behind the scenes, given that certain programming patterns are followed. This is similar to most other “real” OO databases. The effect is that the objects are perceived to “live in the database”.

Design and implementation

op_lite is implemented as a small C++ class library. The library provides helper classes for managing databases, base classes for deriving user defined persistent classes, and classes for declaring persistent member data.  From the figure below it is clear that op_lite relies on SQLite for low level implementation. Several libraries exist that encapsulate relational databases, but op_lite tries to take a different approach than most of these. In op_lite, the use of SQLite is mostly considered an implementation detail, however a very useful one. Reading the source code of an application using op_lite will not show many signs of SQLite being used, it is mostly hidden from view.

image

All in all, the reasons for using SQLite as back end are
– it is a zero-configuration, in-process engine
– it is proven technology, extensively tested
– it is very efficient
– it is open source
– it is very well documented
– it is portable (both source code and databases)
– it supports virtually unlimited size databases
– it allows using standard SQLite tools for special purpose operations
– it relieves the author of op_lite of developing a competing back end :-)

Using the library

For an in-depth description of the library, please see the white paper mentioned early in this post.  Here,  we just give a small taste for what it looks like. Assuming the application needs a 2-dimensional Point class, containing x- and y- floating point coordinates, declaring it as a persistent class using op_lite may look like something this:

image

An op_lite persistent class needs to inherit from op_object, or from another class derived from op_object. It also needs to have a default constructor, plus override the pure virtual function op_layout declared in op_object. The persistent data members are declared using op_double, one of the supported persistent types:

image

Looking at the Point.cpp implementation, we find more characteristics of persistent classes. Notice that persistent members must be initialised using op_construct taking the member variable as a parameter or op_construct_v1 which also takes an initialisation value of the corresponding transient type.

image

Furthermore, we notice the use of op_bind  in the op_layout overload. Here, each member variable is “bound” so that it will appear in the database.  This is all that is required to read and write data!  Once we have a persistent class like Point, we can create persistent objects in a database this way:

image

In this tiny example, several important aspects are illustrated. One is the use of op_mgr() to create or access databases. In this case we create a new database file with internal logical name “poly_shapes” , stored in the given file path “db_path”.

The next thing that happens is that an op_transaction is declared as a stack object. This starts a database transaction, plus it is a clean way of making sure that the Point instances do not cause memory leak at the termination of the if scope. The transaction causes the objects created within the scope to be committed to the database, plus the transient cache objects are automatically removed, while the persistent objects remain in the database. Finally, the database file is closed.

For more details, including how to restore persistent objects from the database, see the poly_shapes example code in the test example folder that is found in the source code download.

Other features

Sometimes, a persistent object contains a pointer to another persistent object, possibly of a different type.  This is done using the op_ptr<T> template. For example, a persistent pointer to a Point is declared as op_ptr<Point> .  When  op_ptr<Point> is stored in the database, it is represented as text:  “0 Point 123”. The first value (zero) indicates the format, the second value (class name) indicates the concrete type of the object, and the third value is the object’s persistent identifier.

All you have to do to get at this object is to dereference the op_ptr<Point>  variable using the –> operator, just like normal pointers. When you do that, op_lite will create a cached Point instance on your behalf and initiate the member variables with whatever is store in the database. However, this means that op_lite must know which C++ class to instantiate when it sees a text like “Point”. This is achieved via the “type factory”, where an application declares the persistent classes it is using. Such code must be executed at each startup of an op_lite application.

image

Another interesting capability is to be able to use C++ container classes as member variables, as in the example below. In this case, the ShapeCollection  is a persistent class that has a persistent vector of persistent polymorphic pointers as a member variable. Implementation of persistent containers is achieved using of the MessagePack library, which is provided as part of the source code.

image

The following “cheat sheet” provides an overview of the various classes provided in op_lite. For a more complete description, see the white paper and the provided example source code.

image

Building op_lite from source code

There are 2 ways op_lite can built, either using the Code::Blocks project file, or via the provided makefiles generated from the Code::Blocks project file. Using the makefiles is the easiest option as they have been prepared to have few dependencies. Before building op_lite you must download and build boost and MessagePack .

Building with ‘Makefile.msvc’ on Windows

The file ‘Makefile.msvc’ builds op_lite on Windows using MS Visual Studio 2010 (Express or full edition). To use the makefile, open the “Visual Studio Command Prompt (2010)” from the Windows start menu, navigate to the op_lite source directory. Edit the file Makefile.msvc and adjust the two lines on the top, so they point to where boost and MessagePack have been installed and built (replace the bold parts below)

MSGPACK_INCLUDE = E:\\cpde3\\zdep\\3rdparty\\msgpack\\msgpack-c\\include
BOOST_INCLUDE = E:\\cpde3\\zdep\\3rdparty\\boost\\boost_1_55_0

Then run the makefile

$ nmake -f Makefile.msvc

The generated op_lite.lib and op_lite.dll files are found in the .cmp\msvc\bin\Release subfolder.

Building with ‘Makefile’ on Linux

The file ‘Makefile’ builds op_lite under Linux using g++. To use the makefile, open a terminal window in the op_lite source directory. Edit the file Makefile and adjust the two lines on the top, so they point to where boost and MessagePack have been installed and built.

MSGPACK_INCLUDE = /usr/local
BOOST_INCLUDE = /home/ca/home_work/cpde_root/zdep/3rdparty/boost/boost_1_55_0/

Then run the makefile

$ make

The generated libop_lite.so shared object library is found in the .cmp\gcc\bin\Release subfolder.

Feedback wanted

I would like to have your feedback on this library, please comment below. If the interest is sufficient, the library could move to github or similar places. Right now I publish it here as a beta for review. I hope you enjoy it :-)

Åpent brev til Røyken Kommune (1)

To those who do not speak Norwegian: The following is an open letter to the local authorities of Røyken Kommune. Via a “consultant” they “request” that I install a water meter in my house, and in the same sentence they declare that if I object they will force me to do it. A time limit for compliance is given. In the same letter, they also declare it is their intention to apply extensive remote surveillance. There is no water shortage in Norway, quite the opposite, and no credible reason is given, other than they “want to reduce water consumption down to 2004 level”. I have paid all my taxes and bills since I moved here in 1993, I intend to continue to do so and I have no excessive water usage. I suspect the real reason behind this is to establish a way to increase pricing of basic resources like H2O, just like they already have started taxing the exhaling of harmless CO2. I am now waiting for the taxation of O2 intake to be established.  The second suspected reason is what is clearly stated in their letter: Extensive surveillance and control over citizens. All of this is supposedly decided locally. However,  it is centrally organised, disguised as “local democracy” implemented via unelected consultants working for unelected entities I have never heard of.

Nylig mottok jeg et brev fra en “konsulent” ved navn Brynhild Oddrunsdottir, som hevder å opptre på vegne av Røyken Kommune gjennom noe som kalles “Viva iks”. Jeg aner ikke hvem “Viva iks” er, og det er ikke gitt noen dokumentasjon på deres fullmakt til å opptre på vegne av politisk valgte myndigheter. Overskriften i brevet er som følger:

image

I brevet hevdes det at pålegget er et ledd i å “trygge leveransen av drikkevann” og at det totale vannuttaket av en eller annen uspesifisert grunn skal reduseres til “2004-nivå”. Det er ikke gitt noen rasjonell grunn for et slikt mål. Vi bor ikke i Sahara, tvert imot oppstår jevnlig overvann i området her jeg bor. Videre ser det ut til at den som har skrevet dette anser at lovlydige innbyggere som med selvfølgelighet har betalt kommunale avgifter i alle år, nå skal utsettes for vilkårlige pålegg, overvåking og ekstra utgifter.

image

Det hele framstilles som om det er et ledd i lokalt demokrati. Realiteten er at dette er styrt fra sentrale myndigheter og byråkrater, samt eksekvert gjennom obskure konsulentvirsomheter ingen har hørt om. Det som videre er er bekymringsfullt er at den sannsynlige reelle begrunnelsen ikke er uttalt: Etablere grunnlag for framtidig inntektsøkning gjennom økt skattlegging og begrensing av den enkeltes tilgang til fundamentale ressurser som vann (H2O), kombinert med økt overvåking.

image

Sentrale myndigheter har forlengst igangsatt skattlegging av utpust (CO2), uten noen rasjonell begrunnelse utover det som er “etablert” gjennom IPCC, inntil nylig ledet av jernbaneingeniøren Rajendra Pachauri, nå gjenstand for politietterforsking i India.  Nå følges dette opp gjennom uttalt ambisjon om begrenset tilgang på H2O, via overvåking som etablerer grunnlag for kontroll og skattlegging av noe vi allerede betaler for. Når kommer det brev om at vårt inntak av O2 må begrenses til 2004-nivå?

Jeg anmoder konsulenten å skaffe troverdig dokumentasjon som utvetydig viser at hun opptrer lovlig på oppdrag fra demokratisk valgte myndigheter. Brevet jeg har mottatt inneholder ingen slik dokumentasjon. Jeg anmoder også om en mer utfyllende informasjon om hvorfor “Målet er å reduserer det totale vannintaket til 2004-nivå”, samt hvilke andre tiltak kommunen har iverksatt for å oppnå dette målet. Er det den enkeltes vannuttak som skal begrenses til 2004-nivå, eller er det kommunens totale uttak det er snakk om? Hvordan forholder dette seg til endringen av innbyggertallet i kommunen?

Ved mottak av slik dokumentasjon vil jeg rette meg etter anmodningen, om enn ikke med særlig glede. Tilliten til lokale myndigheter er ikke styrket med dette.

A second experiment: Boolean operations

In the previous post, it was shown how it is possible to convert a real object into a 3d computer model, suitable for replication using a 3d printer.  This was done simply using a simple flatbed scanner and some software.  The object chosen there (the wrench/spanner) was 2-dimensional if you ignore the thickness, so some may say this was cheating a bit. Could we achieve a similar effect with a more truly 3-dimensional object? The following object is our second replication challenge:

image

This object is not entirely flat, so it is a more challenging task to create a virtual replica of it.  If we put it on the flatbed scanner and scan it from 2 projections, from below and from the side, we get the result below (scanner lid open). The only thing done here is to present the two projections in the same image and crop away irrelevant areas to the left and right.

image

We now give these images the same treatment as in the first experiment. That means stretching the histogram, blurring the surfaces and using curve tools in a bitmap editor.  The goal is to emphasize the edges in the to projections, and remove anything else in the images. Below, the resulting projections are shown together for illustration purposes, but observe that each projection is treated separately.

thing_AB_photoshop

We then give both of these  images the same treatment as before, using potrace, inkscape and pstoedit. Again, the results are simplified profiles in DXF file format, using only LINE segments:

“thing_A.dxf”

thing_A_dxf

“thing_B.dxf”

thing_B_dxf

This time, we employ some more of the powerful tools of OpenSCAD, that is ‘Boolean operations’.  For the uninitiated it can be compared to mathematical set operations,  for example intersection, union and difference.  But instead of operating on mathematical sets, OpenSCAD operates on 3-dimensional solid objects. Watch what happens if we define 3 solid objects (box, thing_A and thing_B) and subtract them from each other in the right order:

image

Not bad, huh? A small miracle… Again, how did this happen? Look at the solids we used. Below shows “thing_A” in yellow and “thing_B” in transparent grey. These were the bodies extruded from the image projections.

image

We may compare “box” (red) and “thing_A”  (transparent grey) in a similar manner:

image

What happens is two subsequent Boolean operations:

1. The red box is the original positive body, and “thing_A” gets subtracted from it.  That makes the “thing” without the holes.

2. Then, “thing_B” is subtracted from the result of 1. It is as if the holes get punched out using a punching tool. In many ways, that is exactly what happens.

The final result is the green “thing” as shown in OpenSCAD above. We can also save this as an STL file,  a collection of 3d triangles, and present them in wireframe mode:
image

Such triangles are what 3d printers need. Or to be more precise, it is the starting point of 3d printing. When printing, the triangles are cut with horizontal planes from bottom to top, also a kind of Boolean operation, the resulting intersections are horizontal line segments that can be used to generate G-code to steer the printer motors.

But that subject is for some other time.

A reverse 3d-printing experiment

I am in the process of buying a 3d-printer, so a good idea is to look at ways of creating 3d models to print. I have some time yet until I get the printer, so it is a good time to learn about the software you may need to master. Of course, the printer has its own software, but you also have to use other programs that are independent of the actual printer. In this post we shall look at some possibilities using mostly free, open source programs. The main exception is use of Photoshop, but I presume Gimp or even inkscape could do the same job as Photoshop here, I’m just using what I know a bit better.

The starting point when using a 3d printer is a virtual 3d computer model of the object you are printing. But sometimes you have a real object and want to create a 3d replica. This will be the subject of our “reverse 3d-printing experiment”:  Create a 3D computer replica of a real object. Below is our test specimen, a nostalgic object as it is a metal wrench (or ‘spanner’ if you are in the UK) I got as a kid. It came with my very first bicycle.  Can we make a computer replica?

image

If we place the wrench on our cheap flatbed scanner, maybe there is a way to obtain an accurate profile of it? let us try:

image

Below left is the raw output from the flatbed scanner,  a BMP file.  On the right is the same image after slight manipulation using an old Photoshop CS2. The features employed was to select the background with the “colour range” feature,  adding some feathering to create a smooth edge.  Then one left-over background area was clipped.  After that, the inverse of the selection was chosen, and the “levels” feature was used to blacken the wrench. Finally some “Gaussian blur” was applied to the whole image in order to soften the edges even more, and remove any remains of edge highlights from the scanning. The result is basically a black and white image of the wrench. But it is still just a raster image.

image

What we need is a vectorized representation of the wrench edges.  The following steps are a little convoluted, but It can really be simplified by improving the DXF file support in one of the open source programs. But until we have that, we can do the following:

Vectorizing the bitmap image

The first thing we do is to run potrace, a program  that boasts the feature we want:  “Transforming bitmaps into vector graphics” . We are going to require a file in DXF format, describing the wrench profile, and potrace can generate DXF files. However, it creates a DXF file with some features not understood by other programs, so we have to take a detour via SVG format and Encapsulated Postscript (EPS) format before we return to a simpler representation of DXF that can be used. That means a version with only simple LINES. Below is how I did it, using both Windows and linux along the way. First we run ‘potrace’ to get the SVG file from the fixed-up BMP file:

image

Let us copy that SVG file over to a Linux Kubuntu machine and run a couple of programs there. First we install inkscape and another program called pstoedit, based on some tips found here.

$ sudo apt-get install inscape
$ sudo apt-get install pstoedit

Now that we have the required software to complete our vectorization detour, let us use them.  First we create an intermediate EPS file using inkskape

$ inkscape -E intermediate.eps Wrench_fix.svg

Second we create the final, simplified DXF file using pstoedit, using the option “-polyaslines” to create a simplified DXF file with individual, straight lines. No polylines or spline curves. The final vectorized file is ‘wrench_os.dxf’ here

$ pstoedit -dt -f dxf:-polyaslines\ -mm intermediate.eps wrench_os.dxf

We can now open and view the created DXF file in for example LibreOffice and observe what we have created. It is no longer a bitmap image, but instead a trace of the wrench edges, i.e. a series of vectors.

image

Creating a 3D model

This is where the fun begins in earnest.  There is a really good, and totally free program called OpenSCAD which has some extremely powerful features that enables modelling of 3D objects.  This includes so called “Boolean operations” in CSG modelling, but also features for extruding 3D objects from 2D profiles like we have just created.  So let us try the following single command in OpenSCAD and watch what happens:

linear_extrude(height = 10) import(“Wrench_os.dxf”);

From that single line, we got something we recognise!image

What happened here? We had created the wrench profile in the DXF file. To understand what happened, you can read the above OpenSCAD commands right to left.

First, we imported the DXF file containing a profile in the XY-plane. Second, we extruded (a ‘sweep’ if you prefer) the complete profile 10 units in the Z-direction. The result was a totally recognizable virtual 3d wrench, looking just like the original, nostalgic bicycle wrench. 

With this model, we have everything required for creating a 3d printed replica, the next logical step in such a printing process would be to create an STL-file, which simply contains a number of 3-dimensional triangles describing the outer surface of the wrench model.

image

To prove that it works, we can view the generated STL file in a free STL viewer (chosen by random):

image

The STL file is available (zipped) here.

Conclusions

There are some incredibly powerful and free software tools available that can be used in combination with a bit of creativity to arrive at some rather impressive results. This is just great. OpenSCAD is a key tool, so this author will spend some time learning it better.  A great introduction to OpenSCAD are these tutorials (recommended):

How to use Openscad (1), tricks and tips to design a parametric 3D object
How to use Openscad (2): variables and modules for parametric designs
How to use Openscad (3): iterations, extrusions and more modularity!
How to use Openscad (4): children and advanced topics

There will be more on 3d printers from this blog.