Weather camera database

At the top of this page, you may have seen the weather station link, showing up-to-date temperature, pressure and more. The page updates automatically every 10 minutes, 24/7. The weather station was installed 31. Jan 2010 taking readings every 10 minutes. Beginning 04 Jan 2012, the frequency of readings was increased to every 5 minutes, and it has been doing that since then, almost without any pause.  31 January 2015 will mark the 5th anniversary of the weather station and the associated web page, pumping out graphs like this:

image

On 03 Sep 2014, my new Raspberry PI based weather camera became operational, it has been capturing images every minute 24/7 since then, adding visual context to the other data. It is easier to understand the causes of the temperature fluctuations when you see what the weather actually looked like at the time.  The weather camera construction and setup will be subject of later post(s) here.

The database for the weather station has collected only  24MB of data since 2010, but capturing images is something completely different, even though the weather camera is set up to save images at half resolution and as JPEG with high compression. Keeping all those images as individual files in the file system, eventually causes trouble. It is also cumbersome to review old images that way.  Therefore, I have started developing a database & viewer for the images and other data produced by the weather camera. Eventually, this will be merged with the weather station data, but first we need to handle the substantial volume of image data coming from the camera.

As a first test of some ideas, I created a prototype database and viewer application to see how responsive it would be. I took all images from October 2014 – more than 44 000 individual images! – and put them all in the same database, resulting in a file size of 2.95GB. Here is a small glimpse of the responsiveness, which I find quite adequate.

Testing the weather camera database

Even though this appears to be working fine, I would like to find ways of compressing the data more, without compromising on the final image quality, so there will be more tests before I settle down on the “final system”. I am thinking of presenting temperature or other graphs alongside the images, to give better overview. The image database is based on SQLite, but I have added my own C++ layer on top of it, which makes it more suitable for object orientation. This layer may become open source at some stage, if it turns out the way I want it.

The next video below, isn’t really about the database as such, but it illustrates with better image quality what the weather camera can do. Of course, producing such time lapse videos from the database is a possible goal.

Weather camera time lapse 04. September 2014

If you are interested in any of this, and want to know more detail, tell me about it!

7 thoughts on “Weather camera database”

  1. Excellent way of telling, and pleasant paragraph to obtain data regarding
    my presentation focus, which i am going to
    present in school.

  2. Very impressive.. An additional weather camera can make the weather view more interesting as well as the data can prove to be extremely useful.

  3. @David: Thank you for the feedback! I have now 11 months of data … almost half a million images …. the database keeps growing in size, and the performance stays the same :-) Yes, I have contemplated setting up another camera, I have the hardware to do it. Perhaps I will use it to further develop the camera design.

  4. Great setup and incredible location there, Carsten. I am very interested in weather and I find your post very inspirational. Just both the new Raspberry Pi 3 and this seems to be the perfect project for it. What do you use to protect the Pi from bad weather.

Leave a Reply

Your email address will not be published. Required fields are marked *