Several new features in Simple DNS Plus v. 5 require a new way to store the DNS zone list on disk.

Previous versions use a standard DNS "boot" file – a plain text file listing each zone line by line.
This format is very simple for anyone to read and update, which is why we have stuck with it this long.
Version 5 however needs to store additional information for each zone and update this much more frequently, making a different storage form necessary.

All the data from the "boot" file is actually cached in memory while Simple DNS Plus is running, so theoretically we could just read in all the data at startup, update the data in memory while running, and then save it all back to a file at shutdown. Only problem is if the computer shuts down unexpectedly (looses power), then the latest changes would not be recorded, leaving a mess.
So we need an efficient way to keep a continuously synchronized copy of the data on disk.

Using a full scale database system such as MySQL or Microsoft SQL Server would be overkill for this.

I spent a few days evaluating lightweight embedded database systems (see previous blog entry), but ultimately decided that this was too slow and added too much overhead and complexity as well.

The solution I eventually came up with was a simple proprietary file structure specifically shaped around the unique data of the DNS zone list with a few design ideas borrowed from xBase.
We’ll call it the "Simple DNS Plus Zone Database" file format – or ".sdpzdb" files.
Simple DNS Plus will store the file position of each record (zone item) in memory eliminating the need for database indexes, and making updates/deletes extremely efficient.

While this provides a lot of benefits, there is one major downside; since the new file format is binary, it can’t just be opened and edited with notepad like the old "boot" file.
We believe that was an important "feature", and so we will do what we can to make it easy to work with the new format as well.

For starters we have put together a small "Zone Database Viewer" application to quickly browse the data.
This can also export the data to the old "boot" file format or a CSV file.

And we plan on providing a .NET library to access the file as a standard "system.data.DataTable" object.

One remaining issue we have to deal with is concurrent updates to the database from different processes. Both the GUI record editor module (editrecs.exe) and the main server module (sdnsmain.exe) can update the zone list – potentially at the same time.
We can solve this either by using file locks, or possibly having all updates go through the main server module. The later would turn Simple DNS Plus into a simple database server, which may open up some other interesting possibilities…
We haven’t decided which way to go just yet. 

I spent the past few days evaluating different embedded database system for possible use in the upcoming Simple DNS Plus v. 5.

Simple DNS Plus v. 5 will be .NET based, so my first requirement was that the database system had a .NET API or data provider.
Ideally it should run on both 32 and 64 bit Windows versions and under MONO on non-Windows systems (Linux, MacOS, Solaris, etc.).
And of course it should be fast and small as possible.

I looked at the following options:

1) Microsoft SQL Server Compact Edition (MS SQL CE).
(http://www.microsoft.com/sql/editions/compact/default.mspx)

At first this looked like the obvious choice since it is practically part of the Visual Studio toolbox (or will be in Visual Studio "Orcas").
However, at this time there is no 64 bit version, which would also prevent Simple DNS Plus from getting certified for Windows Vista (see http://channel9.msdn.com/ShowPost.aspx?PostID=312747).
And from what little information I can find, Microsoft has not committed to a 64 bit version in the "Orcas" timeframe either
(see http://blogs.msdn.com/stevelasker/archive/2007/01/16/sql-server-compact-edition-3-1-released.aspx#1753365)

Of course being win32 code it probably doesn’t run on non-Windows systems.
2) VistaDB
(http://www.vistadb.net)

This database system is a 100% .NET managed code DLL which can be linked directly into another .NET assembly. So this is truly an "embedded" database system.
The same code runs on both 32 and 64 bit – and even on non-Windows systems (under MONO).
The downside is that this is rather slow compared to the other database systems (according to my own simple speed tests), and the documentation is not great.

3) SQLite
(http://www.sqlite.org and http://sqlite.phxsoftware.com)

This was significantly faster than both MS SQL CE and VistaDB.

For .NET development, this is available as a hybrid DLL containing both the native code SQLite engine and the .NET provider.
Because of the included native code, there are separate DLLs for win32, x64, Itanium, and .NET compact which makes the total distribution size significantly larger than the other database engines.

A potential issue is that it has no type safety. Anything can be stored in any column.

I imagine the hybrid DLLs cannot be used on non-Windows systems, but the SQLite engine itself is available for many operating systems and as C source code, so there is probably some way around that…

4) Codebase
(http://www.codebase.com)

About 10 years ago I used this database engine to build a larger invoicing system with VB3 16 bit.
It was VERY fast back then and still is today.
It uses xBase data files which explains the speed (very simple and efficient file layout), but also why the data file becomes very large (data fields are padded to maximum length).
It has a .NET API but is itself native code. I am not sure if it works under MONO.

5) Firebird
(http://www.firebirdsql.org)

I didn’t actually get around to testing this, but wanted to mention it because it looks like another good option.
It has a .NET data provider but is itself native code. I am not sure if it works under MONO.

And the winner is… none!

Basically they were all just overkill for what I needed.
Had I chosen one, it would have been either VistaDB because it best fit my requirements or SQLite because it was very fast.

Instead of a database I ended up using a "homemade" highly optimized proprietary file structure.

A quick comparison of inserting 150.000 random records (representing 150.000 DNS zones):

VistaDB:
2 minutes

MS SQL CE:
1 minute

SQLite:
25 seconds

"Homemade":  
1/3 second (yes, one third of one second)

And the resulting "homemade" file was significantly smaller than any of the database files.

Read more about the "homemade" file structure here...