LZO

From GPWiki

LZO is an open source lossless compression library. According to the LZO website (http://www.oberhumer.com/opensource/lzo/), it "offers pretty fast compression and *extremly* fast decompression." It does not support streaming, as the similar compression library Zlib (http://www.gzip.org/zlib/) does. It's licensed under the GNU General Public License.

Libraries like LZO can help speed up load times for games. If data is compressed then there is less data that needs to be transfered from the hard drive, CD-ROM or DVD-ROM, or network, hence speeding things up.

How to use LZO

First, take a look at the official LZO documentation: http://www.oberhumer.com/opensource/lzo/lzodoc.php.

You'll quickly see that there are many different compression algorithms within LZO. Here's a list of them:

1. LZO1
2. LZO1A
3. LZO1B
4. LZO1C
5. LZO1F
6. LZO1X
7. LZO1Y
8. LZO1Z
9. LZO2A

That's a pretty big choice of algorithms. So which one is the best? The short answer is LZO1X. The long answer is that it depends. Here is some info from the official docs:

My experiments have shown that LZO1B is good with a large blocksize or with very redundant data, LZO1F is good with a small blocksize or with binary data and that LZO1X is often the best choice of all. LZO1Y and LZO1Z are almost identical to LZO1X - they can achieve a better compression ratio on some files.

It may take a little bit of experimentation to find which one works the best. Just remember you must use the same algorithm for compression and decompression.

Each algorithm you use, whether for compression or decompression, will require a header file that contains the function prototypes for that algorithm. The header you need is just the name of the algorithm and then ".h." For example, "lzo1x.h" would be used for the LZO1X algorithm.

Building You'll need to build the LZO library from the sources (unless you can find a prebuilt binary version). It's a pretty simple process.

In MinGW? or *nix:

1. ./configure
2. make
3. make install

In MSVC++:

1. Import the files into new project
2. Build

If you've built LZO into a library (as opposed to just importing the code itself into your project) you'll need to tell your linker to link to the LZO library.

Initialization The first thing to do, whether compressing or decompressing, is initialize the LZO library. This is done through the lzo_init() call. The return value should be checked to make sure everything went okay.

if (lzo_init() != LZO_E_OK)
{
    printf("Uh oh! LZO init failed!\n");
    return(-1);
}

Compression

Once you've decided which one you like it's time for another decision. Each algorithm has several different compression levels. The higher the compression level the slower compression will be (although the documentation says that decompression speed is not affected) and the smaller (generally) the compressed data will be. Each algorithm has its own set of compression levels, so we won't go into all of them here. Each compression level requires a certain amount of working memory for the compressor (and in some cases each compression level can have several possible memory requirements!). It sounds confusing, but let's try an example. The LZO1X algorithm has two compression levels: 1 and 999. Level 999 only has one memory requirement, but level 1 has 4!

So how do we know what the memory requirement for our particular compression algorithm and compression level? The exact memory requirement happens to be #define'ed in the algorithm's header file. If you were to take a peek into lzo1x.h, you'd see LZO1X_1_MEM_COMPRESS, LZO1X_1_11_MEM_COMPRESS, LZO1X_1_12_MEM_COMPRESS, LZO1X_1_15_MEM_COMPRESS, and LZO1X_999_MEM_COMPRESS.

If you guessed you might have to use the correct memory requirement with the matching compression algorithm and level, you'd be correct. As an example, let's pretend we decided to use the LZO1X algorithm with a compression level of 1 and a memory requirement of 11. First we'll need to allocate the working memory.

unsigned char * workingMemory = (unsigned char*)malloc(LZO1X_1_11_MEM_COMPRESS);

Next we need some memory to decompress the data into.

unsigned char * outputData = (unsigned char*)malloc( [size of data] );

Whatever the size of the data you're compressing you'll always need to allocate a little bit more for the output data, since (ultimate worst case senario) it's possible that the data you're trying to compress isn't compressible at all and the LZO compression algorithm will actually bloat the data a little bit. A generally safe amount would be the size of the input data plus an additional 16 bytes per 1024 bytes of input data.

Now that we've got everything allocated, let's compress some data!

unsigned int compressedSize;
lzo1x_1_11_compress( data, dataSize, outputData, &compressedSize, workingMemory );

The first parameter is a pointer to the data that is to be compressed. The second is the size of the data. The third parameter is the pointer to the buffer that should hold the compressed data. The fourth parameter is a pointer to an unsigned integer. The size of the compressed data will be put here once compression is finished. The last parameter is a pointer to the working memory buffer.

Hopefully compressedSize is now a bit smaller than dataSize. Now that you've compressed the data you can write it to a file, send it over the network, or email it to an unsuspecting friend who'll surely appreciate your sense of humor.

Decompression

Compressed data doesn't do much good if we can't uncompress it again. Luckily decompression is much easier than compression. There's no memory requirements or compression levels or any of that to worry about. We only have one thing to do:

lzo1x_decompress( outputData, compressedSize, data, &dataSize, NULL);

The first parameter is the data that is to be decompressed (in this case it's the data that we compressed earlier). The second parameter is the size of the compressed data. The third parameter is where the decompressed data should go. The fourth parameter is a pointer to an unsigned integer, and the size of the uncompressed data will go here. The fourth parameter isn't used and should be NULL.

That's it! No compression levels or memory requirements, just a simple function call. Each algorithm has a decompression function like this one, and they all take the same parameters. Easy!

Watch out! There's some important things to remember when compressing and decompressing data. You MUST use the same algorithm to compress and decompress. If you mix them, the results could be bad. You may get lucky and just end up with mostly uncompressed data with only a little garbage in it, or you could cause a crash.

External Links

• Official LZO Docs -- http://www.oberhumer.com/opensource/lzo/lzodoc.php
• Official LZO FAQ -- http://www.oberhumer.com/opensource/lzo/lzofaq.php
• Examples in the downloadable LZO source archive