PyTables as a Computing Kernel and Compressed Container for Arbitrarily Large Datasets
With the addition of the tables.Expr class and Blosc in 2.2 release, PyTables can be used as a very efficient computing kernel for both out-of-memory and in-memory computations and becomes an excellent replacement of memmap-like solutions (like the numpy.memmap module), with a some important advantages.
First, tables.Expr combines the highly efficient Numexpr JIT, and great I/O capabilities of PyTables/HDF5 for performing complex expressions with homogeneous data containers in PyTables (like Array, CArray, EArray or Column), instead of using the typical NumPy array objects. Second, PyTables offer enhanced compressed containers via Blosc that allows to reduce memory consumption and barely affecting computing performance.
This combination is so powerful, that the evaluation of expressions is generally faster (and sometimes much faster) than using a numpy.memmap approach for out-of-memory computations. Even if datasets are small enough to fit in-memory easily, tables.Expr can even be faster than a pure NumPy computation. This is because PyTables I/O for operands and results (using compressed containers) is done against OS filesystem cache (i.e. in-memory), not directly to disk. Also, when using compressed containers, the I/O to filesystem cache can be reduced quite a few, compensating for the required additional compression/decompression time.
Example: Computing a Polynomial
To show you how effective this combination can be, consider the problem to compute the evaluation of a given polynomial (".25*x**3 + .75*x**2 - 1.5*x - 2") on a certain range ([-1, 1]) with some granularity (2e-7) in x axis. The working set for this is around 160 MB, so everything can be done in memory with a reasonably modern computer. We are going to perform this computation using pure numpy, numpy.memmap and tables.Expr methods. You can run this script (using a PyTables version > 2.2b3) and you will get something like this output (using an Intel Core2 E8400 @ 3GHz processor).
In the attached plot you can see these results more graphically. To start with, tables.Expr is about 10x faster than numpy.memmap and more than 5x faster than the pure numpy approach. This is basically due to the combination of numexpr and high I/O capabilities of PyTables. Secondly, it can be noted that when using Blosc compressor, with compression level 1 (i.e. the fastest), tables.Expr can perform computations at approximately the same speed (barely a 5% slower) than the uncompressed case. Higher compression levels give perceptible slowdown in computation time (around a 70%), but compression ratio improves considerably.
Already existing compressors expose better compression ratio (most specially zlib and bzip2), but they slowdown these in-memory computations considerably. Even LZO, which was one of the fastest compressors available (before Blosc was introduced), is slowing down computations by a 170%.
The Out-Of-Core Computation Paradigm Is Here To Stay
This new paradigm is a really great achievement in the computation world, and the advanced techniques of (making efficient use of memory hierarchies and using compression to save I/O to the memory subsystem) will probably be extensively used in the future for performing computations while reducing memory usage.
Another advantage of this approach is that it is able to perform computations no matter they fit in memory or not, as this is using an out-of-core paradigm for doing computations, and all the memory layers in modern computers (L1, L2, L3 caches, RAM and SSD disks) can be effectively used as caches during your data manipulation. So, you won't need to implement different versions of your code depending whether your data fit in memory or not; just do a single version using this out-of-core paradigm and you are done (unless you have to deal with very small datasets that can fit in your processor caches, where NumPy will still probably expose better performance).
Now, do you still think that using out-of-core computations and compression would slow down your computations? Think twice!