In-memory HDF5 files¶
The HDF5 library provides functions to allow an application to work with a file in memory for faster reads and writes. File contents are kept in memory until the file is closed. At closing, the memory version of the file can be written back to disk or abandoned.
Open an existing file in memory¶
sample.h5 exists in the current folder, it is possible to
open it in memory simply using the CORE driver at opening time.
The HDF5 driver that one intend to use to open/create a file can be specified
using the driver keyword argument of the
>>> import tables >>> h5file = tables.open_file("sample.h5", driver="H5FD_CORE")
The content of the :file`sample.h5` is opened for reading. It is loaded into memory and all reading operations are performed without disk I/O overhead.
the initial loading of the entire file into memory can be time expensive depending on the size of the opened file and on the performances of the disk subsystem.
general information about HDF5 drivers can be found in the Alternate File Storage Layouts and Low-level File Drivers 3 section of the HDF5 User’s Guide 1.
Creating a new file in memory¶
Creating a new file in memory is as simple as creating a regular file, just one needs to specify to use the CORE driver:
>>> import tables >>> h5file = tables.open_file("new_sample.h5", "w", driver="H5FD_CORE") >>> import numpy >>> a = h5file.create_array(h5file.root, "array", numpy.zeros((300, 300))) >>> h5file.close()
In the previous example contents of the in-memory h5file are automatically
saved to disk when the file descriptor is closed, so a new
new_sample.h5 file is created and all data are transferred to disk.
Again this can be time a time expensive action depending on the amount of data in the HDF5 file and depending on how fast the disk I/O is.
Saving data to disk is the default behavior for the CORE driver in PyTables.
This feature can be controlled using the driver_core_backing_store
parameter of the
tables.open_file() function. Setting it to False
disables the backing store feature and all changes in the working h5file
are lost after closing:
>>> h5file = tables.open_file("new_sample.h5", "w", driver="H5FD_CORE", ... driver_core_backing_store=0)
Please note that the driver_core_backing_store disables saving of data, not
In the following example the
sample.h5 file is opened in-memory in
append mode. All data in the existing
sample.h5 file are loaded into
memory and contents can be actually modified by the user:
>>> import tables >>> h5file = tables.open_file("sample.h5", "a", driver="H5FD_CORE", driver_core_backing_store=0) >>> import numpy >>> h5file.create_array(h5file.root, "new_array", numpy.arange(20), title="New array") >>> array2 = h5file.root.array2 >>> print(array2) /array2 (Array(20,)) 'New array' >>> h5file.close()
Modifications are lost when the h5file descriptor is closed.
Memory images of HDF5 files¶
In particular getting a memory image of an HDF5 file is possible only if the file has been opened with one of the following drivers: SEC2 (the default one), STDIO or CORE.
An example of how to get an image:
>>> import tables >>> h5file = tables.open_file("sample.h5") >>> image = h5file.get_file_image() >>> h5file.close()
The memory ìmage of the
sample.h5 file is copied into the ìmage
string (of bytes).
the ìmage string contains all data stored in the HDF5 file so, of course, it can be quite large.
The ìmage string can be passed around and can also be used to initialize a new HDF5 file descriptor:
>>> import tables >>> h5file = tables.open_file("in-memory-sample.h5", driver="H5FD_CORE", driver_core_image=image, driver_core_backing_store=0) >>> print(h5file.root.array) /array (Array(300, 300)) 'Array' >>> h5file.setNodeAttr(h5file.root, "description", "In memory file example")