Test error in test_carray.py in heavy mode

[ivilata]

Running test_carray.py in heavy mode results in the following errors:

HDF5-DIAG: Error detected in HDF5 library version: 1.6.6 thread 0.  Back trace follows.
  #000: H5Dio.c line 667 in H5Dwrite(): file selection+offset not within extent
    major(14): Dataspace interface
    minor(04): Out of range
EHDF5-DIAG: Error detected in HDF5 library version: 1.6.6 thread 0.  Back trace follows.
  #000: H5Dio.c line 667 in H5Dwrite(): file selection+offset not within extent
    major(14): Dataspace interface
    minor(04): Out of range
E
======================================================================
ERROR: Some basic checks for carrays exceeding 2**31 rows
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tables/tests/test_carray.py", line 1995, in setUp
    array[s] = na
  File "tables/array.py", line 562, in __setitem__
    self._modify(startl, stepl, countl, nparr)
  File "hdf5Extension.pyx", line 937, in hdf5Extension.Array._modify
HDF5ExtError: Internal error modifying the elements (H5ARRAYwrite_records returned errorcode -6)

======================================================================
ERROR: Some basic checks for carrays exceeding 2**31 rows
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tables/tests/test_carray.py", line 1995, in setUp
    array[s] = na
  File "tables/array.py", line 562, in __setitem__
    self._modify(startl, stepl, countl, nparr)
  File "hdf5Extension.pyx", line 937, in hdf5Extension.Array._modify
HDF5ExtError: Internal error modifying the elements (H5ARRAYwrite_records returned errorcode -6)

----------------------------------------------------------------------
Ran 2 tests in 0.020s

FAILED (errors=2)

The error is caused by tests Rows64bitsTestCase1 and Rows64bitsTestCase2. This only happens in trunk.

[ivilata]

Printing the length of the array created in Rows64bitsTestCase1.setUp() raises the following exception:

ValueError: __len__() should return >= 0

Looking at the size of the array using h5dump shows that its size has probably been mangled by some overflow:

DATASET "array" {
  DATATYPE  H5T_STD_I8LE
  DATASPACE  SIMPLE { ( 18446744072414584320 ) / ( 18446744072414584320 ) }

[ivilata]

Created branch source:/branches/std-bigarray-147.

[ivilata]

There seems to be a problem with internal conversions from numpy.int64 to Python integers:

>>> import numpy
>>> numpy.__version__
'1.0.3.1'
>>> class C(object):
...   def __len__(self):
...     return numpy.int64(3000000000)
... 
>>> len(C())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: __len__() should return >= 0

However, it seems to be fixed in NumPy 1.0.4:

>>> import numpy
>>> numpy.__version__
'1.0.4'
>>> class C(object):
...   def __len__(object):
...     return numpy.int64(3000000000)
... 
>>> len(C())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to int

Thus, adding .item() in malloc_dims() and Leaf.__len__ (the fix being used in the new branch) may be unnecessary with 1.0.4. The problem doesn't show up in 64-bit platforms, which already use 64-bit integers.

[ivilata]

I think that source:/branches/std-bigarray-147 in its current state fixes the issue. I've upgraded the required version of NumPy to 1.0.4, which doesn't need the .item() workaround.

Waiting to see where the next workaround will be needed is too risky now that numpy.int64 numbers are all over the place, acting as size types. I think that requiring a corrected version of NumPy for 2.1 is a sensible solution.

Reassigning for review.

[faltet]

Yeah. Seems good. In the test of len() of a big CArray, perhaps you may want to use the:

{4:"32-bit", 8:"64-bit"}[numpy.int_(0).itemsize]

trick to check if you are on a 32-bit or 64-bit platform.

Also, it's fortunate for tests that CArray doesn't allocate the data in time creation ;)

[ivilata]

Fixed in r3421 (trunk).