cf.Data

class cf.Data(data=None, units=None, _FillValue=None, chunk=True)

Bases: object

An N-dimensional data array with units and masked values.

• Contains an N-dimensional, indexable and broadcastable array with many similarities to a numpy array.
• Contains the units of the array elements.
• Supports masked arrays, regardless of whether or not it was initialized with a masked array.
• Uses Large Amounts of Massive Arrays (LAMA) functionality to store and operate on arrays which are larger then the available memory.

Indexing

A data array is indexable in a similar way to numpy array indexing but for two important differences:

• Size 1 dimensions are never removed.

  An integer index i takes the i-th element but does not reduce the rank of the output array by one.

• When advanced indexing is used on more than one dimension, the advanced indices work independently.

  When more than one dimension’s slice is a 1-d boolean array or 1-d sequence of integers, then these indices work independently along each dimension (similar to the way vector subscripts work in Fortran), rather than by their elements.

Examples

>>> d.shape
[12, 19, 73, 96]
>>> d[0, :, [0,1], [0,1,2]].shape
[1, 19, 2, 3]

Conversion to a numpy array

The data array may be converted to either a numpy array view or an independent numpy array of the underlying data with the varray and array attributes respectively. Changing a numpy array view in place will also change the data array. Note that the numpy array created with the array or varray attribute forces all of the data to be read into memory at the same time, which may not be possible for very large arrays.

Initialization

Parameters :

data : array-like, optional

The data for the array.

units : str or Units, optional

The units of the data.

_FillValue : object, optional

The fill value of the data. By default the numpy fill value appropriate to the data type will be used.

chunk : bool, optional

If True then the data array will be partitioned if it is larger than the chunk size.

Examples

>>> d = cf.Data(5)
>>> d = cf.Data([1,2,3], units='K')
>>> import numpy   
>>> d = cf.Data(numpy.arange(10).reshape(2,5), units=cf.Units('m/s'), _FillValue=-999)
>>> d = cf.Data(('f', 'l', 'y'))

add_partitions(extra_boundaries, adim, existing_boundaries=None)

Examples

>>> d.add_partitions(    )

all()

Test whether all array elements evaluate to True.

Masked values are considered as True during computation.

Examples

>>> d.array
array([0, 3, 0])
>>> d.all()
False

>>> d.array
array([1, 3, 2])
>>> d.all()
True

any()

Test whether any array elements evaluate to True.

Masked values are considered as True during computation.

Examples

>>> d.array
array([0, 0, 0])
>>> d.any()
False

>>> d.array
array([0, 3, 0])
>>> d.any()
True

change_dimension_names(dim_name_map)

Change the dimension names.

The dimension names are arbitrary (though unique), so mapping them to another arbitrary (though unique) set does not change the data array values, units, dimension directions nor dimension order.

Examples

>>> d.order
['dim0', 'dim1', 'dim2']
>>> dim_name_map
{'dim0': 'dim1',
 'dim1': 'dim0',
 'dim2': 'dim2',
 'dim3': 'dim3'}
>>> d.change_dimension_names(dim_name_map)
>>> d.order
['dim1', 'dim0', 'dim2']

chunk(chunksize=None, extra_boundaries=None, chunk_dims=None)

Parameters :

chunksize : int, optional

extra_boundaries : sequence of lists or tuples, optional

chunk_dims : sequence of lists or tuples, optional

Returns :

extra_boundaries, chunk_dims : {list, list}

Examples

>>> d.chunk()
>>> d.chunk(100000)
>>> d.chunk(extra_boundaries=([3, 6],), chunk_dims=['dim0'])
>>> d.chunk(extra_boundaries=([3, 6], [40, 80]), chunk_dims=['dim0', 'dim1'])

copy()

Return a deep copy.

Equivalent to copy.deepcopy(d).

Returns :

out :

The deep copy.

Examples

>>> e = d.copy()

dump(id=None, omit=())

equals(other, rtol=None, atol=None, traceback=False)

True if two data arrays are logically equal, False otherwise.

Parameters :

other :

The object to compare for equality.

atol : float, optional

The absolute tolerance for all numerical comparisons, By default the value returned by the ATOL function is used.

rtol : float, optional

The relative tolerance for all numerical comparisons, By default the value returned by the RTOL function is used.

traceback : bool, optional

If True then print a traceback highlighting where the two instances differ.

Returns :

out : bool

Whether or not the two instances are equal.

Examples

>>> d.equals(d)
True
>>> d.equals(d + 1)
False

expand_aggregating_dims(adim)

expand_dims(axis=0, dim='None', direction=True)

no check is done for dim already being in self.order

hash()

Return a hash value for the data array.

Note that generating the hash realizes the entire array in memory, which may not be possible for large arrays.

Returns :

out : int

The hash value.

Examples

>>> d.hash()
3632586161869339209

new_dimension_name()

Return a dimension name not being used by the data array.

Note that a partition of the data array may have dimensions which don’t belong to the data array itself.

Returns :

out : str

The new dimension name.

Examples

>>> d.order
['dim1', 'dim0']
>>> d.partitions.info('order')
[['dim0', 'dim0'],
 ['dim1', 'dim0', 'dim2']]
>>> d.new_dimension_name()
'dim3'

override_units(new_units)

Override the data array units in place.

Not to be confused with setting the Units attribute to units which are equivalent to the original units.

This is different to setting the Units attribute, as the new units need not be equivalent to the original ones and the data array elements will not be changed to reflect the new units.

Parameters :

new_units : str or Units

The new units for the data array.

Returns :

None

Examples

>>> d.Units
<CF Units: hPa>
>>> d.first_datum
100000.0
>>> d.override_units('km')
>>> d.Units
<CF Units: km>
>>> d.first_datum
100000.0
>>> d.override_units(cf.Units('watts'))
>>> d.Units
<CF Units: watts>
>>> d.first_datum
100000.0

partition_boundaries()

reverse_dims(axes=None)

Reverse the directions of data array dimensions in place.

Equivalent to indexing the specified dimensions with ::-1.

Parameters :

axes : int or sequence of ints

Reverse the dimensions whose positions are given. By default all dimensions are reversed.

Returns :

out : list of ints

The axes which were reversed, in arbitrary order.

Examples

>>> d.ndim
3
>>> d.reverse_dims()
>>> d.reverse_dims(1)
>>> e = d[::-1, ::-1, :]
>>> d.reverse_dims([0, 1]).equals(e)
True

save_to_disk(itemsize=None)

Parameters :itemsize : int, optional Returns :out : bool

Examples

>>> 
>>>

set_location_map()

squeeze(axes=None)

Remove size 1 dimensions from the shape of the data in place.

Parameters :

axes : int or tuple of ints, optional

The axes to be squeezed given by their positions. If unset then all size one dimensions of the data array are removed.

Returns :

out : tuple of ints

The axes which were squeezed as a tuple of their positions.

Examples

>>> v.shape
[1]
>>> v.squeeze()
>>> v.shape
[]

>>> v.shape
[1, 2, 1, 3, 1, 4, 1, 5, 1, 6, 1]
>>> v.squeeze(axis=2).shape
[1, 2, 3, 1, 4, 1, 5, 1, 6, 1]
>>> v.squeeze(axis=(0,)).shape
[2, 3, 1, 4, 1, 5, 1, 6, 1]
>>> v.squeeze(axis=(2, 4)).shape
[2, 3, 4, 5, 1, 6, 1]
>>> v.squeeze().shape
[2, 3, 4, 5, 6]

to_disk()

to_memory(regardless=False)

Store the data array in memory if it is smaller than the chunk size.

Parameters :

regardless : bool, optional

If True then store the data array in memory regardless of its size.

Returns :

None

Examples

>>> d.to_memory()
>>> d.to_memory(True)

transpose(axes=None)

axes: list of ints, optional

By default, reverse the dimensions, otherwise permute the axes according to the values given.

ufunc(func, *args, **kwargs)

Return a

Units

Deleting the Units attribute actually sets it to undefined units, so the Data object is guaranteed to always have the Units attribute.

Examples

>>> del d.Units
>>> print d.Units
<CF Units: >

array

A numpy array copy the data array.

Examples

>>> a = d.array
>>> type(a)
<type 'numpy.ndarray'>

binary_mask

The binary missing data mask of the data array.

The binary mask has 0 where the data array has missing data and 1 otherwise.

Examples

>>> d.mask.array
array([ True, False,  True, False], dtype=bool)
>>> b = d.binary_mask.array
array([0, 1, 0, 1], dtype=int32)

dtype

Numpy data-type of the data array.

If the array is partitioned internally into sub-arrays with different data-types, then the normal data-type coercion rules apply. For example, if the partitions have data-types ‘int32’ and ‘float32’ then the data array’s data-type will be ‘float32’.

Examples

>>> type(f.dtype)
<type 'numpy.dtype'>
>>> f.dtype
dtype('float64')

first_datum

The first element of the data array.

May be retrieved or set.

Equivalent to x[(0,) * x.ndim].array.item() or x[(0,) * x.ndim] = y

Examples

>>> d.array
array([[1, 2],
       [3, 4]])
>> d.first_datum
1
>> d.first_datum = 999
>> d.array
array([[999,   2],
       [  3,   4]])

is_masked

True if the data array has any masked values.

Examples

>>> d.is_masked
True

is_scalar

True if the data array is a 0-d scalar array.

Examples

>>> d.ndim
0
>>> d.is_scalar
True

>>> d.ndim >= 1
True
>>> d.is_scalar
False

last_datum

The last element of the data array.

May be retrieved or set.

Equivalent to x[(-1,) * x.ndim].array.item() or x[(-1,) * x.ndim] = y

Examples

>>> d.array
array([[1, 2],
       [3, 4]])
>> d.last_datum
4
>> d.last_datum = 999
>> d.array
array([[  1,   2],
       [  3, 999]])

mask

The boolean missing data mask of the data array.

The boolean mask has True where the data array has missing data and False otherwise.

The mask may be set to the equivalent of ‘no missing data’ (i.e. all elements are False) by deleting the attribute.

Examples

>>> d.shape
[12, 73, 96]
>>> m = d.mask
>>> m.dtype
dtype('bool')
>>> m.shape
[12, 73, 96]

>>> del d.mask
>>> d.array.mask
False
>>> import numpy
>>> a.array.mask is numpy.ma.nomask
True

ndim

Number of dimensions in the data array.

Examples

>>> d.shape
[73, 96]
>>> d.ndim
2

shape

List of the data array’s dimension sizes.

Note that this attribute is a list, not a tuple.

Examples

>>> d.shape
[73, 96]

size

Number of elements in the data array.

Examples

>>> d.shape
[73, 96]
>>> d.size
7008

varray

A numpy array view the data array.

Note that making changes to elements of the returned view changes the underlying data.

Examples

>>> a = d.varray
>>> type(a)
<type 'numpy.ndarray'>
>>> a
array([0, 1, 2, 3, 4])
>>> a[0] = 999
>>> d.varray
array([999, 1, 2, 3, 4])