cf.Partition

class cf.Partition(**kwargs)

Bases: object

A partition.

Initialization

Parameters :

data : numpy array-like, optional

The data for the partition. Must be a numpy array or any array storing object with a similar interface.

direction : dict or bool, optional

The direction of each dimension of the partition’s data. It is a boolean if the partition’s data is a scalar array, otherwise it is a dictionary keyed by the dimensions’ identities as found in order.

location : list, optional

The location of the partition’s data in the master array.

order : list, optional

The identities of the dimensions of the partition’s data. If the partition’s data a scalar array then it is an empty list.

part : list, optional

The subspace of the partition’s data to be returned when it is accessed. If the partition’s data is to be returned complete then part may be an empty list.

shape : list, optional

The shape of the partition’s data as a subspace of the master array. If the master array is a scalar array then shape is an empty list.

Units : Units, optional

The units of the partition’s data.

Examples

>>> p = Partition(data      = numpy.arange(20).reshape(2,5,1),
...               direction = {'dim0', True, 'dim1': False, 'dim2': True},
...               location  = [(0, 6), (1, 3), (4, 5)],
...               order     = ['dim1', 'dim0', 'dim2'],
...               shape     = [5, 2, 1],
...               part      = [],
...               Units     = cf.Units('K'))

>>> p = Partition(data      = numpy.arange(20).reshape(2,5,1),
...               direction = {'dim0', True, 'dim1': False, 'dim2': True},
...               location  = [(0, 6), (1, 3), (4, 5)],
...               order     = ['dim1', 'dim0', 'dim2'],
...               shape     = [5, 2, 1],
...               part      = [slice(None, None, -1), [0,1,3,4], slice(None)],
...               Units     = cf.Units('K'))

>>> p = Partition(data      = numpy.array(4),
...               direction = True,
...               location  = [(4, 5)],
...               order     = ['dim1'],
...               shape     = [1],
...               part      = [],
...               Units     = cf.Units('K'))

Partition attributes

data
direction
in_memory	True if and only if the partition’s data is in memory as opposed to on disk.
indices	The indices of the master array which correspond to this partition’s data.
isscalar	True if and only if the partition’s data is a scalar array.
location
on_disk	Return True if the partition’s data is on disk as opposed to in memory.
order
part
shape

Partition methods

close	Close the partition.
conform	After a partition has been conformed, the partition must be closed (with the close method) before another partition is conformed, otherwise a memory leak could occur.
copy	Return a deep copy.
iterarray_indices	Return an iterator over indices of the conformed array.
itermaster_indices	Return an iterator over indices of the master array which are spanned by the conformed array.
new_part	Return the part attribute updated for new indices.
to_disk	Store the partition’s data in a temporary file on disk in place.
update_from	Completely update the partition with another partition’s attributes in place.

close(save=False)

Close the partition.

Closing the partition is important for memory management. The partition should always be closed after it is conformed to prevent memory leaks.

Closing the partition does one of three things, depending on the values of the partition’s _original and _save attributes and on the save parameter: * Nothing. * Stores the data in a temporary file. * Reverts the partition to a previous state.

Parameters :

save : bool, optional

If True and the partition is not to be reverted to a previous state then force its data to be stored in a temporary file.

Returns :

None

Examples

>>> p.close()

conform(order=None, direction=None, units=None, save=False, revert_to_file=False, hardmask=True, dtype=None)

After a partition has been conformed, the partition must be closed (with the close method) before another partition is conformed, otherwise a memory leak could occur. For example:

>>> order          = partition_array.order
>>> direction      = partition_array.direction
>>> units          = partition_array.units
>>> save           = partition_array.save_to_disk()
>>> revert_to_file = True
>>> for partition in partition_array.flat():
...
...    # Conform the partition
...    partition.conform(order, direction, units, save, revert_to_file)
...
...    # [ Some code to operate on the conformed partition ]
...
...    # Close the partition
...    partition.close()
...
...    # Now move on to conform the next partition 
...
>>>  

Parameters :

order : list

direction : dict

units : Units

save : bool, optional

• If False then the conformed partition’s data is to be kept in memory when the partition’s close method is called.
• If True and revert_to_file is False then the conformed partition’s data will be to be saved to a temporary file on disk when the partition’s close method is called.
• If True and revert_to_file is True and the pre-conformed partition’s data was in memory then the conformed partition’s data will be saved to a temporary file on disk when the partition’s close method is called.
• If True and revert_to_file is True and the pre-conformed partition’s data was on disk then the file pointer will be reinstated when the partition’s close method is called.

revert_to_file : bool, optional

• If False and save is True then the conformed partition’s data will be saved to a temporary file on disk when the partition’s close method is called.
• If True and save is True and the pre-conformed partition’s data was on disk then the file pointer will be reinstated when the partition’s close method is called.
• Otherwise does nothing.

dtype : numpy.dtype, optional

Convert the data array to this data type. By default no conversion occurs.

hardmask : bool, optional

If False then force the data array’s mask to be soft. By default the mask is forced to be hard.

Returns :

out : numpy array

The partition’s conformed data as a numpy array. The same numpy array is stored as an object identity by the partition’s data attribute.

copy()

Return a deep copy.

Equivalent to copy.deepcopy(p).

Returns :

out :

A deep copy.

Examples

>>> q = p.copy()

iterarray_indices()

Return an iterator over indices of the conformed array.

The array is not conformed.

Returns :

out : generator

An iterator over indices of the conformed array.

Examples

>>> p.shape
[2, 1, 3]
>>> for index in p.iterarray_indices():
...     print index
...
(0, 0, 0)
(0, 0, 1)
(0, 0, 2)
(1, 0, 0)
(1, 0, 1)
(1, 0, 2)

itermaster_indices()

Return an iterator over indices of the master array which are spanned by the conformed array.

The array is not conformed.

Returns :

out : generator

An iterator over indices of the master array which are spanned by the conformed array.

Examples

>>> p.location
[(3, 5), (0, 1), (0, 3)]
>>> for index in p.itermaster_indices():
...     print index
...
(3, 0, 0)
(3, 0, 1)
(3, 0, 2)
(4, 0, 0)
(4, 0, 1)
(4, 0, 2)

new_part(indices, dim2position, data_direction)

Return the part attribute updated for new indices.

Does not change the partition in place.

Parameters :

indices : list

dim2position : dict

data_direction : dict

Returns :

out : list

Examples

>>> p.part = p.new_part(indices, dim2position, data_direction)

to_disk()

Store the partition’s data in a temporary file on disk in place.

Assumes that the partition’s data is currently in memory, but this is not checked.

Returns :None

Examples

>>> p.to_disk()

update_from(other)

Completely update the partition with another partition’s attributes in place.

The updated partition is always independent of the other partition.

Parameters :other : Partition Returns :None

Examples

>>> p.update_from(q)

in_memory

True if and only if the partition’s data is in memory as opposed to on disk.

Examples

>>> p.in_memory
False

indices

The indices of the master array which correspond to this partition’s data.

Returns :

out : tuple

A tuple of slice objects or, if the partition’s data is a scalar array, an empty tuple.

Examples

>>> p.data.shape
(5, 7)
>>> p.indices
(slice(0, 5), slice(2, 9))

>>> p.data.shape
()
>>> p.indices
()

isscalar

True if and only if the partition’s data is a scalar array.

Examples

>>> p.data.ndim
0
>>> p.isscalar
True

>>> p.data.ndim
1
>>> p.isscalar
False

on_disk

Return True if the partition’s data is on disk as opposed to in memory.

Examples

>>> p.on_disk
True