cf.Partition

class cf.Partition(**kwargs)

Bases: object

A partition.

Initialization

Parameters :

data : numpy array-like

The data for the partition. May be a numpy array or any object which shares the numpy __getitem__, dtype, ndim, size, and shape interface

direction : dict or bool

The direction of each dimension of the partition’s data. It is a boolean if the partition’s data is a scalar array, otherwise it is a dictionary keyed by the dimensions’ identities as found in order.

location : list

The location of the partition’s data in the master array.

order : list

The identities of the dimensions of the partition’s data. If the partition’s data a scalar array then it is an empty list.

part : list

The subspace of the partition’s data to be returned when it is accessed. If the partition’s data is to be returned complete then part may be an empty list.

shape : list

The shape of the partition’s data as a subspace of the master array. If the master array is a scalar array then shape is an empty list.

Units : Units

The units of the partition’s data.

Examples

>>> p = Partition(data      = numpy.arange(20).reshape(2,5,1),
                  direction = {'dim0', True, 'dim1': False, 'dim2': True},
                  location  = [(0, 6), (1, 3), (4, 5)],
                  order     = ['dim1', 'dim0', 'dim2'],
                  shape     = [5, 2, 1],
                  Units     = cf.Units('K'),
                  part      = [],

>>> p = Partition(data      = numpy.arange(20).reshape(2,5,1),
                  direction = {'dim0', True, 'dim1': False, 'dim2': True},
                  location  = [(0, 6), (1, 3), (4, 5)],
                  order     = ['dim1', 'dim0', 'dim2'],
                  shape     = [5, 2, 1],
                  Units     = cf.Units('K'),
                  part      = [slice(None, None, -1), [0,1,3,4], slice(None)],

>>> p = Partition(data      = numpy.array(4),
                  direction = True,
                  location  = [(4, 5)],
                  order     = ['dim1'],
                  shape     = [1],
                  Units     = cf.Units('K'),
                  part      = [],

close(save=False)

Close the partition.

Closing the partition is important for memory management. The partition should always be closed after it is conformed to prevent memory leaks.

Closing the partition does one of three things, depending on the values of the partition’s _original and _save attributes and on the save parameter: * Nothing. * Stores the data in a temporary file. * Reverts the partition to a previous state.

Parameters :

save : bool, optional

If True and the partition is not to be reverted to a previous state then force its data to be stored in a temporary file.

Returns :

None

Examples

>>> p.close()

conform(data_order, data_direction, units, save=False, revert_to_file=False)

After a partition has been conformed, the partition must be closed (with the close method) before another partition is conformed, otherwise a memory leak could occur. For example:

>>> order          = partition_array.order
>>> direction      = partition_array.direction
>>> units          = partition_array.units
>>> save           = partition_array.save_to_disk()
>>> revert_to_file = True
>>> for partition in partition_array.flat():
...
...    # Conform the partition
...    partition.conform(order, direction, units, save, revert_to_file)
...
...    # [ Some code to operate on the conformed partition ]
...
...    # Close the partition
...    partition.close()
...
...    # Now move on to conform the next partition 
...
>>>  

Parameters :

order : list

direction : dict

units : Units

save : bool, optional

• If False then the conformed partition’s data is to be kept in memory when the partition’s close method is called.
• If True and revert_to_file is False then the conformed partition’s data will be to be saved to a temporary file on disk when the partition’s close method is called.
• If True and revert_to_file is True and the pre-conformed partition’s data was in memory then the conformed partition’s data will be saved to a temporary file on disk when the partition’s close method is called.
• If True and revert_to_file is True and the pre-conformed partition’s data was on disk then the file pointer will be reinstated when the partition’s close method is called.

revert_to_file : bool, optional

• If False and save is True then the conformed partition’s data will be saved to a temporary file on disk when the partition’s close method is called.
• If True and save is True and the pre-conformed partition’s data was on disk then the file pointer will be reinstated when the partition’s close method is called.
• Otherwise does nothing.

Returns :

out : numpy array

The partition’s conformed data as a numpy array. The same numpy array is stored as an object identity by the partition’s data attribute.

copy()

Return a deep copy.

Equivalent to copy.deepcopy(p).

Returns :

out :

A deep copy.

Examples

>>> q = p.copy()

new_part(indices, dim2position, data_direction)

Return the part attribute updated for new indices.

Does not change the partition in place.

Parameters :

indices : list

dim2position : dict

data_direction : dict

Returns :

out : list

Examples

>>> p.part = p.new_part(indices, dim2position, data_direction)

to_disk()

Store the partition’s data in a temporary file on disk in place.

Assumes that the partition’s data is currently in memory, but this is not checked.

Returns :None

Examples

>>> p.to_disk()

update_from(other)

Completely update the partition with another partition’s attributes in place.

The updated partition is always independent of the other partition.

Parameters :other : Partition Returns :None

Examples

>>> p.update_from(q)

in_memory

True if and only if the partition’s data is in memory as opposed to on disk.

Examples

>>> p.in_memory
False

indices

The indices of the master array which correspond to this partition’s data.

Returns :

out : tuple

A tuple of slice objects or, if the partition’s data is a scalar array, an empty tuple.

Examples

>>> p.data.shape
(5, 7)
>>> p.indices
(slice(0, 5), slice(2, 9))

>>> p.data.shape
()
>>> p.indices
()

is_scalar

True if and only if the partition’s data is a scalar array.

Examples

>>> p.data.ndim
0
>>> p.is_scalar
True

>>> p.data.ndim
1
>>> p.is_scalar
False

on_disk

Return True if the partition’s data is on disk as opposed to in memory.

Examples

>>> p.on_disk
True