Manipulating cf.Field objects

Manipulating a field generally involves operating on its data array and making any necessary changes to the field’s domain to make it consistent with the new array.

Data array

Conversion to a numpy array

A field’s data array may be converted to either an independent numpy array or a numpy array view (numpy.ndarray.view) with its array and varray attributes respectively:

>>> a = f.array
>>> print a
[[2 -- 4 -- 6]]
>>> a[0, 0] = 999
>>> print a
[[999 -- 4 -- 6]]
>>> print f.array
[[2 -- 4 -- 6]]

Changing the numpy array view in place will also change the field’s data array in-place:

>>> v = f.varray
>>> print v
[[2 -- 4 -- 6]]
>>> v[0, 0] = 999
>>> print f.array
[[999 -- 4 -- 6]]

A field supports the numpy array protocol and so may be used as input to any of the numpy array creation functions:

>>> print f.array
[[2 -- 4 -- 6]]
>>> numpy.all(f.array == numpy.ma.array(f))
True
>>> print numpy.ma.array(a, dtype=float)
[[2. -- 4. -- 6.]]

Note

The numpy array created by the varray or array attributes forces all of the data to be read into memory at the same time, which may not be possible for very large arrays.

Data mask

A copy of a field’s missing data mask is returned by its mask attribute.

This mask is an independent field in its own right, and so changes to it will not be seen by the field which generated it. See the assignment section for details on how to edit the field’s mask in place.

Copying

A deep copy of a variable may be created with its copy method, which is functionally equivalent to, but faster than, using the copy.deepcopy function:

>>> g = f.copy()
>>> import copy
>>> g = copy.deepcopy(f)

Copying utilizes LAMA copying functionality.

Subspacing

Subspacing a field means subspacing its data array and its domain in a consistent manner.

A field may be subspaced via its subspace attribute. This attribute returns an object which may be indexed to select a subspace by data array index values (f.subspace[indices]) or called to select a subspace by dimension coordinate array values (f.subspace(**coordinate_values)):

>>> g = f.subspace[0, ...]
>>> g = f.subspace(latitude=30, longitude=cf.wi(0, 90, 'degrees'))

The result of subspacing a field is a new, independent field whose data array and, crucially, any data arrays within the field’s metadata (such as coordinates, ancillary variables, transforms, etc.) are appropriate subspaces of their originals:

>>> print f
air_temperature field summary
-----------------------------
Data            : air_temperature(time(12), latitude(73), longitude(96)) K
Cell methods    : time: mean
Dimensions      : time(12) = [1860-01-16 12:00:00, ..., 1860-12-16 12:00:00]
                : latitude(73) = [-90, ..., 90] degrees_north
                : longitude(96) = [0, ..., 356.25] degrees_east
                : height(1) = [2] m
>>> g = f.subspace[-1, :, 48::-1]
>>> print g
air_temperature field summary
-----------------------------
Data            : air_temperature(time(1), latitude(73), longitude(49)) K
Cell methods    : time: mean
Dimensions      : time(1) = [1860-12-16 12:00:00]
                : latitude(73) = [-90, ..., 90] degrees_north
                : longitude(49) = [180, ..., 0] degrees_east
                : height(1) = [2] m

Subspacing utilizes LAMA subspacing functionality.

Indexing

Subspacing by dimension indices uses an extended Python slicing syntax, which is similar to numpy array indexing:

>>> f.shape
(12, 73, 96)
>>> f.subspace[...].shape
(12, 73, 96)
>>> f.subspace[slice(0, 12), :, 10:0:-2].shape
(12, 73, 5)
>>> lon = f.coord('longitude').array
>>> f.subspace[..., lon<180]

There are two important extensions to the numpy indexing functionality:

  • Size 1 dimensions are never removed.

    An integer index i takes the i-th element but does not reduce the rank of the output array by one:

    >>> f.shape
    (12, 73, 96)
    >>> f.subspace[0].shape
    (1, 73, 96)
    >>> f.subspace[3, slice(10, 0, -2), 95:93:-1].shape
    (1, 5, 2)
    
  • When advanced indexing is used on more than one dimension, the advanced indices work independently.

    When more than one dimension’s slice is a 1-d boolean sequence or 1-d sequence of integers, then these indices work independently along each dimension (similar to the way vector subscripts work in Fortran), rather than by their elements:

    >>> f.shape
    (12, 73, 96)
    >>> f.subspace[:, [0, 72], [5, 4, 3]].shape
    (12, 2, 3)
    

    Note that the indices of the last example would raise an error when given to a numpy array.

Coordinate values

Subspacing by values of 1-d coordinates allows a subspaced field to be defined via coordinate values of its domain. The benefits of subspacing in this fashion are:

  • The dimensions to be subspaced may identified by name.
  • The position in the data array of each dimension need not be known and the dimensions to be subspaced may be given in any order.
  • Dimensions for which no subspacing is required need not be specified.
  • Size 1 dimensions of the domain which are not spanned by the data array may be specified.

Coordinate values are provided as keyword arguments to a call to the subspace attribute. Coordinates are identified by their identity or their dimension’s identifier in the field’s domain.

>>> f.subspace().shape
(12, 73, 96)
>>> f.subspace(latitude=0).shape
(12, 1, 96)
>>> f.subspace(latitude=cf.wi(-30, 30)).shape
(12, 25, 96)
>>> f.subspace(long=cf.ge(270, 'degrees_east'), lat=cf.set([0, 2.5, 10])).shape
(12, 3, 24)
>>> f.subspace(latitude=cf.lt(0, 'degrees_north'))
(12, 36, 96)
>>> f.subspace(latitude=[cf.lt(0, 'degrees_north'), 90])
(12, 37, 96)
>>> import math
>>> f.subspace('exact', longitude=cf.lt(math.pi, 'radian'), height=2)
(12, 73, 48)
>>> f.subspace(height=cf.gt(3))
IndexError: No indices found for 'height' values gt 3
>>> f.subspace(dim2=3.75).shape
(12, 1, 96)

Note that if a comparison function (such as cf.wi) does not specify any units, then the units of the named coordinate are assumed.

Assignment

Elements of a field’s data array may be changed by assigning values directly to a subspace of the field defined by the subspace attribute or by using the setdata method.

Assignment uses LAMA functionality, so it is possible to assign to fields which are larger than the available memory.

Array elements may be set from a field or logically scalar object, using the same metadata-aware broadcasting rules as for field arithmetic and comparison operations. In the subspace case, the object attribute must be broadcastable to the defined subspace, whilst in the setdata case the object must be broadcastable to the field itself.

The treatment of missing data elements depends on the value of field’s hardmask attribute. If it is True then masked elements will not unmasked, otherwise masked elements may be set to any value. In either case, unmasked elements may be set to any value (including missing data).

Set all values to 273.15:

>>> f.subspace[...] = 273.15

or equivalently:

>>> f.setdata(273.15, None)

Double the values where longitude is zero degrees:

>>> index = f.indices(longitude=0)
>>> f.subspace[index] = f.suspace[index] * 2

or equivalently:

>>> f.setdata(f*2, None, longitude=0)

Assignment by one dimensionsal coordinate values is also possible. For example, to set all values lying between 210 and 270 degrees longitude and -5 and 5 degrees latitude to missing data:

>>> f.setdata(cf.masked, None,
...           longitude=cf.wi(210, 270, 'degrees_east'),
...           latitude=cf.wi(-5, 5, 'degrees_north'))

Set all values less than 10 Celcius to 10 Celcius:

>>> x = cf.Data(10, 'K @ 273.15')
>>> f.setdata(x, None, f<x)

Set all values less than 273.15 to 1 and all other values to -1:

>>> f.setdata(1, -1, f<273.15)

Set all values less than 280 and greater than 290 to missing data and multiply all other elements by -1:

>>> f.setdata(cf.masked, -f, (f<280) | (f>290))

Selection

Fields may be tested for matching given conditions with the match method and selected by matching given conditions with the select method. Both methods share the same interface. Conditions may be given on:

Field conditions Example
CF properties 'standard_name'
Coordinate values coord={'latitude': 0}
Coordinate cell sizes cellsize={'time': cf.wi(28, 31, 'days')}
Number of axes rank=3

For example:

>>> f
[<CF Field: eastward_wind(grid_latitude(110), grid_longitude(106)) m s-1>,
 <CF Field: air_temperature(time(12), latitude(73), longitude(96)) K>]
>>> f.match('air_temperature')
[False, True]
>>>  f.select('air_temperature')
[<CF Field: air_temperature(time(12), latitude(73), longitude(96)) K>]
>>>  f.select('air_temperature', rank=2)
[]
>>>  f.select('air_temperature', cvalue={'latitude': cf.gt(0)}, rank=cf.ge(3))
[<CF Field: air_temperature(time(12), latitude(73), longitude(96)) K>]

Any of the match and select arguments may also be used with cf.read to select fields when reading from files:

>>> f = cf.read('file*.nc', match={'match': 'air_temperature', 'rank': cf.gt(2)})

Selection may also be applied to a field, rather than a field list. In this case, the select method returns the field itself, if there is a match:

>>> f
<CF Field: air_temperature(time(12), latitude(73), longitude(96)) K>
>>> f.match('air_temperature')
True
>>> f.select('air_temperature')
<CF Field: air_temperature(time(12), latitude(73), longitude(96)) K>
>>> f.select('eastward_wind')
[]

Aggregation

Fields are aggregated into as few multidimensional fields as possible with the cf.aggregate function, which implements the CF aggregation rules.

>>> f
[<CF Field: air_temperature(time(12), latitude(73), longitude(96)) K>,
 <CF Field: air_temperature(latitude(73), longitude(96)) K @ 273.15>]
>>> print f
air_temperature field summary
-----------------------------
Data            : air_temperature(time(12), latitude(73), longitude(96)) K
Cell methods    : time: mean
Dimensions      : time(12) = [1860-01-16 12:00:00, ..., 1860-12-16 12:00:00]
                : latitude(73) = [-90, ..., 90] degrees_north
                : longitude(96) = [0, ..., 356.25] degrees_east
                : height(1) = [2] m
air_temperature field summary
-----------------------------
Data            : air_temperature(latitude(73), longitude(96)) K @ 273.15
Cell methods    : time: mean
Dimensions      : time(12) = [1859-12-16 12:00:00]
                : longitude(96) = [356.25, ..., 0] degrees_east
                : latitude(73) = [-90, ..., 90] degrees_north
                : height(1) = [2] m
...
>>> g = cf.aggregate(f)
>>> g
[<CF Field: air_temperature(time(13), latitude(73), longitude(96)) K>]
>>> print g
air_temperature field summary
-----------------------------
Data            : air_temperature(time(13), latitude(73), longitude(96)) K
Cell methods    : time: mean
Dimensions      : time(13) = [1859-12-16 12:00:00, ..., 1860-12-16 12:00:00]
                : latitude(73) = [-90, ..., 90] degrees_north
                : longitude(96) = [0, ..., 356.25] degrees_east
                : height(1) = [2] m

By default, the fields returned by cf.read have been aggregated:

>>> f = cf.read('file*.nc')
>>> len(f)
1
>>> f = cf.read('file*.nc', aggregate=False)
>>> len(f)
12

Arithmetic and comparison

Arithmetic, bitwise and comparison operations are defined on a field as element-wise operations on its data array which yield a new cf.Field object or, for augmented assignments, modify the field’s data array in-place.

Comparison operators

__lt__ The rich comparison operator <
__le__ The rich comparison operator <=
__eq__ The rich comparison operator ==
__ne__ The rich comparison operator !=
__gt__ The rich comparison operator >
__ge__ The rich comparison operator >=

Binary arithmetic operators

__add__ The binary arithmetic operation +
__sub__ The binary arithmetic operation -
__mul__ The binary arithmetic operation *
__div__ The binary arithmetic operation /
__truediv__ The binary arithmetic operation / (true division)
__floordiv__ The binary arithmetic operation //
__pow__ The binary arithmetic operations ** and pow

Binary arithmetic operators with reflected (swapped) operands

__radd__ The binary arithmetic operation + with reflected operands
__rsub__ The binary arithmetic operation - with reflected operands
__rmul__ The binary arithmetic operation * with reflected operands
__rdiv__ The binary arithmetic operation / with reflected operands
__rtruediv__ The binary arithmetic operation / (true division) with reflected
__rfloordiv__ The binary arithmetic operation // with reflected operands
__rpow__ The binary arithmetic operations ** and pow with reflected

Augmented arithmetic assignments

__iadd__ The augmented arithmetic assignment +=
__isub__ The augmented arithmetic assignment -=
__imul__ The augmented arithmetic assignment *=
__idiv__ The augmented arithmetic assignment /=
__itruediv__ The augmented arithmetic assignment /= (true division)
__ifloordiv__ The augmented arithmetic assignment //=
__ipow__ The augmented arithmetic assignment **=

Unary arithmetic operators

__neg__ The unary arithmetic operation -
__pos__ The unary arithmetic operation +
__abs__ The unary arithmetic operation abs

Binary bitwise operators

__and__ The binary bitwise operation &
__or__ The binary bitwise operation |
__xor__ The binary bitwise operation ^
__lshift__ The binary bitwise operation <<
__rshift__ The binary bitwise operation >>

Binary bitwise operators with reflected (swapped) operands

__rand__ The binary bitwise operation & with reflected operands
__ror__ The binary bitwise operation | with reflected operands
__rxor__ The binary bitwise operation ^ with reflected operands
__rlshift__ The binary bitwise operation << with reflected operands
__rrshift__ The binary bitwise operation >> with reflected operands

Augmented bitwise assignments

__iand__ The augmented bitwise assignment &=
__ior__ The augmented bitwise assignment |=
__ixor__ The augmented bitwise assignment ^=
__ilshift__ The augmented bitwise assignment <<=
__irshift__ The augmented bitwise assignment >>=

Unary bitwise operators

__invert__ The unary bitwise operation ~

A field’s data array is modified in a very similar way to how a numpy array would be modified in the same operation, i.e. broadcasting ensures that the operands are compatible and the data array is modified element-wise.

Broadcasting is metadata-aware and will automatically account for arbitrary configurations, such as dimension order, but will not allow fields with incompatible metadata to be combined, such as adding a field of height to one of temperature.

The resulting field’s metadata will be very similar to that of the operands which are also fields. Differences arise when the existing metadata can not correctly describe the newly created field. For example, when dividing a field with units of metres by one with units of seconds, the resulting field will have units of metres per second.

Arithmetic and comparison utilizes LAMA functionality so data arrays larger than the available physical memory may be operated on.

Broadcasting

The term broadcasting describes how data arrays of the operands with different shapes are treated during arithmetic, comparison and assignment operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.

The general broadcasting rules are similar to the broadcasting rules implemented in numpy, the only difference occurring when both operands are fields, in which case the fields are temporarily conformed so that:

  • The fields have matching units.
  • Dimensions are aligned according to their coordinates’ metadata to ensure that matching dimensions are broadcast against each other.
  • Common dimensions have matching axis directions.

This restructuring of the field ensures that the matching dimensions are broadcast against each other.

Broadcasting is done without making needless copies of data and so is usually very efficient.

Valid operands

A field may be combined or compared with the following objects:

Object Description
int, long, float The field’s data array is combined with the python scalar
cf.Data with size 1

The field’s data array is combined with the cf.Data object’s scalar value, taking into account:

  • Different but equivalent units
cf.Field

The two field’s must satisfy the field combination rules. The fields’ data arrays and domains are combined taking into account:

  • Dimension identities
  • Array units
  • Dimension orders
  • Dimension directions
  • Missing data values

A field may appear on the left or right hand side of an operator.

Warning

Combining a numpy array on the left with a field on the right does work, but will give generally unintended results – namely a numpy array of fields.

Resulting metadata

When creating a new field which has different physical properties to the input field(s) the units will also need to be changed:

>>> f.units
'K'
>>> f += 2
>>> f.units
'K'
>>> f.units
'K'
>>> f **= 2
>>> f.units
'K2'
>>> f.units, g.units
('m', 's')
>>> h = f / g
>>> h.units
'm s-1'

When creating a new field which has a different domain to the input fields, the new domain will in general contain the superset of dimensions from the two input fields, but may not have some of either input field’s auxiliary coordinates or size 1 dimension coordinates. Refer to the field combination rules for details.

Floating point errors

It is possible to set the action to take when an arithmetic operation produces one of the following floating-point errors:

Error Description
Division by zero Infinite result obtained from finite numbers.
Overflow Result too large to be expressed.
Invalid operation Result is not an expressible number, typically indicates that a NaN was produced.
Underflow Result so close to zero that some precision was lost.

For each type of error, one of the following actions may be chosen:

  • Take no action. Allows invalid values to occur in the result data array.
  • Print a RuntimeWarning (via the Python warnings module). Allows invalid values to occur in the result data array.
  • Raise a FloatingPointError exception.

The treatment of floating-point errors is set with cf.Data.seterr. Converting invalid numbers to masked values after an arithmetic operation may be done with the cf.Field.mask_invalid method. It is also possible to mask invalid numbers during arithmetic operations (see cf.Data.mask_fpe).

Note that these setting apply to all data array arithmetic within the cf package.

Data manipulation methods

A field has methods which manipulate the its data array. Many of these behave similarly to their numpy counterparts with the same name but always change the field’s data array in-place. New, independent fields with the same changes may be created with equivalently named module functions.

Field method Description Function
clip Clip (limit) the values in the data array cf.clip
Collapse by statistical operations cf.collapse
cos Trigonometric cosine of the data array cf.cos
expand_dims Expand the shape of the data array cf.expand_dims
flip Flip dimensions of the field cf.flip
sin Trigonometric sine of the data array cf.sin
squeeze Remove size 1 dimensions from the field’s data array cf.squeeze
transpose Permute the dimensions of the data array cf.transpose
unsqueeze Insert size 1 dimensions from the field’s domain into its data array cf.unsqueeze

Manipulating other variables

A field is a subclass of cf.Variable, and that class and other subclasses of cf.Variable share generally similar behaviours and methods:

Class Description
cf.AuxiliaryCoordinate A CF auxiliary coordinate construct.
cf.CellMeasure A CF cell measure construct containing information that is needed about the size, shape or location of the field’s cells.
cf.Coordinate Base class for storing a coordinate.
cf.CoordinateBounds A CF coordinate’s bounds object containing cell boundaries or intervals of climatological time.
cf.DimensionCoordinate A CF dimension coordinate construct.
cf.Variable Base class for storing a data array with metadata.

In general, different dimension identities, different dimension orders and different dimension directions are not considered, since these objects do not contain a coordinate system required to define these properties (unlike a field).

Coordinates

Coordinates are a special case as they may contain a data array for their coordinate bounds which needs to be treated consistently with the main coordinate array. If a coordinate has bounds then all coordinate methods also operate on the bounds in a consistent manner:

>>> c
<CF Coordinate: latitude(73, 96)>
>>> c.bounds
<CF CoordinateBounds: (73, 96, 4)>
>>> d = c.subspace[0:10]
>>> d.shape
(10, 96)
>>> d.bounds.shape
(10, 96, 4)
>>> d.transpose([1, 0])
>>> d.shape
(96, 10)
>>> d.bounds.shape
(96, 10, 4)

Note

If the coordinate bounds are operated on independently, care should be taken not to break consistency with the parent coordinate.