Field manipulation ================== Manipulating a field generally involves operating on its data array and making any necessary changes to the field's space to make it consistent with the new array. Data array ---------- A field's data array is stored by the *Data* attribute as a :class:`.Data` object: >>> type(f.Data) This :class:`.Data` object: * Contains an N-dimensional array with many similarities to a :ref:`numpy array ` * Contains the :ref:`units ` of the array elements. * Uses :ref:`LAMA functionality ` to store and operate on arrays which are larger then the available memory. * Supports masked arrays [1]_, regardless of whether or not it was initialized with a masked array. Data mask ^^^^^^^^^ The data array's mask may be retrieved and deleted with the field's :attr:`~cf.Field.mask` attribute. The mask is returned as a :class:`.Data` object: >>> f.shape (12, 73, 96) >>> m = f.mask >>> type(m) >>> m.dtype dtype('bool') >>> m.shape [12, 73, 96] >>> m.array.shape (12, 73, 96) >>> del f.mask >>> f.array.mask False >>> import numpy >>> f.array.mask is numpy.ma.nomask True Conversion to a numpy array ^^^^^^^^^^^^^^^^^^^^^^^^^^^ A field's data array may be converted to either a numpy array view (:meth:`numpy.ndarray.view`) or an independent numpy array of the underlying data with the :meth:`~cf.Field.varray` and :meth:`~cf.Field.array` attributes respectively: >>> a = d.array >>> type(a) >>> v = d.varray >>> type(v) >>> type(v.base) Changing a numpy array view in place will also change the data array: >>> d.array array([1, 2, 3]) >>> v = d.varray >>> v[0] = -999 >>> d.array array([-999, 2, 3]) .. warning:: The numpy array created with the *array* or *varray* attribute forces all of the data to be read into memory at the same time, which may not be possible for very large arrays. Copying ------- A deep copy of a variable may be created with its :meth:`~cf.Field.copy` method or equivalently with the :func:`copy.deepcopy` function: >>> g = f.copy() >>> import copy >>> g = copy.deepcopy(f) Copying utilizes :ref:`LAMA copying functionality `. .. _Subspacing: Subspacing ---------- Subspacing a field means subspacing its data array and its space in a consistent manner. A field may be subspaced with its :attr:`~cf.Field.subspace` attribute. This attribute may be **indexed** to select a subspace from dimension index values (``f.subspace[indices]``) or **called** to select a subspace from dimension coordinate array values (``f.subpacet(coordinate_values)``): >>> g = f.subspace[0, ...] >>> g = f.subspace(latitude=30, longitude=cf.inside(0, 90, 'degrees')) The result of subspacing a field is a new field whose data array and, crucially, any data arrays within the field's metadata (such as coordinates, for example) are subspaces of their originals: >>> print f Data : air_temperature(time, latitude, longitude) Cell methods : time: mean Dimensions : time(12) = [15, ..., 345] days since 1860-1-1 : latitude(73) = [-90, ..., 90] degrees_north : longitude(96) = [0, ..., 356.25] degrees_east : height(1) = [2] m Auxiliary coords: >>> g = f.subspace[-1, :, 48::-1] >>> print g Data : air_temperature(time, latitude, longitude) Cell methods : time: mean Dimensions : time(1) = [345] days since 1860-1-1 : latitude(73) = [-90, ..., 90] degrees_north : longitude(49) = [180, ..., 0] degrees_east : height(1) = [2] m Auxiliary coords: The new subspaced field is independent of the original. Subspacing utilizes :ref:`LAMA subspacing functionality `. .. _indexing: Indexing ^^^^^^^^ Subspacing by dimension indices uses an extended Python slicing syntax, which is similar :ref:`numpy array indexing `. There are two extensions to the numpy indexing functionality: * **Size 1 dimensions are never removed.** An integer index `i` takes the `i`-th element but does not reduce the rank of the output array by one. * **When advanced indexing is used on more than one dimension, the advanced indices work independently.** When more than one dimension's slice is a 1-d boolean array or 1-d sequence of integers, then these indices work independently along each dimension (similar to the way vector subscripts work in Fortran), rather than by their elements. >>> print f Data : air_temperature(time, latitude, longitude) Cell methods : time: mean Dimensions : time(12) = [15, ..., 345] days since 1860-1-1 : latitude(73) = [-90, ..., 90] degrees_north : longitude(96) = [0, ..., 356.25] degrees_east : height(1) = [2] m Auxiliary coords: >>> f.shape (12, 73, 96) >>> f.subspace[...].shape (12, 73, 96) >>> f.subspace[0].shape (1, 73, 96) >>> f.subspace[0,...].shape (1, 73, 96) >>> f.subspace[::-1, ...].shape (12, 73, 96) >>> f.subspace[0:5, ..., slice(None, None, 2)].shape (5, 73, 48) >>> lon = f.coord('longitude').array >>> f.subspace[..., lon<90] (12, 73, 24) >>> f.subspace[[1,2], [1,2,3], [1,2,3,4]].shape (2, 3, 4) Note that the indices of the last example above would raise an error when given to a numpy array. Coordinate values ^^^^^^^^^^^^^^^^^ Subspacing by coordinate values allows a subspaced field to be defined by particular coordinate values of its space. Subspacing by coordinate values is functionally equivalent to subspacing by :ref:`indexing ` -- internally, the selected coordinate values are in fact converted to dimension indices. Coordinate values are provided as arguments to a **call** to the *subspace* method. The benefits to subspacing in this fashion are: * The dimensions to be subspaced are identified by name. * The position in the data array of each dimension need not be known. * Dimensions for which no subspacing is required need not be specified. * Size 1 dimensions of the space which are not spanned by the data array may be specified. >>> print f Data : air_temperature(time, latitude, longitude) Cell methods : time: mean Dimensions : time(12) = [15, ..., 345] days since 1860-1-1 : latitude(73) = [-90, ..., 90] degrees_north : longitude(96) = [0, ..., 356.25] degrees_east : height(1) = [2] m Auxiliary coords: >>> f.subspace(latitude=0).shape (12, 1, 96) >>> f.subspace(latitude=cf.inside(-30, 30)).shape (12, 25, 96) >>> f.subspace(longitude=cf.ge(270, 'degrees_east'), latitude=[0, 2.5, 10]).shape (12, 3, 24) >>> f.subspace(latitude=cf.lt(0, 'degrees_north')) (12, 36, 96) >>> f.subspace(latitude=[cf.lt(0, 'degrees_north'), 90]) (12, 37, 96) >>> import math >>> f.subspace(longitude=cf.lt(math.pi, 'radian'), height=2) (12, 73, 48) >>> f.subspace(height=cf.gt(3)) IndexError: No indices found for 'height' values gt 3 Note that if a comparison function (such as :func:`~cf.inside`) does not specify any units, then the units of the named coordinate are assumed. Selection --------- Fields may be tested for matching given conditions and selected according to those matches with the :meth:`~cf.Field.match` and :meth:`~cf.Field.subset` methods. Conditions may be given on: * The field's CF properties (*prop* parameter). * Any other of the field's attributes (*attr* parameter). * The field's coordinate values (*coord* parameter). * The field's coordinate cell sizes (*cellsize* parameter). >>> f >>> f.match(attr={'ncvar': 'tas'}) True >>> g = f.subset(attr={'ncvar': 'tas'}) >>> g is f True >>> f [, ] >>> f.match(prop={'standard_name': '.*temperature'}) [False, True] >>> g = f.subset(prop={'standard_name': '.*temperature'}, coord={'longitude': 0}) >>> g [] All of these keywords may be used with the :func:`~cf.read` function to select on input: >>> f = cf.read('file*.nc', prop={'standard_name': '.*temperature'}, coord={'longitude': 0}) Aggregation ----------- Fields are aggregated into as few multidimensional fields as possible with the :func:`~cf.aggregate` function: >>> f [, , , , , , , ] >>> cf.aggregate(f) >>> f [, ] By default, the the fields return from the :func:`~cf.read` function have been aggregated: >>> f = cf.read('file*.nc') >>> len(f) 1 >>> f = cf.read('file*.nc', aggregate=False) >>> len(f) 12 Aggregation implements the `CF aggregation rules `_. Assignment ---------- The preferred way of changing elements of a field's data array in place is with the :meth:`~cf.Field.setitem` method [2]_. Assignment uses :ref:`LAMA functionality `, so it is possible to assign to subspaces which are larger than the available memory. >>> print f Data : air_temperature(time, latitude, longitude) Cell methods : time: mean Dimensions : time(12) = [15, ..., 345] days since 1860-1-1 : latitude(73) = [-90, ..., 90] degrees_north : longitude(96) = [0, ..., 356.25] degrees_east : height(1) = [2] m Auxiliary coords: Arithmetic and comparison ------------------------- Arithmetic and comparison operations on a field are defined as element-wise operations on the field's data array, and return a field as the result: * When using a field in unary or binary arithmetic operations (such as ``abs()``, ``+`` or ``**``) a new, independent field is created with a modified data array. * When using a field in augmented arithmetic operations (such as ``-=``), the field's data array is modified in place. * When using a field in comparison operations (such as ``<`` or ``!=``) a new, independent field is created with a boolean data array. A field's data array is modified in a very similar way to how a numpy array would be modified in the same operation, i.e. :ref:`broadcasting ` ensures that the operands are compatible and the data array is modified element-wise. Broadcasting is metadata-aware and will automatically account for arbitrary configurations, such as dimension order, but will not allow incompatible metadata to be combined, such as adding a field of height to one of temperature. The :ref:`resulting field's metadata ` will be very similar to that of the operands which are also fields. Differences arise when the existing metadata can not correctly describe the newly created field. For example, when dividing a field with units of *metres* by one with units of *seconds*, the resulting field will have units of *metres/second*. Arithmetic and comparison utilizes :ref:`LAMA functionality ` so data arrays larger than the available physical memory may be operated on. .. _broadcasting: Broadcasting ^^^^^^^^^^^^ The term broadcasting describes how data arrays of the operands with different shapes are treated during arithmetic and comparison operations. Subject to certain constraints, the smaller array is "broadcast" across the larger array so that they have compatible shapes. The general broadcasting rules are similar to the :mod:`broadcasting rules implemented in numpy `, the only difference being when both operands are fields, in which case the fields are temporarily conformed so that: * Dimensions are aligned according to the coordinates' metadata to ensure that matching dimensions are broadcast against each other. * Common dimensions have matching units. * Common dimensions have matching axis directions. This restructuring of the field ensures that the matching dimensions are broadcast against each other. Broadcasting is done without making needless copies of data and so is usually very efficient. What a field may be combined with ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A field may be combined or compared with the following objects: +---------------+----------------------------------------------------+ | Object | Description | +===============+====================================================+ |:obj:`int`, | The field's data array is combined with | |:obj:`long`, | the python scalar | |:obj:`float` | | +---------------+----------------------------------------------------+ |:class:`.Data` | The field's data array | |with size 1 | is combined with the :class:`.Data` object's scalar| | | value, taking into account: | | | | | | * Different but equivalent units | +---------------+----------------------------------------------------+ |:class:`.Field`| The two field's must satisfy the field combination | | | rules. The fields' data arrays and spaces are | | | combined taking into account: | | | | | | * Identities of dimensions | | | * Different but equivalent units | | | * Different dimension orders | | | * Different dimension directions | +---------------+----------------------------------------------------+ A field may appear on the left or right hand side of an operator, but note the following warning: .. warning:: Combining a numpy array on the *left* with a field on the right does work, but will give generally unintended results -- namely a numpy array of fields. .. _resulting_metadata: Resulting metadata ^^^^^^^^^^^^^^^^^^ When creating any new field, the field's `history` CF property is updated to record the operation. The fields existing name-like CF properties may also need to be changed: >>> f.standard_name 'air_temperature' >>> f += 2 >>> f.standard_name 'air_temperature' >>> f.history 'air_temperature+2' >>> f.standard_name 'air_temperature' >>> f **= 2 >>> f.standard_name AttributeError: 'Field' object has no attribute 'standard_name' >>> f.history 'air_temperature**2' >>> f.standard_name, g.standard_name ('air_temperature', 'eastward_wind') >>> h = f * g >>> h.standard_name AttributeError: 'Field' object has no attribute 'standard_name' >>> h.long_name 'air_temperature*eastward_wind' >>> h.history 'air_temperature*eastward_wind' When creating a new field which has different physical properties to the input field(s) the units will also need to be changed: >>> f.units 'K' >>> f += 2 >>> f.units 'K' >>> f.units 'K' >>> f **= 2 >>> f.units 'K2' >>> f.units, g.units ('m', 's') >>> h = f / g >>> h.units 'm/s' When creating a new field which has a different space to the input fields, the new space will in general contain the superset of dimensions from the two input fields, but may not have some of either input field's auxiliary coordinates or size 1 dimension coordinates. Refer to the field combination rules for details. Overloaded operators ^^^^^^^^^^^^^^^^^^^^ A field defines the following overloaded operators for arithmetic and comparison. .. admonition:: Rich comparison operators ======== ======== Operator Method ======== ======== ``<`` __lt__() ``<=`` __le__() ``==`` __eq__() ``!=`` __ne__() ``>`` __gt__() ``>=`` __ge__() ======== ======== .. admonition:: Binary arithmetic operators ================= ============== =============== ============ Operator Methods Notes ================= ============== =============== ============ ``+`` __add__() __radd__() ``-`` __sub__() __rsub__() ``*`` __mul__() __rmul__() ``/`` __div__() __rdiv__() ``/`` __truediv__() __rtruediv__() ``//`` __floordiv__() __rfloordiv__() ``%`` __mod__() __rmod__() ``divmod()`` __divmod__() __rdivmod__() ``**``, ``pow()`` __pow__() __rpow__() ``&`` __and__() __rand__() Bit-wise AND ``^`` __xor__() __rxor__() Bit-wise XOR ``|`` __or__() __ror__() Bit-wise OR ================= ============== =============== ============ .. admonition:: Augmented arithmetic operators ======== ================ ============ Operator Method Notes ======== ================ ============ ``+=`` __iadd__() ``-=`` __isub__() ``*=`` __imul__() ``/`` __idiv__() ``/`` __itruediv__() ``//=`` __ifloordiv__() ``%=`` __imod__() ``**=`` __ipow__() ``&=`` __iand__() Bit-wise AND ``^=`` __ixor__() Bit-wise XOR ``|=`` __ior__() Bit-wise OR ======== ================ ============ .. admonition:: Unary arithmetic operators ========= ============ Operator Method ========= ============ ``-`` __neg__() ``+`` __pos__() ``abs()`` __abs__() ``~`` __invert__() ========= ============ Manipulation routines --------------------- A field has attributes and methods which return information about its data array or manipulate the data array in some manner. Many of these behave similarly to their numpy counterparts with the same name but, where appropriate, return :class:`.Field` objects rather than numpy arrays. .. admonition:: Attributes .. tabularcolumns:: |l|l|l| ============================== =============================== =========================== Field attribute Description Numpy counterpart ============================== =============================== =========================== :meth:`~cf.Field.size` Number of elements in the data :attr:`numpy.ndarray.size` array :meth:`~cf.Field.shape` Tuple of the data array's :attr:`numpy.ndarray.shape` dimension sizes :meth:`~cf.Field.ndim` Number of dimensions in the :attr:`numpy.ndarray.ndim` data array :meth:`~cf.Field.dtype` Numpy data-type of the data :attr:`numpy.ndarray.dtype` array ============================== =============================== =========================== .. admonition:: Methods .. tabularcolumns:: |l|l|l| ============================== ============================== ============================ Field method Description Numpy counterpart ============================== ============================== ============================ :meth:`~cf.Field.clip` Clip (limit) the values in the :func:`numpy.ma.clip` data array :meth:`~cf.Field.cos` Trigonometric cosine of :func:`numpy.ma.cos` the data array :meth:`~cf.Field.expand_dims` Expand the shape of the data :func:`numpy.ma.expand_dims` array :meth:`~cf.Field.flip` Flip dimensions of the field :meth:`~cf.Field.sin` Trigonometric sine of :func:`numpy.ma.sin` the data array :meth:`~cf.Field.squeeze` Remove size 1 dimensions from :func:`numpy.ma.squeeze` the field's data array :meth:`~cf.Field.transpose` Permute the dimensions of the :func:`numpy.ma.transpose` data array :meth:`~cf.Field.unsqueeze` Insert size 1 dimensions from the field's space into its data array ============================== ============================== ============================ Manipulating other variables ---------------------------- Subspacing, assignment, arithmetic and comparison operations on other :class:`.Variable` types (such as :class:`.Coordinate`, :class:`.CoordinateBounds`, :class:`.CellMeasure`) are very similar to those for fields. In general, different dimension identities, different dimension orders and different dimension directions are not considered, since these objects do not contain the coordinate system required to define these properties (unlike a field). Coordinates ^^^^^^^^^^^ Coordinates are a special case as they may contain a data array for their coordinate bounds which needs to be treated consistently with the main coordinate array: >>> type(c) >>> type(c.bounds) >>> c.shape (12,) >>> c.bounds.shape (12, 2) >>> d = c.subspace[0:6] >>> d.shape (6,) >>> d.bounds.shape (6, 2) .. warning:: If the coordinate bounds are operated on directly, consistency with the parent coordinate may be broken. ---- .. rubric:: Footnotes .. [1] Arrays that may have missing or invalid entries .. [2] See the :meth:`~cf.Field.setitem` documentation for how to assign in other ways.