.. currentmodule:: cf .. default-role:: obj Field manipulation ================== Manipulating a field generally involves operating on its data array and making any necessary changes to the field's domain to make it consistent with the new array. Data array ---------- Conversion to a numpy array ^^^^^^^^^^^^^^^^^^^^^^^^^^^ A field's data array may be converted to either an independent numpy masked array or a numpy masked array view (`numpy.ndarray.view`) with its `~Field.varray` and `~Field.array` attributes respectively: >>> a = f.array >>> print a [[2 -- 4 -- 6]] >>> a[0, 0] = 999 >>> print a [[999 -- 4 -- 6]] >>> print f.array [[2 -- 4 -- 6]] >>> va = f.varray >>> va[0, 0] = 999 >>> print f.array [[999 -- 4 -- 6]] Note that changing the numpy array view in place will also change the field's data array in-place. .. note:: The numpy array created by the `~Field.varray` or `~Field.array` attributes forces all of the data to be read into memory at the same time, which may not be possible for very large arrays. Data mask ^^^^^^^^^ The mask may be set or editted in-place with the `~Field.setmask` method. Copying ------- A deep copy of a variable may be created with its `~Field.copy` method or equivalently with the :py:obj:`copy.deepcopy` function: >>> g = f.copy() >>> import copy >>> g = copy.deepcopy(f) Copying utilizes :ref:`LAMA copying functionality `. .. _Subspacing: Subspacing ---------- Subspacing a field means subspacing its data array and its domain in a consistent manner. A field may be subspaced via its `~Field.subspace` attribute. This attribute returns an object which may be **indexed** to select a subspace by dimension index values (``f.subspace[indices]``) or **called** to select a subspace by dimension coordinate array values (``f.subspace(**coordinate_values)``): >>> g = f.subspace[0, ...] >>> g = f.subspace(latitude=30, longitude=cf.wi(0, 90, 'degrees')) The result of subspacing a field is a new, independent field whose data array and, crucially, any data arrays within the field's metadata (such as coordinates, ancillary variables, transforms, *etc.*) are appropriate subspaces of their originals: >>> print f air_temperature field summary ----------------------------- Data : air_temperature(time(12), latitude(73), longitude(96)) K Cell methods : time: mean Dimensions : time(12) = [1860-01-16 12:00:00, ..., 1860-12-16 12:00:00] : latitude(73) = [-90, ..., 90] degrees_north : longitude(96) = [0, ..., 356.25] degrees_east : height(1) = [2] m >>> g = f.subspace[-1, :, 48::-1] >>> print g air_temperature field summary ----------------------------- Data : air_temperature(time(1), latitude(73), longitude(49)) K Cell methods : time: mean Dimensions : time(1) = [1860-12-16 12:00:00] : latitude(73) = [-90, ..., 90] degrees_north : longitude(49) = [180, ..., 0] degrees_east : height(1) = [2] m Subspacing utilizes :ref:`LAMA subspacing functionality `. .. _indexing: Indexing ^^^^^^^^ Subspacing by dimension indices uses an extended Python slicing syntax, which is similar to :ref:`numpy array indexing `: >>> f.shape (12, 73, 96) >>> f.subspace[...].shape (12, 73, 96) >>> f.subspace[slice(0, 12), :, 10:0:-2].shape (12, 73, 5) >>> lon = f.coord('longitude').array >>> f.subspace[..., lon<180] There are two important extensions to the numpy indexing functionality: * Size 1 dimensions are never removed. An integer index *i* takes the *i*-th element but does not reduce the rank of the output array by one: >>> f.shape (12, 73, 96) >>> f.subspace[0, ...].shape (1, 73, 96) >>> f.subspace[3, slice(10, 0, -2), 95].shape (1, 5, 1) * When advanced indexing is used on more than one dimension, the advanced indices work independently. When more than one dimension’s slice is a 1-d boolean sequence or 1-d sequence of integers, then these indices work independently along each dimension (similar to the way vector subscripts work in Fortran), rather than by their elements: >>> f.shape (12, 73, 96) >>> f.subspace[:, [0, 72], [5, 4, 3]].shape (12, 2, 3) Note that the indices of this example would raise an error when given to a numpy array. Coordinate values ^^^^^^^^^^^^^^^^^ Subspacing by values of 1-d coordinates allows a subspaced field to be defined via coordinate values of its domain. The benefits of subspacing in this fashion are: * The dimensions to be subspaced may identified by name. * The position in the data array of each dimension need not be known and the dimensions to be subspaced may be given in any order. * Dimensions for which no subspacing is required need not be specified. * Size 1 dimensions of the domain which are not spanned by the data array may be specified. Coordinate values are provided as keyword arguments to a call to the `~Field.subspace` attribute. Coordinates are identified by their `~Coordinate.identity` or their dimension's identifier in the field's domain. >>> f.subspace().shape (12, 73, 96) >>> f.subspace(latitude=0).shape (12, 1, 96) >>> f.subspace(latitude=cf.wi(-30, 30)).shape (12, 25, 96) >>> f.subspace(long=cf.ge(270, 'degrees_east'), lat=cf.set([0, 2.5, 10])).shape (12, 3, 24) >>> f.subspace(latitude=cf.lt(0, 'degrees_north')) (12, 36, 96) >>> f.subspace(latitude=[cf.lt(0, 'degrees_north'), 90]) (12, 37, 96) >>> import math >>> f.subspace(longitude=cf.lt(math.pi, 'radian'), height=2) (12, 73, 48) >>> f.subspace(height=cf.gt(3)) IndexError: No indices found for 'height' values gt 3 >>> f.subspace(dim2=3.75).shape (12, 1, 96) Note that if a comparison function (such as `cf.wi`) does not specify any units, then the units of the named coordinate are assumed. Selection --------- Fields may be tested for matching given conditions with the `~Field.match` method and selected by matching given conditions with the `~Field.select` method. Both methods share the same interface. Conditions may be given on: ===================== ============== ============================================== Field conditions Method keyword Example ===================== ============== ============================================== CF properties *prop* ``prop={'standard_name': '.*pressure.*'}`` Other attributes *attr* ``attr=dict(ncvar='tas')`` Coordinate values *coord* ``coord={'latitude': 0}`` Coordinate cell sizes *cellsize* ``cellsize={'time': cf.wi(28, 31, 'days')}`` ===================== ============== ============================================== For example: >>> f [, ] >>> f.match(prop={'standard_name': '.*temperature'}) [False, True] >>> g = f.select(prop={'standard_name': '.*temperature'}, coord={'longitude': 0}) >>> g [] Any of these keywords may also be used with `cf.read` to select on input: >>> f = cf.read('file*.nc', prop={'standard_name': '.*temperature'}, ... coord={'longitude': 0}) Selection may also be applied to a field, rather than a field list. In this case, the `~Field.select` method returns the field itself if there is a match: >>> f >>> f.match(prop=dict(standard_name=cf.eq('air_temperature')) True >>> g = f.select(prop=dict(standard_name=cf.eq('air_temperature')) >>> g is f True Aggregation ----------- Fields are aggregated into as few multidimensional fields as possible with the `cf.aggregate` function, which implements the `CF aggregation rules `_. >>> f [, , , , , , , ] >>> cf.aggregate(f) >>> f [, ] By default, the fields returned by `cf.read` have been aggregated: >>> f = cf.read('file*.nc') >>> len(f) 1 >>> f = cf.read('file*.nc', aggregate=False) >>> len(f) 12 Aggregation implements the `CF aggregation rules `_. Assignment ---------- The preferred way of changing elements of a field's data array in place is with the field's `~Field.setitem` method [#f2]_. Assignment uses :ref:`LAMA functionality `, so it is possible to assign to subspaces which are larger than the available memory. Array elements may be set from a field or logically scalar object, using the same :ref:`broadcasting rules ` as for field arithmetic and comparison operations. Set all values to 273.15: >>> f.setitem(273.15) Set all values to between 210 and 270 degrees longitude and -5 and 5 degrees latitude to 273.15: >>> f.setitem(273.15, dict(longitude=cf.wi(210, 270, 'degrees_east'), latitude=cf.wi(-5, 5, 'degrees_north')) Set all values less the 10 Celcius to 10 Celcius: >>> f.setitem(cf.Data(10, 'K @ 273.15'), condition=cf.lt(10, 'K @ 273.15')) Set all unmasked values to 300: >>> f.setitem(300, masked=False) .. _Arithmetic-and-comparison: Arithmetic and comparison ------------------------- Arithmetic and comparison operations are defined on a field as element-wise array operations which yield a new `cf.Field` object or, for augmented arithmetic assignments, modify the field's data array in-place. **Comparison operators** .. autosummary:: :nosignatures: :toctree: generated/ :template: method.rst ~cf.Field.__lt__ ~cf.Field.__le__ ~cf.Field.__eq__ ~cf.Field.__ne__ ~cf.Field.__gt__ ~cf.Field.__ge__ **Binary arithmetic operators** .. autosummary:: :nosignatures: :toctree: generated/ :template: method.rst ~cf.Field.__add__ ~cf.Field.__sub__ ~cf.Field.__mul__ ~cf.Field.__div__ ~cf.Field.__truediv__ ~cf.Field.__floordiv__ ~cf.Field.__pow__ ~cf.Field.__and__ ~cf.Field.__or__ ~cf.Field.__xor__ **Binary arithmetic operators with reflected (swapped) operands** .. autosummary:: :nosignatures: :toctree: generated/ :template: method.rst ~cf.Field.__radd__ ~cf.Field.__rsub__ ~cf.Field.__rmul__ ~cf.Field.__rdiv__ ~cf.Field.__rtruediv__ ~cf.Field.__rfloordiv__ ~cf.Field.__rpow__ ~cf.Field.__rand__ ~cf.Field.__ror__ ~cf.Field.__rxor__ **Augmented arithmetic assignments** .. autosummary:: :nosignatures: :toctree: generated/ :template: method.rst ~cf.Field.__iadd__ ~cf.Field.__isub__ ~cf.Field.__imul__ ~cf.Field.__idiv__ ~cf.Field.__itruediv__ ~cf.Field.__ifloordiv__ ~cf.Field.__ipow__ ~cf.Field.__iand__ ~cf.Field.__ior__ ~cf.Field.__ixor__ **Unary arithmetic operators** .. autosummary:: :nosignatures: :toctree: generated/ :template: method.rst ~cf.Field.__neg__ ~cf.Field.__pos__ ~cf.Field.__abs__ ~cf.Field.__invert__ A field's data array is modified in a very similar way to how a numpy array would be modified in the same operation, i.e. :ref:`broadcasting ` ensures that the operands are compatible and the data array is modified element-wise. Broadcasting is metadata-aware and will automatically account for arbitrary configurations, such as dimension order, but will not allow incompatible metadata to be combined, such as adding a field of height to one of temperature. The :ref:`resulting field's metadata ` will be very similar to that of the operands which are also fields. Differences arise when the existing metadata can not correctly describe the newly created field. For example, when dividing a field with units of *metres* by one with units of *seconds*, the resulting field will have units of *metres per second*. Arithmetic and comparison utilizes :ref:`LAMA functionality ` so data arrays larger than the available physical memory may be operated on. .. _broadcasting: Broadcasting ^^^^^^^^^^^^ The term broadcasting describes how data arrays of the operands with different shapes are treated during arithmetic, comparison and assignment operations. Subject to certain constraints, the smaller array is "broadcast" across the larger array so that they have compatible shapes. The general broadcasting rules are similar to the :mod:`broadcasting rules implemented in numpy `, the only difference occurring when both operands are fields, in which case the fields are temporarily conformed so that: * Dimensions are aligned according to their coordinates' metadata to ensure that matching dimensions are broadcast against each other. * Common dimensions have matching units. * Common dimensions have matching axis directions. This restructuring of the field ensures that the matching dimensions are broadcast against each other. Broadcasting is done without making needless copies of data and so is usually very efficient. Valid operands ^^^^^^^^^^^^^^ A field may be combined or compared with the following objects: +----------------+----------------------------------------------------+ | Object | Description | +================+====================================================+ |:py:obj:`int`, | The field's data array is combined with | |:py:obj:`long`, | the python scalar | |:py:obj:`float` | | +----------------+----------------------------------------------------+ |`Data` | The field's data array | |with size 1 | is combined with the `cf.Data` object's scalar | | | value, taking into account: | | | | | | * Different but equivalent units | +----------------+----------------------------------------------------+ |`Field` | The two field's must satisfy the field combination | | | rules. The fields' data arrays and domains are | | | combined taking into account: | | | | | | * Dimension identities | | | * Array units | | | * Dimension orders | | | * Dimension directions | | | * Missing data values | +----------------+----------------------------------------------------+ A field may appear on the left or right hand side of an operator. .. warning:: Combining a numpy array on the *left* with a field on the right does work, but will give generally unintended results -- namely a numpy array of fields. .. _resulting_metadata: Resulting metadata ^^^^^^^^^^^^^^^^^^ When creating a new field which has different physical properties to the input field(s) the units will also need to be changed: >>> f.units 'K' >>> f += 2 >>> f.units 'K' >>> f.units 'K' >>> f **= 2 >>> f.units 'K2' >>> f.units, g.units ('m', 's') >>> h = f / g >>> h.units 'm s-1' When creating a new field which has a different domain to the input fields, the new domain will in general contain the superset of dimensions from the two input fields, but may not have some of either input field's auxiliary coordinates or size 1 dimension coordinates. Refer to the field combination rules for details. Data manipulation methods -------------------------- A field has methods which manipulate the its data array. Many of these behave similarly to their numpy counterparts with the same name but always change the field's data array in-place. New fields with the same changes may be created with equivalently named :ref:`module functions `. .. tabularcolumns:: |l|l|l|l| ==================== ============================== ====================== ================ Field method Description Numpy counterpart Function ==================== ============================== ====================== ================ `~Field.clip` Clip (limit) the values in the `numpy.ma.clip` `cf.clip` data array `~Field.cos` Trigonometric cosine of `numpy.ma.cos` `cf.cos` the data array `~Field.expand_dims` Expand the shape of the data `numpy.ma.expand_dims` `cf.expand_dims` array `~Field.flip` Flip dimensions of the field .. `cf.flip` `~Field.sin` Trigonometric sine of `numpy.ma.sin` `cf.sin` the data array `~Field.squeeze` Remove size 1 dimensions from `numpy.ma.squeeze` `cf.squeeze` the field's data array `~Field.transpose` Permute the dimensions of the `numpy.ma.transpose` `cf.transpose` data array `~Field.unsqueeze` Insert size 1 dimensions from .. `cf.unsqueeze` the field's domain into its data array ==================== ============================== ====================== ================ Manipulating other variables ---------------------------- A field is a subclass of `cf.Variable`, and that class and other subclasses of `cf.Variable` share generally similar behaviours and methods: ===================== =============================================== Class Description ===================== =============================================== `cf.CellMeasure` A CF cell measure construct containing information that is needed about the size, shape or location of the field's cells. `cf.Coordinate` A CF dimension or auxiliary coordinate construct. `cf.CoordinateBounds` A CF coordinate's bounds object containing cell boundaries or intervals of climatological time. `cf.Variable` Base class for storing a data array with metadata. ===================== =============================================== In general, different dimension identities, different dimension orders and different dimension directions are not considered, since these objects do not contain a coordinate system required to define these properties (unlike a field). Coordinates ^^^^^^^^^^^ Coordinates are a special case as they may contain a data array for their coordinate bounds which needs to be treated consistently with the main coordinate array. If a coordinate has bounds then all coordinate methods also operate on the bounds in a consistent manner: >>> c >>> c.bounds >>> d = c.subspace[0:10] >>> d.shape (10, 96) >>> d.bounds.shape (10, 96, 4) >>> d.transpose([1, 0]) >>> d.shape (96, 10) >>> d.bounds.shape (96, 10, 4) .. note:: If the coordinate bounds are operated on independently, care should be taken not to break consistency with the parent coordinate. ---- .. rubric:: Footnotes .. [#f2] Other methods are available. See `cf.Field.setitem` for details.