tasks.meta module¶

meta.py

functions for persisting metadata about tables loaded via ETL

class tasks.meta.CurrentSession¶

Bases: object

begin()¶

commit()¶

get()¶

rollback()¶

class tasks.meta.Geometry¶

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)¶

column_expression(col)¶

get_col_spec(*arg, **kw)¶

class tasks.meta.Linestring¶

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)¶

column_expression(col)¶

get_col_spec(*arg, **kw)¶

class tasks.meta.OBSColumn(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Describes the characteristics of data in a column, which can exist in multiple physical tables.

These should only be instantiated to generate the return value of ColumnsTask.columns(). Any other usage could have unexpected consequences.

id¶: The unique identifier for this column. Should be qualified by the module name of the class that created it. Is automatically generated by ColumnsTask if left unspecified, and automatically qualified by module name either way.

type¶: The type of this column – for example, Text, Numeric, Geometry, etc. This is used to generate a schema for any table using this column.

name¶: The human-readable name of this column. This is used in the catalog and API.

description¶: The human-readable description of this column. THis is used in the catalog.

weight¶: A numeric weight for this column. Higher weights are favored in certain rankings done by the API. Defaults to zero, which hides the column from catalog.

aggregate¶: How this column can be aggregated. For example, populations can be summed, so a population column should be sum. Should be left blank if it is not to aggregate.

tables¶: Iterable of all linked :class:`~.meta.OBSColumnTable`s, which could be traversed to find all tables with this column.

tags¶: Iterable of all linked :class:`~.meta.OBSColumnTag`s, which could be traversed to find all tags applying to this column.

targets¶: Dict of all related columns. Format is <OBSColumn>: <reltype>.

version¶: A version control number, used to determine whether the column and its metadata should be updated.

extra¶: Arbitrary additional information about this column stored as JSON.

aggregate

catalog_lonlat()¶: Return tuple (longitude, latitude) for the catalog for this measurement.

children()¶

denominators()¶: Return the OBSColumn denominator of this column.

description

extra

geom_timespans()¶: Return a dict of geom columns and timespans that this measure is available for.

has_catalog_image()¶: Returns True if this column has a pre-generated image for the catalog.

has_children()¶: Returns True if this column has children, False otherwise.

has_denominators()¶: Returns True if this column has no denominator, False otherwise.

id

index_type¶

is_cartographic()¶: Returns True if this column is a geometry that can be used for cartography.

is_geomref()¶: Returns True if the column is a geomref, else Null

is_interpolation()¶: Returns True if this column is a geometry that can be used for interpolation.

license_tags()¶: Return license tags.

name

source_tags()¶: Return source tags.

sources¶

summable()¶: Returns True if we can sum this column and calculate for arbitrary areas.

tables

tags: A descriptor that presents a read/write view of an object attribute.

targets: A descriptor that presents a read/write view of an object attribute.

type

unit()¶: Return the OBSTag unit of this column.

version

weight

class tasks.meta.OBSColumnTable(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Glues together OBSColumn and OBSTable. If this object exists, the related column should exist in the related table, and can be selected with colname.

Unique along both (column_id, table_id) and (table_id, colname).

These should never be created manually. Their creation and removal is handled automatically as part of targets.TableTarget.update_or_create_metadata().

column_id¶: ID of the linked OBSColumn.

table_id¶: ID of the linked OBSTable.

colname¶: Column name for selecting this column in a table.

column¶: The linked OBSColumn.

table¶: The linked OBSTable.

extra¶: Extra JSON information about the data. This could include statistics like max, min, average, etc.

colname

colname_constraint = UniqueConstraint(Column('table_id', Text(), ForeignKey('obs_table.id'), table=<obs_column_table>, primary_key=True, nullable=False), Column('colname', Text(), table=<obs_column_table>, nullable=False))¶

column

column_id

extra

table

table_id

class tasks.meta.OBSColumnTableTile(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

This table contains column/table summary data using raster tiles.

column_id¶

constraint = ForeignKeyConstraint(<sqlalchemy.sql.base.ColumnCollection object>, None, ondelete='cascade')¶

table_id¶

tile¶

tile_id¶

class tasks.meta.OBSColumnTableTileSimple(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

This table contains column/table summary data using raster tiles.

column_id¶

cvxhull_restraint = Index('obs_column_table_simple_chtile', <sqlalchemy.sql.functions.Function at 0x7f9d186ebe90; ST_ConvexHull>)¶

table_id¶

tct_constraint = UniqueConstraint(Column('table_id', Text(), table=<obs_column_table_tile_simple>, primary_key=True, nullable=False), Column('column_id', Text(), table=<obs_column_table_tile_simple>, primary_key=True, nullable=False), Column('tile_id', Integer(), table=<obs_column_table_tile_simple>, primary_key=True, nullable=False))¶

tile¶

tile_id¶

class tasks.meta.OBSColumnTag(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Glues together OBSColumn and OBSTag. If this object exists, the related column should be tagged with the related tag.

Unique along (column_id, tag_id).

These should never be created manually. Their creation and removal is handled automatically as part of targets.TableTarget.update_or_create_metadata().

column_id¶: ID of the linked OBSColumn.

tag_id¶: ID of the linked OBSTag.

column¶: The linked OBSColumn.

tag¶: The linked OBSTag.

column

column_id

tag

tag_id

class tasks.meta.OBSColumnToColumn(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Relates one column to another. For example, a Text column may contain identifiers that are unique to geometries in another Geometry column. In that case, an OBSColumnToColumn object of reltype GEOM_REF should indicate the relationship, and make relational joins possible between tables that have both columns with those that only have one.

These should never be created manually. Their creation should be handled automatically from specifying OBSColumn.targets.

These are unique on (source_id, target_id).

source_id¶: ID of the linked source OBSColumn.

target_id¶: ID of the linked target OBSColumn.

reltype¶: required text specifying the relation type. Examples are GEOM_REF and ~.meta.DENOMINATOR.

source¶: The linked source OBSColumn.

target¶: The linked target OBSColumn.

reltype

source

source_id

target

target_id

class tasks.meta.OBSDumpVersion(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

This table contains one entry for every generated dump. Automatically updated by Dump.

dump_id¶

class tasks.meta.OBSTable(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Describes a physical table in our database.

These should never be instantiated manually. They are automatically created by output(). The unique key is :attr:~.meta.OBSTable.id:.

id¶: The unique identifier for this table. Is always qualified by the module name of the class that created it.

columns¶: An iterable of all the OBSColumnTable instances contained in this table.

tablename¶: The automatically generated name of this table, which can be used directly in select statements. Of the format obs_<hash>.

timespan¶: An OBSTimespan instance containing information about the timespan this table applies to. Obtained from timespan().

the_geom¶: A simple geometry approximating the boundaries of the data in this table.

description¶: A description of the table. Not used.

version¶: A version control number, used to determine whether the table and its metadata should be updated.

columns

description

geom_column()¶

Return the column geometry column for this table, if it has one.

Returns None if there is none.

geomref_column()¶

Return the geomref column for this table, if it has one.

Returns None if there is none.

id

table_timespan¶

tablename

targets¶: A descriptor that presents a read/write view of an object attribute.

the_geom

timespan

version

class tasks.meta.OBSTableToTable(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Relates one table to another.

These should never be created manually. Their creation should be handled automatically from specifying OBSTable.targets.

These are unique on (source_id, target_id).

source_id¶: ID of the linked source OBSTable.

target_id¶: ID of the linked target OBSTable.

reltype¶: required text specifying the relation type.

source¶: The linked source OBSTable.

target¶: The linked target OBSTable.

reltype

source

source_id

target

target_id

class tasks.meta.OBSTag(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base

Tags permit arbitrary groupings of columns.

They should only be created as part of a tags() implementation.

id¶: The unique identifier for this table. Is always qualified by the module name of the TagsTask that created it, and should never be specified manually.

name¶: The name of this tag. This is exposed in the API and user interfaces.

type¶: The type of this tag. This is used to provide more information about what the tag means. Examples are section, subsection, license, unit, although any arbitrary type can be defined.

description¶: Description of this tag. This may be exposed in the API and catalog.

columns¶: An iterable of all the :class:`~.meta.OBSColumn`s tagged with this tag.

version¶: A version control number, used to determine whether the tag and its metadata should be updated.

columns: A descriptor that presents a read/write view of an object attribute.

description

id

name

type

version

class tasks.meta.OBSTimespan(**kwargs)¶

Bases: tasks.meta.UniqueMixin, sqlalchemy.ext.declarative.api.Base

Describes a timespan table in our database.

alias¶

description¶

id¶

name¶

timespan¶

classmethod unique_filter(query, id)¶

classmethod unique_hash(id)¶

weight¶

class tasks.meta.Point¶

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)¶

column_expression(col)¶

get_col_spec(*arg, **kw)¶

class tasks.meta.Raster¶

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)¶

column_expression(col)¶

get_col_spec(*arg, **kw)¶

class tasks.meta.UniqueMixin¶

Bases: object

classmethod as_unique(session, *arg, **kw)¶

classmethod unique_filter(query, *arg, **kw)¶

classmethod unique_hash(*arg, **kw)¶

class tasks.meta.UpdatedMetaTarget¶

Bases: luigi.target.Target

exists()¶

tasks.meta.catalog_latlng(column_id)¶

tasks.meta.coltocoltargets_creator(coltarget_or_col, reltype)¶

tasks.meta.current_session()¶: Returns the session relevant to the currently operating Task, if any. Outside the context of a Task, this can still be used for manual session management.

tasks.meta.fromkeys(d, l)¶: Similar to the builtin dict fromkeys, except remove keys with a value of None

tasks.meta.get_engine(user=None, password=None, host=None, port=None, db=None, readonly=False)¶

tasks.meta.natural_sort_key(s, _nsre=<_sre.SRE_Pattern object>)¶

tasks.meta.session_commit(task)¶: commit the global session

tasks.meta.session_rollback(task, exception)¶: rollback the global session

tasks.meta.tabletotabletargets_creator(target, reltype)¶

tasks.meta.tag_creator(tagtarget)¶