tasks package

Subpackages

Submodules

tasks.base_tasks module

tasks.carto module

tasks.meta module

meta.py

functions for persisting metadata about tables loaded via ETL

class tasks.meta.CurrentSession

Bases: object

begin()
commit()
get()
rollback()
class tasks.meta.Geometry

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)
column_expression(col)
get_col_spec(*arg, **kw)
class tasks.meta.Linestring

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)
column_expression(col)
get_col_spec(*arg, **kw)
class tasks.meta.OBSColumn(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Describes the characteristics of data in a column, which can exist in multiple physical tables.

These should only be instantiated to generate the return value of ColumnsTask.columns(). Any other usage could have unexpected consequences.

id

The unique identifier for this column. Should be qualified by the module name of the class that created it. Is automatically generated by ColumnsTask if left unspecified, and automatically qualified by module name either way.

type

The type of this column – for example, Text, Numeric, Geometry, etc. This is used to generate a schema for any table using this column.

name

The human-readable name of this column. This is used in the catalog and API.

description

The human-readable description of this column. THis is used in the catalog.

weight

A numeric weight for this column. Higher weights are favored in certain rankings done by the API. Defaults to zero, which hides the column from catalog.

aggregate

How this column can be aggregated. For example, populations can be summed, so a population column should be sum. Should be left blank if it is not to aggregate.

tables

Iterable of all linked :class:`~.meta.OBSColumnTable`s, which could be traversed to find all tables with this column.

tags

Iterable of all linked :class:`~.meta.OBSColumnTag`s, which could be traversed to find all tags applying to this column.

targets

Dict of all related columns. Format is <OBSColumn>: <reltype>.

version

A version control number, used to determine whether the column and its metadata should be updated.

extra

Arbitrary additional information about this column stored as JSON.

aggregate
catalog_lonlat()

Return tuple (longitude, latitude) for the catalog for this measurement.

children()
denominators()

Return the OBSColumn denominator of this column.

description
extra
geom_timespans()

Return a dict of geom columns and timespans that this measure is available for.

has_catalog_image()

Returns True if this column has a pre-generated image for the catalog.

has_children()

Returns True if this column has children, False otherwise.

has_denominators()

Returns True if this column has no denominator, False otherwise.

id
index_type
is_cartographic()

Returns True if this column is a geometry that can be used for cartography.

is_geomref()

Returns True if the column is a geomref, else Null

is_interpolation()

Returns True if this column is a geometry that can be used for interpolation.

license_tags()

Return license tags.

name
source_tags()

Return source tags.

sources
summable()

Returns True if we can sum this column and calculate for arbitrary areas.

tables
tags

A descriptor that presents a read/write view of an object attribute.

targets

A descriptor that presents a read/write view of an object attribute.

type
unit()

Return the OBSTag unit of this column.

version
weight
class tasks.meta.OBSColumnTable(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Glues together OBSColumn and OBSTable. If this object exists, the related column should exist in the related table, and can be selected with colname.

Unique along both (column_id, table_id) and (table_id, colname).

These should never be created manually. Their creation and removal is handled automatically as part of targets.TableTarget.update_or_create_metadata().

column_id

ID of the linked OBSColumn.

table_id

ID of the linked OBSTable.

colname

Column name for selecting this column in a table.

column

The linked OBSColumn.

table

The linked OBSTable.

extra

Extra JSON information about the data. This could include statistics like max, min, average, etc.

colname
colname_constraint = UniqueConstraint(Column('table_id', Text(), ForeignKey('obs_table.id'), table=<obs_column_table>, primary_key=True, nullable=False), Column('colname', Text(), table=<obs_column_table>, nullable=False))
column
column_id
extra
table
table_id
class tasks.meta.OBSColumnTableTile(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

This table contains column/table summary data using raster tiles.

column_id
constraint = ForeignKeyConstraint(<sqlalchemy.sql.base.ColumnCollection object>, None, ondelete='cascade')
table_id
tile
tile_id
class tasks.meta.OBSColumnTableTileSimple(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

This table contains column/table summary data using raster tiles.

column_id
cvxhull_restraint = Index('obs_column_table_simple_chtile', <sqlalchemy.sql.functions.Function at 0x7f9d186ebe90; ST_ConvexHull>)
table_id
tct_constraint = UniqueConstraint(Column('table_id', Text(), table=<obs_column_table_tile_simple>, primary_key=True, nullable=False), Column('column_id', Text(), table=<obs_column_table_tile_simple>, primary_key=True, nullable=False), Column('tile_id', Integer(), table=<obs_column_table_tile_simple>, primary_key=True, nullable=False))
tile
tile_id
class tasks.meta.OBSColumnTag(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Glues together OBSColumn and OBSTag. If this object exists, the related column should be tagged with the related tag.

Unique along (column_id, tag_id).

These should never be created manually. Their creation and removal is handled automatically as part of targets.TableTarget.update_or_create_metadata().

column_id

ID of the linked OBSColumn.

tag_id

ID of the linked OBSTag.

column

The linked OBSColumn.

tag

The linked OBSTag.

column
column_id
tag
tag_id
class tasks.meta.OBSColumnToColumn(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Relates one column to another. For example, a Text column may contain identifiers that are unique to geometries in another Geometry column. In that case, an OBSColumnToColumn object of reltype GEOM_REF should indicate the relationship, and make relational joins possible between tables that have both columns with those that only have one.

These should never be created manually. Their creation should be handled automatically from specifying OBSColumn.targets.

These are unique on (source_id, target_id).

source_id

ID of the linked source OBSColumn.

target_id

ID of the linked target OBSColumn.

reltype

required text specifying the relation type. Examples are GEOM_REF and ~.meta.DENOMINATOR.

source

The linked source OBSColumn.

target

The linked target OBSColumn.

reltype
source
source_id
target
target_id
class tasks.meta.OBSDumpVersion(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

This table contains one entry for every generated dump. Automatically updated by Dump.

dump_id
class tasks.meta.OBSTable(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Describes a physical table in our database.

These should never be instantiated manually. They are automatically created by output(). The unique key is :attr:~.meta.OBSTable.id:.

id

The unique identifier for this table. Is always qualified by the module name of the class that created it.

columns

An iterable of all the OBSColumnTable instances contained in this table.

tablename

The automatically generated name of this table, which can be used directly in select statements. Of the format obs_<hash>.

timespan

An OBSTimespan instance containing information about the timespan this table applies to. Obtained from timespan().

the_geom

A simple geometry approximating the boundaries of the data in this table.

description

A description of the table. Not used.

version

A version control number, used to determine whether the table and its metadata should be updated.

columns
description
geom_column()

Return the column geometry column for this table, if it has one.

Returns None if there is none.

geomref_column()

Return the geomref column for this table, if it has one.

Returns None if there is none.

id
table_timespan
tablename
targets

A descriptor that presents a read/write view of an object attribute.

the_geom
timespan
version
class tasks.meta.OBSTableToTable(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Relates one table to another.

These should never be created manually. Their creation should be handled automatically from specifying OBSTable.targets.

These are unique on (source_id, target_id).

source_id

ID of the linked source OBSTable.

target_id

ID of the linked target OBSTable.

reltype

required text specifying the relation type.

source

The linked source OBSTable.

target

The linked target OBSTable.

reltype
source
source_id
target
target_id
class tasks.meta.OBSTag(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Tags permit arbitrary groupings of columns.

They should only be created as part of a tags() implementation.

id

The unique identifier for this table. Is always qualified by the module name of the TagsTask that created it, and should never be specified manually.

name

The name of this tag. This is exposed in the API and user interfaces.

type

The type of this tag. This is used to provide more information about what the tag means. Examples are section, subsection, license, unit, although any arbitrary type can be defined.

description

Description of this tag. This may be exposed in the API and catalog.

columns

An iterable of all the :class:`~.meta.OBSColumn`s tagged with this tag.

version

A version control number, used to determine whether the tag and its metadata should be updated.

columns

A descriptor that presents a read/write view of an object attribute.

description
id
name
type
version
class tasks.meta.OBSTimespan(**kwargs)

Bases: tasks.meta.UniqueMixin, sqlalchemy.ext.declarative.api.Base

Describes a timespan table in our database.

alias
description
id
name
timespan
classmethod unique_filter(query, id)
classmethod unique_hash(id)
weight
class tasks.meta.Point

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)
column_expression(col)
get_col_spec(*arg, **kw)
class tasks.meta.Raster

Bases: sqlalchemy.sql.type_api.UserDefinedType

bind_expression(bindvalue)
column_expression(col)
get_col_spec(*arg, **kw)
class tasks.meta.UniqueMixin

Bases: object

classmethod as_unique(session, *arg, **kw)
classmethod unique_filter(query, *arg, **kw)
classmethod unique_hash(*arg, **kw)
class tasks.meta.UpdatedMetaTarget

Bases: luigi.target.Target

exists()
tasks.meta.catalog_latlng(column_id)
tasks.meta.coltocoltargets_creator(coltarget_or_col, reltype)
tasks.meta.current_session()

Returns the session relevant to the currently operating Task, if any. Outside the context of a Task, this can still be used for manual session management.

tasks.meta.fromkeys(d, l)

Similar to the builtin dict fromkeys, except remove keys with a value of None

tasks.meta.get_engine(user=None, password=None, host=None, port=None, db=None, readonly=False)
tasks.meta.natural_sort_key(s, _nsre=<_sre.SRE_Pattern object>)
tasks.meta.session_commit(task)

commit the global session

tasks.meta.session_rollback(task, exception)

rollback the global session

tasks.meta.tabletotabletargets_creator(target, reltype)
tasks.meta.tag_creator(tagtarget)

tasks.poi module

tasks.sched module

class tasks.sched.TestTask(*args, **kwargs)

Bases: luigi.task.Task

output()
run()
time = <luigi.parameter.Parameter object>

tasks.simplification module

tasks.simplify module

tasks.socrata module

Bigmetadata tasks

tasks to download and create metadata

class tasks.socrata.LoadSocrataCSV(*args, **kwargs)

Bases: luigi.task.Task

dataset = <luigi.parameter.Parameter object>
domain = <luigi.parameter.Parameter object>
requires()
target()

tasks.sphinx module

tasks.tags module

tasks.targets module

tasks.tools module

tasks.tools.remove_from_do(id)

tasks.util module

tasks.whosonfirst module

Module contents