Commands

Pipeline Commands

Commands used to manipulate pipelines.

Usage:

dataduct pipeline [-h] {create,validate,activate,visualize}

Arguments:

  • -h, --help: Show help message and exit.

Create, Validate, Activate

Usage:

dataduct pipeline {create,validate,activate}
    [-h] [-m MODE] [-f] [-t TIME_DELTA] [-b] [--frequency FREQUENCY]
    pipeline_definitions [pipeline_definitions ...]
  • create: Creates a pipeline locally.
  • validate: Creates a pipeline on Amazon DataPipeline and validates the pipeline.
  • activate: Creates and validates the pipeline, then activates the pipeline.

Arguments:

  • -h, --help: Show help message and exit.
  • -m MODE, --mode MODE: Mode or config variables to use. e.g. -m production
  • -f, --force: Destroy previous version of this pipeline, if they exist.
  • -t TIME_DELTA, --time_delta TIME_DELTA: Timedelta the pipeline by x time difference. e.g. -t "1 day"
  • -b, --backfill: Indicates that the timedelta supplied is for a backfill.
  • -frequency FREQUENCY: Frequency override for the pipeline.
  • pipeline_definitions: The YAML defintions of the pipeline.

Visualize

Visualizes the pipeline into a PNG file.

Usage:

dataduct pipeline visualize [-h] [-m MODE] [--activities_only]
                            filename pipeline_definitions
                            [pipeline_definitions ...]

Arguments:

  • -h --help: Show help message and exit.
  • -m MODE, --mode MODE: Mode or config variables to use. e.g. -m production
  • --activities_only: Visualize pipeline activities only.
  • filename: The filename for saving the visualization.
  • pipeline_definitions: The YAML defintions of the pipeline.

Database Commands

Commands used to generate SQL for various actions.

Create, Drop, Grant, Recreate

  • create: Generates SQL to create relations.
  • drop: Generates SQL to drop relations.
  • grant: Generates SQL to grant permissions.
  • recreate: Generates SQL to recreate relations.

Usage:

dataduct database {create,drop,grant,recreate}
    [-h] [-m MODE] table_definitions [table_definitions ...]

Arguments:

  • -h, --help: Show help message and exit.
  • -m MODE, --mode MODE: Mode or config variables to use. e.g. -m production
  • table_definitions: The SQL definitions of the relations.

Visualize

Creates an entity relationship diagram of the tables as a PNG file.

Usage:

dataduct database visualize [-h] [-m MODE]
                            filename table_definitions
                            [table_definitions ...]

Arguments:

  • -h, --help: Show help message and exit.
  • -m MODE, --mode MODE: Mode or config variables to use. e.g. -m production
  • filename: The filename for saving the visualization.
  • table_definitions: The SQL definitions of the tables.

Configuration Commands

Commands used to synchronize the config file from/to Amazon S3.

Sync To S3

Uploads the local config file to S3. Will automatically detect the location of the config file.See the config documentation for more information.

Usage:

dataduct config sync_to_s3 [-h] [-m MODE]

Arguments:

  • -h, --help: Show help message and exit.
  • -m MODE, --mode MODE: Mode or config variables to use. e.g. -m production

Sync From S3

Downloads the local config file from S3 and saves it to a file.

Usage:

dataduct config sync_from_s3 [-h] [-m MODE] filename

Arguments:

  • -h, --help: Show help message and exit.
  • -m MODE, --mode MODE: Mode or config variables to use. e.g. -m production
  • filename: The filename for saving the config file.