Dataduct

Dataduct - DataPipeline for humans

Dataduct is a wrapper built on top of AWS Datapipeline which makes it easy to create ETL jobs. All jobs can be specified as a series of steps in a YAML file and would automatically be translated into datapipeline with appropriate pipeline objects.

Features include:

• Visualizing pipeline activities
• Extracting data from different sources such as RDS, S3, local files
• Transforming data using EC2 and EMR
• Loading data into redshift
• Transforming data inside redshift
• QA data between the source system and warehouse

It is easy to create custom steps to augment the DSL as per the requirements. As well as running a backfill with the command line interface.

Contents:

• Introduction
• Installation
  • Installation using pip
  • Installing in the developer environment
  • Setup Autocomplete
• Commands
  • Pipeline Commands
  • Database Commands
  • Configuration Commands
  • SQL Shell Commands
• Config
  • Config Parameters
• Creating an ETL
  • Writing a Dataduct YAML File
  • Header Information
• Steps and Pipeline Objects
  • Definition of a Step
  • Common
  • Extract S3
  • Extract Local
  • Extract RDS
  • Extract Redshift
  • Transform
  • SQL Command
  • EMR Streaming
  • Load Redshift
  • Pipeline Dependencies
  • Create Load Redshift
  • Load, Reload, Primary Key Check
  • Upsert
  • Reload
  • Create Update SQL
  • Primary Key Check
  • Count Check
  • Column Check
• Input and Output Nodes
  • Input Nodes
  • Output Nodes
• Hooks
  • Available Hooks
  • Creating a hook
• Code documentation
  • dataduct.config package
  • dataduct.data_access package
  • dataduct.database package
  • dataduct.etl package
  • dataduct.pipeline package
  • dataduct.qa package
  • dataduct.s3 package
  • dataduct.steps package
  • dataduct.tests package
  • dataduct.utils package

Indices and tables

• Index
• Module Index
• Search Page