Centipede - Service-based Pipelines for Document Processing

Centipede is a de-centralized pipeline for processing documents. It consists of many stages, which may perform tasks such as downloading files, extracting texts, detecting language and encoding or indexing a document to a search index.

See: http://bit.ly/1zSCEAE