Apache NiFi

NiFi is a system to process and distribute data - a dataflow lifecycle automation tool that acquires and delivers data across enterprise systems in real time. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.


Based on concepts of Flow-Based Programming, NiFi features a user interface and fine-grained data provenance tools. The interface allows users to intuitively understand and interact with the data flow directly in the browser, promoting faster and safer iteration.


Flow-Based Programming is a programming paradigm that uses a “data factory” metaphor for designing and building applications. The data provenance features allow the user to see how an object flowed through
the system, replay it and visualize what happened to it before and after key stages, thereby simplifying data flows that are often large, complex directed graphs involving transformations, forks, joins...etc.

See: https://nifi.apache.org/