Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. Beam Pipelines are defined using one of the provided SDKs and executed in one of the Beam’s supported runners (distributed processing back-ends) including Apache Flink, Apache Samza, Apache Spark, and Google Cloud Dataflow.

History

Apache Beam is one implementation of the Dataflow model paper. The Dataflow model is based on previous work on distributed processing abstractions at Google, in particular on FlumeJava and Millwheel.

Apache Beam

Google released an open SDK implementation of the Dataflow model in 2014 and an environment to execute Dataflows locally (non-distributed) as well as in the Google Cloud Platform service.

Apache Beam

Timeline

Apache Beam makes minor releases every 6 weeks.

Apache Beam

See also

List of Apache Software Foundation projects

References

Apache Software Foundation

Apache Software Foundation projects

Big data products

Cluster computing

Distributed stream processing

Google software

Hadoop

Java platform

Free software programmed in Java (programming language)