SAMZA-1067; Physical execution graph and planner for fluent API
authorXinyu Liu <xiliu@xiliu-ld.linkedin.biz>
Thu, 16 Mar 2017 01:27:01 +0000 (18:27 -0700)
committerXinyu Liu <xiliu@xiliu-ld.linkedin.biz>
Thu, 16 Mar 2017 01:27:01 +0000 (18:27 -0700)
commit1676eabc21eae2b0d5aa7d22890b1f7eb1cbb441
treecd7f3c205f9fcbf770780c8d8201f49ce71541f0
parent5f7b250278f2a1a91b75438a2e80cb70a8347040
SAMZA-1067; Physical execution graph and planner for fluent API

Initial commit for the physical graph and plan. Design is there: https://issues.apache.org/jira/secure/attachment/12856670/SAMZA-1067.0.pdf.

The commit includes:

1) Physical ProcessorGraph, where each processor represents a physical execution unit (e.g. a job in Yarn).
2) A planner does the following:
   - create ProcessorGraph from StreamGraph. For this phase, the graph only contains a single node (single stage);
   - figure out the partitions of intermediate topics
   - create the topics

Please note currently the planner is used in the remote runner for now. Further changes/refactoring/cleanup are expected to be integrated with local runner.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish Venkatraman <jvenkatraman@linkedin.com>

Closes #75 from xinyuiscool/SAMZA-1067
samza-core/src/main/java/org/apache/samza/execution/ExecutionPlanner.java [new file with mode: 0644]
samza-core/src/main/java/org/apache/samza/execution/ProcessorGraph.java [new file with mode: 0644]
samza-core/src/main/java/org/apache/samza/execution/ProcessorNode.java [new file with mode: 0644]
samza-core/src/main/java/org/apache/samza/execution/StreamEdge.java [new file with mode: 0644]
samza-core/src/main/java/org/apache/samza/operators/spec/OperatorSpecs.java
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala
samza-core/src/main/scala/org/apache/samza/util/Util.scala
samza-core/src/test/java/org/apache/samza/execution/TestExecutionPlanner.java [new file with mode: 0644]
samza-core/src/test/java/org/apache/samza/execution/TestProcessorGraph.java [new file with mode: 0644]