SAMZA-1282: Spinning up more containers than number of tasks.
authorShanthoosh Venkataraman <svenkataraman@linkedin.com>
Tue, 25 Jul 2017 22:32:22 +0000 (15:32 -0700)
committernavina <navina@apache.org>
Tue, 25 Jul 2017 22:32:22 +0000 (15:32 -0700)
commit57758615b3f8713364ac1afecab4f5355f64d1d4
treecfe049cad7022432533b39346681bd65958062c1
parent35143b676f23f69324b442fd1061318a663538f0
SAMZA-1282: Spinning up more containers than number of tasks.

Changes

* Stop streamProcessor in onNewJobModelAvailable eventHandler(instead of onNewJobModelConfirmed
  eventHandler) when it's not part of the group and prevent it from joining the barrier.
* When numContainerIds > numTaskModels, generate JobModel by choosing lexicographically
  least `x` containerIds(where x = numTaskModels).
* Added unit and integration tests in appropriate classes to verify the expected behavior.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Boris Shkolnik <boryas@apache.org>, Navina Ramesh <navina@apache.org>

Closes #244 from shanthoosh/more_processor_than_tasks
samza-core/src/main/java/org/apache/samza/container/grouper/task/GroupByContainerIds.java
samza-core/src/main/java/org/apache/samza/processor/StreamProcessor.java
samza-core/src/main/java/org/apache/samza/zk/ZkJobCoordinator.java
samza-core/src/main/java/org/apache/samza/zk/ZkJobCoordinatorFactory.java
samza-core/src/test/java/org/apache/samza/container/grouper/task/TestGroupByContainerIds.java
samza-core/src/test/java/org/apache/samza/zk/TestZkJobCoordinator.java [new file with mode: 0644]
samza-test/src/test/java/org/apache/samza/test/processor/TestZkLocalApplicationRunner.java