samza.git
5 months agoSAMZA-1329: Switch SamzaTaskProxy to use LocalityManager.
Shanthoosh Venkataraman [Thu, 8 Jun 2017 00:57:54 +0000 (17:57 -0700)] 
SAMZA-1329: Switch SamzaTaskProxy to use LocalityManager.

TasksResource(/tasks/) in samza-rest is relying on `JobModelManager.readJobModel` to get jobmodel from (CoordinatorStream, jobConfig). This created binary dependencies of systemstreams defined in task.inputs of job config into samza-rest. Managing those dependencies is hard and unnecessary.

This PR updates `TasksResource` to use `LocalityManager` to read task to container locality.
In the future, this api will read jobmodel from `SamzaMetadatasystem` store.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #219 from shanthoosh/fix-2

5 months agoSAMZA-1326: Revert to use getGlobalAppId for zk path
Boris Shkolnik [Wed, 7 Jun 2017 21:23:13 +0000 (14:23 -0700)] 
SAMZA-1326: Revert to use getGlobalAppId for zk path

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #216 from sborya/getGlobalId

5 months agoSAMZA-1221, SAMZA-1101: Internal cleanup for High-Level API implementation.
Prateek Maheshwari [Wed, 7 Jun 2017 19:42:11 +0000 (12:42 -0700)] 
SAMZA-1221, SAMZA-1101: Internal cleanup for High-Level API implementation.

SAMZA-1221: Separated the OperatorSpec and MessageStream DAGs so that they're now duals of each other. Users interact with and construct the MessageStream DAG; we create and use the OperatorSpec DAG internally.
Moved common OperatorSpec functionality (getId, getOpCode, getOpName etc.) to the OperatorSpec abstract base class.
Added a new JoinOperatorSpec and PartialJoinOperatorImpls which are created from JoinOperatorSpec in OperatorGraphImpl.
Added a new InputOperatorSpec and InputOperatorImpl (previously RootOperatorImpl). InputOperatorSpec is created when StreamGraph#getInputStream is called.
SAMZA-1101: Added a new OutputOperatorSpec and OutputOperatorImpl for partitionBy and sendTo. These are Separate from SinkOperatorSpec for and SinkOperatorImpl for sink. We don't need to create a sinkFn for partitionBy and sendTo anymore.
Updated most unit tests to use the new classes and avoid reflection.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish V <vjagadish1989@apache.org>

Closes #194 from prateekm/internal-cleanup

5 months agoWebsite updates for 0.13.0 release
Jacob Maes [Wed, 7 Jun 2017 17:50:20 +0000 (10:50 -0700)] 
Website updates for 0.13.0 release

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #215 from jmakes/samza-0.13.0-website

5 months agoUpdate docs for ClosableFunction
vjagadish1989 [Mon, 5 Jun 2017 17:32:20 +0000 (10:32 -0700)] 
Update docs for ClosableFunction

5 months agoSAMZA-1271; Guarantee predictable, deterministic order for operator initialization...
vjagadish1989 [Mon, 5 Jun 2017 17:27:23 +0000 (10:27 -0700)] 
SAMZA-1271; Guarantee predictable, deterministic order for operator initialization and finalization

Currently, the order of initialization of operators in the Samza high level API is not deterministic. The non-determinism arises from two primary causes:

- No fixed order of iteration for all subscribed `OperatorSpec`s for a given `MessageStream`
- No fixed order of iteration for all the `OperatorImpl`s in the `OperatorImplGraph`

We aim to provide the following 2 guarantees in this patch:
For any 2 operators A, B in the graph, if B consumes the output of A:
- A is initialized before B is initialized
- A is finalized only after B is finalized

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<pmaheshw@linkedin.com>

Closes #211 from vjagadish1989/deterministic_order

5 months agoSAMZA-1317: Changelog validation error for topics with period in the
Jacob Maes [Sat, 3 Jun 2017 03:42:14 +0000 (20:42 -0700)] 
SAMZA-1317: Changelog validation error for topics with period in the

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #213 from jmakes/samza-1317-migration-fix

5 months agoSAMZA-1317: Changelog topic configuration should accept streams.[streamId] configurations
Jacob Maes [Fri, 2 Jun 2017 21:53:31 +0000 (14:53 -0700)] 
SAMZA-1317: Changelog topic configuration should accept streams.[streamId] configurations

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #212 from jmakes/samza-1317-migration-fix

5 months agoUpdate RELEASE documentation to include steps related to integration tests
vjagadish1989 [Fri, 2 Jun 2017 02:13:48 +0000 (19:13 -0700)] 
Update RELEASE documentation to include steps related to integration tests

5 months agoMake run-app.sh executable
Xinyu Liu [Thu, 1 Jun 2017 21:55:17 +0000 (14:55 -0700)] 
Make run-app.sh executable

5 months agoSAMZA-1305. SAMZA-1306. Unit test - zk unavailable.
Boris Shkolnik [Thu, 1 Jun 2017 18:51:22 +0000 (11:51 -0700)] 
SAMZA-1305. SAMZA-1306. Unit test - zk unavailable.

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #198 from sborya/unitTestZkUnavailable2

5 months agoAdding gpg key for Jake Maes
Jacob Maes [Thu, 1 Jun 2017 15:38:36 +0000 (08:38 -0700)] 
Adding gpg key for Jake Maes

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #209 from jmakes/gpg-key

5 months agoSAMZA-1264; Make Operator Functions Closable
vjagadish1989 [Thu, 1 Jun 2017 05:20:14 +0000 (22:20 -0700)] 
SAMZA-1264; Make Operator Functions Closable

- Added `close()` to the lifecycle of `OperatorImpl`s, and all `Function`s.
- Added unit tests to verify calls to `close()`

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<pmaheshw@linkedin.com>

Closes #208 from vjagadish1989/operator_functions

5 months agoSAMZA-1234: Documentation for 0.13.0 release
Jacob Maes [Thu, 1 Jun 2017 00:23:30 +0000 (17:23 -0700)] 
SAMZA-1234: Documentation for 0.13.0 release

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>, Boris Shkolnik <boryas@apache.org>

Closes #204 from jmakes/samza-1236-tutorial-1

5 months agoSAMZA-1309; Debounce time config
Boris Shkolnik [Fri, 26 May 2017 23:31:45 +0000 (16:31 -0700)] 
SAMZA-1309; Debounce time config

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #203 from sborya/DebounceConfig

5 months agoSAMZA-1313 : Rename StandaloneJobCoordinator to PassthroughJobCoordinator
Boris Shkolnik [Fri, 26 May 2017 22:45:52 +0000 (15:45 -0700)] 
SAMZA-1313 : Rename StandaloneJobCoordinator to PassthroughJobCoordinator

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #206 from sborya/RenameStandAloneGJobCoordinator

5 months agoUpdating committer page on samza.apache.org
Navina Ramesh [Thu, 25 May 2017 22:47:57 +0000 (15:47 -0700)] 
Updating committer page on samza.apache.org

Giving more screen time for currently active Apache Samza committers (no JIRA)

Author: Navina Ramesh <navina@apache.org>

Reviewers: Yi Pan <nickpan47@gmail.com>

Closes #201 from navina/reOrgCommitters

6 months agoSAMZA-1128 : Remove dependency of debounce timer from the CoordinationUtils
Navina Ramesh [Thu, 25 May 2017 00:46:12 +0000 (17:46 -0700)] 
SAMZA-1128 : Remove dependency of debounce timer from the CoordinationUtils

This patch addresses the following, apart from the main bug that barrier implementation uses a different scheduler that the Jobcoordinator's main thread.
* Removes CoordinationUtils#getBarrier, BarrierForVersionUpgrade interface
* Renamed ZkBarrierForVersionUpgrade to ZkBarrier and introduces a listener ZkBarrierListener
* Simplified the ZkBarrier class and its integration test

Author: Navina Ramesh <navina@apache.org>

Reviewers: Boris Shkolnik <boryas@apache.org>, Bharath Kumarasubramanian <bkumarasubramanian@linkedin.com>

Closes #195 from navina/SAMZA-1128

6 months agoSAMZA-1307 - Fix ZkKeyBuilder null checks for pathPrefix
Navina Ramesh [Wed, 24 May 2017 06:33:09 +0000 (23:33 -0700)] 
SAMZA-1307 - Fix ZkKeyBuilder null checks for pathPrefix

Author: Navina Ramesh <navina@apache.org>

Reviewers: Jagadish V <jagadish@apache.org>

Closes #202 from navina/SAMZA-1307

6 months agoImprove documentation for Resource Localization
vjagadish1989 [Tue, 23 May 2017 00:42:32 +0000 (17:42 -0700)] 
Improve documentation for Resource Localization

This is a follow-up to Fred Ji's original PR : https://github.com/apache/samza/pull/191 .

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #199 from vjagadish1989/doc-improvements

6 months agoSAMZA-1280; document for the general/universal resource localization in YARN
Fred Ji [Mon, 22 May 2017 22:09:32 +0000 (15:09 -0700)] 
SAMZA-1280; document for the general/universal resource localization in YARN

This PR added a MD for localizing resource in Samza on YARN  by configuring path, local.name, local.type and local.visibility, and also updated the configuration table and index table.

Author: Fred Ji <fji@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #191 from fredji97/resourceLocalizationDoc

6 months agoSAMZA-1298; Create zk path.
Boris Shkolnik [Mon, 22 May 2017 18:58:53 +0000 (11:58 -0700)] 
SAMZA-1298; Create zk path.

if ZK path contains extra path at the end, it needs to be created in ZK at first connect.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Jagadish <jagadish@apache.org>

Closes #197 from sborya/createZkPath

6 months agoSAMZA-1291: StandAlone config
Boris Shkolnik [Mon, 22 May 2017 16:36:12 +0000 (09:36 -0700)] 
SAMZA-1291: StandAlone config

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #192 from sborya/StandAloneConfig

6 months agoSAMZA-1193. re-enable the test TestZkBarrierForVersionUpgrade.testZkBarrierForVersion...
Boris Shkolnik [Sat, 20 May 2017 00:25:26 +0000 (17:25 -0700)] 
SAMZA-1193. re-enable the test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

The code changed significantly. I've rerun the test multiple times both with gradle and intelij. It passed every time. I suggest we enable it back.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #181 from sborya/testZkBarrierForVersionUpgrade

6 months agoSAMZA-1258. Integration tests. Happy Path.
Boris Shkolnik [Fri, 19 May 2017 17:03:00 +0000 (10:03 -0700)] 
SAMZA-1258. Integration tests. Happy Path.

This is reincarnation of PR #157
It was not possible to merge/rebase that branch, so I had to create another one and reapply the changes.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #178 from sborya/unitTestHappyPath4

6 months agoSAMZA-1287: Updated instructions for running integration and stability tests.
Prateek Maheshwari [Mon, 15 May 2017 21:34:56 +0000 (14:34 -0700)] 
SAMZA-1287: Updated instructions for running integration and stability tests.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #187 from prateekm/integ-test-documentation

6 months agoSAMZA-1290: Fix javadoc for ApplicationRuner
Jacob Maes [Fri, 12 May 2017 22:22:47 +0000 (15:22 -0700)] 
SAMZA-1290: Fix javadoc for ApplicationRuner

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #190 from jmakes/samza-1290

6 months agoSAMZA-1289: Default id generator if not configured
Xinyu Liu [Fri, 12 May 2017 21:20:15 +0000 (14:20 -0700)] 
SAMZA-1289: Default id generator if not configured

Right now in standalone deployment we require the user to provide an id generator. Since most of the time the users can simply use the UUID generator for id generation, we should default it using that.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish V <vjagadish1989@apache.org>

Closes #189 from xinyuiscool/SAMZA-1289

6 months agoSAMZA-1288: Add null check for sink OutputStream
Xinyu Liu [Fri, 12 May 2017 16:49:04 +0000 (09:49 -0700)] 
SAMZA-1288: Add null check for sink OutputStream

The logic to generate json for Sink operator does not check whether the output stream is null. This causes null pointer exception.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apached.org>

Closes #188 from xinyuiscool/SAMZA-1288

6 months agoIncrease the plan graph size
Xinyu Liu [Thu, 11 May 2017 20:39:48 +0000 (13:39 -0700)] 
Increase the plan graph size

Increase the canvas size to a standard 24 inch resolution and also the scaling factor.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #186 from xinyuiscool/PLAN

6 months agoSAMZA-1174; Profiling state store performance
jmehar2 [Thu, 11 May 2017 19:56:31 +0000 (12:56 -0700)] 
SAMZA-1174; Profiling state store performance

This is s commit for [SAMZA-1174](https://issues.apache.org/jira/browse/SAMZA-1174). This commit involves gathering a log of operations (read, write, delete, etc.) happening on the state and publishing them into a kafka topic. It is names as "Access Log" behaving similar to changelog, but gathering log information.

Author: jmehar2 <jmehar2@illinois.edu>
Author: Jayasi Mehar <jayasi05@gmail.com>
Author: s-noghabi <abdolla2@illinois.edu>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jagadish V <jagadish@apache.org>

Closes #132 from s-noghabi/master

6 months agoSAMZA-1286: Close zk connection in ZkController.stop()
Boris Shkolnik [Thu, 11 May 2017 18:35:55 +0000 (11:35 -0700)] 
SAMZA-1286: Close zk connection in ZkController.stop()

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #185 from sborya/ZkControllerClose

6 months agoSAMZA-871; Heart-beat mechanism between JobCoordinator and all running containers release-0.13.0-rc0
Abhishek Shivanna [Wed, 10 May 2017 23:58:14 +0000 (16:58 -0700)] 
SAMZA-871; Heart-beat mechanism between JobCoordinator and all running containers

Author: Abhishek Shivanna <ashivanna@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>, Jagadish V <jagadish@apache.org>

Closes #163 from abhishekshivanna/master

6 months agoSAMZA-1279: Do not process onProcessorChange if list of children is null.
Boris Shkolnik [Wed, 10 May 2017 23:57:42 +0000 (16:57 -0700)] 
SAMZA-1279: Do not process onProcessorChange if list of children is null.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #180 from sborya/NPEinCaseOfZKException

6 months agoSAMZA-1283: Expose the buffered-message-size metric
Xinyu Liu [Wed, 10 May 2017 23:50:49 +0000 (16:50 -0700)] 
SAMZA-1283: Expose the buffered-message-size metric

Regardless of whether we enable size limit for the consumer buffer, this metric helps to see what's the buffer size and make configuring size limit easier.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish Venkatraman <vjagadish1989@gmail.com>

Closes #184 from xinyuiscool/SAMZA-1283

6 months agoSAMZA-1277: Add a static merge() operator that takes all streams to merge as input
Prateek Maheshwari [Wed, 10 May 2017 22:10:02 +0000 (15:10 -0700)] 
SAMZA-1277: Add a static merge() operator that takes all streams to merge as input

Also updated documentation for join and partitionBy.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #182 from prateekm/documentation-cleanup

6 months agoMinor: Re-word a couple of lines in javadoc with explicit notes about failures
vjagadish1989 [Wed, 10 May 2017 19:03:53 +0000 (12:03 -0700)] 
Minor: Re-word a couple of lines in javadoc with explicit notes about failures

6 months agoRevert "Disable flaky tests."
vjagadish1989 [Wed, 10 May 2017 18:41:17 +0000 (11:41 -0700)] 
Revert "Disable flaky tests."

This reverts commit df5aa6c4eeb9b89a40780b19e08bb6d96b97730b.

6 months agoDisable flaky tests.
Shanthoosh Venkataraman [Wed, 10 May 2017 16:27:57 +0000 (09:27 -0700)] 
Disable flaky tests.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh<navina@apache.org>

Closes #158 from shanthoosh/master

6 months agoSAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer
Navina Ramesh [Wed, 10 May 2017 01:24:09 +0000 (18:24 -0700)] 
SAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer

SamzaContainer is not adding custom reporters along side class-loaded reporters from config. This was missed in [SAMZA-1080](https://issues.apache.org/jira/browse/SAMZA-1080).

Author: Navina Ramesh <navina@apache.org>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jagadish <jagadish@apache.org>

Closes #179 from navina/SAMZA-1276

6 months agoSAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset
Navina Ramesh [Wed, 10 May 2017 01:18:52 +0000 (18:18 -0700)] 
SAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset

* ZkCoordinationUtils has a reset interface that deletes the entire Zk tree. This is not desirable.
* Also, fixed flakiness in unit test by unique barrier name in each of the unit tests. Otherwise, they share the same path on Zk and fail during concurrent test execution

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #173 from navina/SAMZA-1272

6 months agoSAMZA-1275: Kafka throws when users configure replication.factor for …
Jacob Maes [Tue, 9 May 2017 22:58:14 +0000 (15:58 -0700)] 
SAMZA-1275: Kafka throws when users configure replication.factor for …

…Kafka default stream

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Xinyu Liu <xiliu@linkedin.com>

Closes #176 from jmakes/samza-1275

6 months agoSAMZA-1268: More javadoc cleanup for public APIs for 0.13 release
Prateek Maheshwari [Tue, 9 May 2017 22:07:32 +0000 (15:07 -0700)] 
SAMZA-1268: More javadoc cleanup for public APIs for 0.13 release

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #175 from prateekm/documentation-cleanup

6 months agoSAMZA-1274; Update kafka-python and kafka broker version for integration tests
Prateek Maheshwari [Tue, 9 May 2017 17:41:50 +0000 (10:41 -0700)] 
SAMZA-1274; Update kafka-python and kafka broker version for integration tests

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #174 from prateekm/integration-test-updates

6 months agoSAMZA-1273: Make StreamConfig.getStreamIds() public
Xinyu Liu [Tue, 9 May 2017 16:27:28 +0000 (09:27 -0700)] 
SAMZA-1273: Make StreamConfig.getStreamIds() public

Making StreamConfig.getStreamIds() public so config provider can scan through all the configured streams and expand some properties if needed.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #172 from xinyuiscool/SAMZA-1273

6 months agoSAMZA-1268: Javadoc cleanup for public APIs for 0.13 release
Prateek Maheshwari [Tue, 9 May 2017 02:24:03 +0000 (19:24 -0700)] 
SAMZA-1268: Javadoc cleanup for public APIs for 0.13 release

This PR cleans up javadocs for Fluent API classes in the samza-api module.
Also updates the TaskContext (existing) and ContextManager (new) interfaces to add support for setting an pass-through user-defined context.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #169 from prateekm/api-docs-cleanup

6 months agoSAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer
Navina Ramesh [Tue, 9 May 2017 00:58:55 +0000 (17:58 -0700)] 
SAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer

This PR depends on PR #153
* Treats all errors in jobcoordinator as FATAL and shuts-down the streamprocessor
* [Bug] Fixed bug reported in SAMZA-1241
* Introduced a callback to be associated with the timer (same callback for every Runnable failure)

**TBD**: some more unit tests

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@apache.org>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #166 from navina/SAMZA-1150

6 months agoSAMZA-1206; Fix TestJMXServer.
Shanthoosh Venkataraman [Tue, 9 May 2017 00:22:14 +0000 (17:22 -0700)] 
SAMZA-1206; Fix TestJMXServer.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh<navina@apache.org>

Closes #164 from shanthoosh/fix-test-jmx-server-1

6 months agoSAMZA-1196; Fix TestJmxReporter
Shanthoosh Venkataraman [Tue, 9 May 2017 00:19:36 +0000 (17:19 -0700)] 
SAMZA-1196; Fix TestJmxReporter

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #140 from shanthoosh/fix-test-jmx-reporter

6 months agoSAMZA-1232; Log configuration value in RunLoopFactory
Shanthoosh Venkataraman [Tue, 9 May 2017 00:18:38 +0000 (17:18 -0700)] 
SAMZA-1232; Log configuration value in RunLoopFactory

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #137 from shanthoosh/adding_logging_into_asyncrunloop

6 months agoDisabled a few flaky tests and added corresponding tickets to fix.
Prateek Maheshwari [Mon, 8 May 2017 20:29:23 +0000 (13:29 -0700)] 
Disabled a few flaky tests and added corresponding tickets to fix.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #171 from prateekm/disable-flaky-test

6 months agoSAMZA-1267: ApplicationRunner#getLocalRunner returns null
Xinyu Liu [Mon, 8 May 2017 16:39:43 +0000 (09:39 -0700)] 
SAMZA-1267: ApplicationRunner#getLocalRunner returns null

Remove ApplicationRunner#getLocalRunner and clean up any usage examples.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #168 from xinyuiscool/SAMZA-1267

6 months agoSAMZA-1155; Validate users configure window.ms when using the fluent API
vjagadish1989 [Sat, 6 May 2017 02:21:29 +0000 (19:21 -0700)] 
SAMZA-1155; Validate users configure window.ms when using the fluent API

We will compute triggering duration as follows:
- If user configures `task.window.ms` we will honor it as the triggering duration
- If not, we will use the `GCD(windowTriggerDurations, joinTTLs)` as the triggering duration.

Changes in this PR:
- Common Interface for all time based triggers
- Additional APIs in `StreamGraphImpl` to recursively traverse all `OperatorSpec`s
- Recursive computation of `triggerInterval` for each `WindowOperatorSpec`
- Tests for all the above

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jacob Maes <jmaes@linkedin.com>, Xinyu Liu <xinyu@apache.org>

Closes #160 from vjagadish1989/samza-1155

6 months agoSAMZA-1266: Unable to use MetricsSnapshotReporterFactory with fluent API
Jacob Maes [Fri, 5 May 2017 22:48:32 +0000 (15:48 -0700)] 
SAMZA-1266: Unable to use MetricsSnapshotReporterFactory with fluent API

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Navina Ramesh <navina@apache.org>

Closes #167 from jmakes/samza-1266

6 months agoSAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector & ZkController
Navina Ramesh [Fri, 5 May 2017 20:50:08 +0000 (13:50 -0700)] 
SAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector & ZkController

Addresses the following:
* Makes LeaderElectionListener to be explicitly registered by the caller
* Removes debouncetimer dependency from ZkLeaderElector implementation
* [Bug] onBecomeLeader was scheduling a task in timer under "OnBecomeLeader", when it should actually be the same as "OnProcessorChange". Otherwise, it will not cancel when there is a new OnProcessorChange event.
* [Transient Test Failure] `TestScheduleAfterDebounceTime` tests were relying on timing controlled by sleep. Fixed it by using latch

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Boris Shkolnik <boryas@apache.org>

Closes #153 from navina/SAMZA-1251

6 months agoSAMZA-1228 : StreamProcessor should stop JmxServer
Navina Ramesh [Fri, 5 May 2017 20:22:35 +0000 (13:22 -0700)] 
SAMZA-1228 : StreamProcessor should stop JmxServer

This is not the solution posted in SAMZA-1228. For now, we are moving jmxserver lifecycle to be within the container. Ideally, it should be within the Streamprocessor so that the job coordinator can also be associated with the same instance.

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #162 from navina/SAMZA-1228

6 months agoSAMZA-1222: Clean up LocalApplicationRunner
Xinyu Liu [Thu, 4 May 2017 22:17:56 +0000 (15:17 -0700)] 
SAMZA-1222: Clean up LocalApplicationRunner

Clean up the LocalApplicationRunner based on the further feedback. The changes include the following:
1. Remove the processorId from the JobCoordinatorFactory/JobCoordinator interfaces
2. LocalApplicationRunner.run() is non-blocking. Add LocalApplicationRunner.waitForFinish() for blocking for completion
3. Remove the config for CooridnatorServiceFactory, and now the CoordinatorService is created based on the type of JobCoordinator.
4. Clean up the StreamProcessor life cycle listener logic inside LocalApplicationRunner.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #135 from xinyuiscool/SAMZA-1222

6 months agoSAMZA-1263: Samza Fluent: NPE is streams.id.samza.system is missing
Jacob Maes [Thu, 4 May 2017 20:57:50 +0000 (13:57 -0700)] 
SAMZA-1263: Samza Fluent: NPE is streams.id.samza.system is missing

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #165 from jmakes/samza-1263

6 months agoSAMZA-1247: MessageStreamImpl#merge shouldn't mutate input collection
Prateek Maheshwari [Thu, 4 May 2017 19:28:58 +0000 (12:28 -0700)] 
SAMZA-1247: MessageStreamImpl#merge shouldn't mutate input collection

Also fixes
SAMZA-1253: MessageStream.merge operator broken for nested types

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>, Jagadish <jvenkatr@linkedin.com>

Closes #159 from prateekm/merge-fixes

6 months agoSAMZA-1224 : Revert job coordinator factory config to the old format
Navina Ramesh [Thu, 4 May 2017 06:17:59 +0000 (23:17 -0700)] 
SAMZA-1224 : Revert job coordinator factory config to the old format

We didn't release since adding this config. So, it is ok to change the format now.

Author: nramesh <nramesh@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #146 from navina/SAMZA-1224

6 months agoSAMZA-1259: LocalApplicationRunner throws exception when configured with a ProcessorI...
Sean McCauliff [Thu, 4 May 2017 05:58:51 +0000 (22:58 -0700)] 
SAMZA-1259: LocalApplicationRunner throws exception when configured with a ProcessorIdGenerator

Minor fix.  Unit tests were updated and passed.

Author: Sean McCauliff <smccauliff@linkedin.com>

Reviewers: Navina Ramesh <nramesh@linkedin.com>

Closes #161 from smccauliff/samza-1259

6 months agoSAMZA-1257: make sure a dataChange listener in LeaderElection is always initialized.
Boris Shkolnik [Thu, 4 May 2017 00:03:08 +0000 (17:03 -0700)] 
SAMZA-1257: make sure a dataChange listener in LeaderElection is always initialized.

xinyuiscool, navina please review.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #156 from sborya/SAMZA-1257

6 months agoSAMZA-1212 - Refactor interaction between StreamProcessor, JobCoordinator and SamzaCo...
Navina Ramesh [Wed, 3 May 2017 22:10:13 +0000 (15:10 -0700)] 
SAMZA-1212 - Refactor interaction between StreamProcessor, JobCoordinator and SamzaContainer

See SAMZA-1212 for motivation toward this refactoring.
Changes here are:
* Removed awaitStart (blocking) method in StreamProcessor, JobCoordinator and SamzaContainer
* Introduced SamzaContainerListener and JobCoordinatorListener interface implemented by StreamProcessor
* Introduced SamzaContainerStatus to handler failures and lifecycle using Listener interfaces

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #148 from navina/SAMZA-1212

6 months agoSAMZA-1256: Improve trace logging for troubleshooting the fluent API
Jacob Maes [Wed, 3 May 2017 20:54:35 +0000 (13:54 -0700)] 
SAMZA-1256: Improve trace logging for troubleshooting the fluent API

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #155 from jmakes/operator-trace-logging

6 months agoSAMZA-1246: ApplicatonRunner.stats() should include exception in case of failure
Xinyu Liu [Wed, 3 May 2017 19:17:22 +0000 (12:17 -0700)] 
SAMZA-1246: ApplicatonRunner.stats() should include exception in case of failure

Current when ApplicationRunner.stats() only returns the enum representing the status. It also need to include the exception if the status is failure.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #154 from xinyuiscool/SAMZA-1246

6 months agofix HdfsSystemAdmin when staging directory is empty
Hai Lu [Tue, 2 May 2017 19:09:55 +0000 (12:09 -0700)] 
fix HdfsSystemAdmin when staging directory is empty

getSystemStreamMetadata has the potential side effect to persist metadata to a staging directory on hdfs. This could fail if staging directory is empty. This patch addresses the issue with test to cover the scenario.

Author: Hai Lu <halu@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #151 from lhaiesp/master

6 months agoSAMZA-1204: Visualize StreamGraph and ExecutionPlan
Xinyu Liu [Tue, 2 May 2017 17:09:22 +0000 (10:09 -0700)] 
SAMZA-1204: Visualize StreamGraph and ExecutionPlan

Once a Samza application (using fluent API) is deployed, an execution plan will be generated by the ExecutionPlanner. The plan JSON will be written to a file (plan.json) under the ./plan directory, which also contains the plan.html and javscripts (js folder).

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>
Author: xinyuiscool <xinyuliu.us@gmail.com>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #127 from xinyuiscool/SAMZA-1204

6 months agoSAMZA-1248 - Fix StandAlone barrier start list.
Boris Shkolnik [Tue, 2 May 2017 01:32:48 +0000 (18:32 -0700)] 
SAMZA-1248 - Fix StandAlone barrier start list.

6 months agoSAMZA-1249: Fix equality for WindowKey for Non-keyed tumbling windows
vjagadish1989 [Mon, 1 May 2017 20:57:25 +0000 (13:57 -0700)] 
SAMZA-1249: Fix equality for WindowKey for Non-keyed tumbling windows

- Fix a `ClassCastException` and an NPE when using Tumbling window without keys
- Fix equality and hashCode for `WindowKey`
- Refactor the `TestWindowOperator` unit tests using simpler types and a mock `MessageCollector`.

More details in `SAMZA-1249`

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jacob Maes <jmaes@linkedin.com>

Closes #149 from vjagadish1989/samza-1249

6 months agoSAMZA-1214: Bug - system-scoped stream default configs may not be honored
Jacob Maes [Mon, 1 May 2017 20:52:13 +0000 (13:52 -0700)] 
SAMZA-1214: Bug - system-scoped stream default configs may not be honored

* Re-introduced deprecated system-stream configs into config table
* Fixed position of task.consumer.batch.size in config table
* Moved system-scoped defaults from StreamConfig to SystemConfig

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #150 from jmakes/samza-1214

6 months agoSAMZA-1250: JobRunner.kill doesn't terminate cleanly with YarnJob.
Jacob Maes [Mon, 1 May 2017 20:44:54 +0000 (13:44 -0700)] 
SAMZA-1250: JobRunner.kill doesn't terminate cleanly with YarnJob.

1. The ClientHelper now checks inactive application IDs so it can get status for terminated jobs in addition to running jobs
2. JobRunner.kill() waits for any finish, not just successful finish.
3. A killed job is now considered successful.

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #152 from jmakes/samza-1250

6 months agoSAMZA-1219; Add metrics for operator message received and execution times
Prateek Maheshwari [Thu, 27 Apr 2017 22:55:11 +0000 (15:55 -0700)] 
SAMZA-1219; Add metrics for operator message received and execution times

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #142 from prateekm/operator-metrics

6 months agoSamza 1214: Allow users to set a default replication.factor for intermediate topics
Jacob Maes [Thu, 27 Apr 2017 22:05:28 +0000 (15:05 -0700)] 
Samza 1214: Allow users to set a default replication.factor for intermediate topics

* Add a new "systems.sysName.default.stream.*" config structure that allows users to set system-wide defaults for streams.
* More thorough testing of system defaults and stream defaults
* Removed the old migration config from the config table since there's no code to support it.
* Moved 2 kafka-specific config accessors out of JobConfig and into KafkaConfig
* Removed duplicate impl of getChangelogStream()

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>

Closes #141 from jmakes/samza-1214

6 months agoSAMZA-1210: Fixing merge issue and container id generations.
Boris Shkolnik [Thu, 27 Apr 2017 21:37:08 +0000 (14:37 -0700)] 
SAMZA-1210: Fixing merge issue and container id generations.

This PR is just for fixing issues introduced by a merge and changing container id generation.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #122 from sborya/mergeFixes

6 months agoSAMZA-1245: Make stream samza.physical.name config name string public
Xinyu Liu [Thu, 27 Apr 2017 20:21:11 +0000 (13:21 -0700)] 
SAMZA-1245: Make stream samza.physical.name config name string public

For certain system such as hdfs, the physical stream name might need to be finalized during the config generation. In order to do that, we will need to expose the stream samza.physical.name config string.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #145 from xinyuiscool/SAMZA-1245

6 months agoSAMZA-1200: Scala compile for samza-core fails with ambiguous reference error for...
Prateek Maheshwari [Wed, 26 Apr 2017 23:58:55 +0000 (16:58 -0700)] 
SAMZA-1200: Scala compile for samza-core fails with ambiguous reference error for some compiler versions

... for some compiler versions.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #143 from prateekm/logging-compile-fix

6 months agoSAMZA-1026: HDFS System Producer should not have Kafka dependency
Prateek Maheshwari [Wed, 26 Apr 2017 23:55:25 +0000 (16:55 -0700)] 
SAMZA-1026: HDFS System Producer should not have Kafka dependency

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <nramesh@linkedin.com>

Closes #144 from prateekm/hdfs-kafka-dependency

6 months agoSAMZA-1233: Create SystemAdmin only for JobCoordinator System in SamzaTaskProxy
Shanthoosh Venkataraman [Wed, 26 Apr 2017 16:05:41 +0000 (09:05 -0700)] 
SAMZA-1233: Create SystemAdmin only for JobCoordinator System in SamzaTaskProxy

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #138 from shanthoosh/fix_samza_task_proxy

6 months agoSAMZA-1220 : Add thread name to SamzaContainer shutdown hook and prevent shutdown...
Navina Ramesh [Tue, 25 Apr 2017 23:27:02 +0000 (16:27 -0700)] 
SAMZA-1220 : Add thread name to SamzaContainer shutdown hook and prevent shutdown deadlock

* SamzaContainerExceptionHandler is written in Java and used by LocalContainerRunner.java

Author: nramesh <nramesh@linkedin.com>

Reviewers: Jagadish Venkataraman <jvenkatr@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #139 from navina/SAMZA-1220

6 months agoSAMZA-1202; Multiple calls to getInputStream() result in non-deterministic behavior
vjagadish1989 [Tue, 25 Apr 2017 00:15:47 +0000 (17:15 -0700)] 
SAMZA-1202; Multiple calls to getInputStream() result in non-deterministic behavior

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<prateekm@linkedin.com>

Closes #136 from vjagadish1989/samza-1202

6 months agoSAMZA-1229; Disk space monitor should only count data in use by the container
Prateek Maheshwari [Mon, 24 Apr 2017 22:03:00 +0000 (15:03 -0700)] 
SAMZA-1229; Disk space monitor should only count data in use by the container

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #134 from prateekm/disk-space-monitor

7 months agoSAMZA-1226: relax type parameters in MessageStream functions
Yi Pan (Data Infrastructure) [Fri, 21 Apr 2017 23:14:41 +0000 (16:14 -0700)] 
SAMZA-1226: relax type parameters in MessageStream functions

relax the type parameter in user supplied functions in fluent API

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Navina Ramesh <navina@apache.org>, Jacob Maes <jmaes@linkedin.com>

Closes #133 from nickpan47/SAMZA-1226 and squashes the following commits:

b8d3461 [Yi Pan (Data Infrastructure)] SAMZA-1226: cleanup code example in StreamApplication javadoc
93fa471 [Yi Pan (Data Infrastructure)] SAMZA-1226: added more unit tests for type-cast functions
18e1e9f [Yi Pan (Data Infrastructure)] SAMZA-1226: address review feedbacks
b5da53b [Yi Pan (Data Infrastructure)] Merge branch 'master' into SAMZA-1226
7981b83 [Yi Pan (Data Infrastructure)] SAMZA-1226: relax type parameters in MessageStream functions

7 months agoSAMZA-868: support elasticsearch version 2.x
Jiri Humpolicek [Thu, 20 Apr 2017 08:25:31 +0000 (01:25 -0700)] 
SAMZA-868: support elasticsearch version 2.x

7 months agoSAMZA-1176: Intermittent TestJoinOperator unit test failure
Prateek Maheshwari [Wed, 19 Apr 2017 21:26:00 +0000 (14:26 -0700)] 
SAMZA-1176: Intermittent TestJoinOperator unit test failure

Join TTL was set too low (10 ms). `joinRetainsMatchedMessagesReverse` will fail if the execution time between line 176 and 186 is longer than that. Increased TTL to 1 min (will not affect test execution time).

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #131 from prateekm/join-test-fix

7 months agoSAMZA-1213 - StreamProcessorLifeCycleAware interface should not use processorId
Navina Ramesh [Wed, 19 Apr 2017 19:33:37 +0000 (12:33 -0700)] 
SAMZA-1213 - StreamProcessorLifeCycleAware interface should not use processorId

Refactoring LocalApplicationRunner s.t. each processor has its own listener instance, instead of a single listener keeping track of all processors.

Author: Navina Ramesh <navina@apache.org>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Xinyu Liu <xiliu@linkedin.com>

Closes #125 from navina/SAMZA-1213

7 months agoSAMZA-1209; Improve error handling in LocalStoreMonitor
Shanthoosh Venkataraman [Wed, 19 Apr 2017 06:18:13 +0000 (23:18 -0700)] 
SAMZA-1209; Improve error handling in LocalStoreMonitor

Changes:

1. Add opt-in configuration to continue garbage collection of local stores
   when there’s a failure in garbage collecting one local store.
2. Handle failures gracefully. In getJobModel, JobModel is expected as return response.
   Incase of failures, returned error-message Map is deserialized to JobModel
   resulting in ClassCastException. Handle 2xx, failures separately. Log error
   messages returned properly in httpGet call.
3. Fix getTasks url format.
4. Minor code cleanup(Switch to using ',' as seperator for job.status.servers configuration
   instead of '.' for ease of use).
5. Fix disabled tests in TestLocalStoreMonitor.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Closes #124 from shanthoosh/fix_config_format_in_localstore_monitor

7 months agoSAMZA-1217: Fix broken build due to TestCheckstyle problem.
Shanthoosh Venkataraman [Wed, 19 Apr 2017 04:23:39 +0000 (21:23 -0700)] 
SAMZA-1217: Fix broken build due to TestCheckstyle problem.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Yi Pan <nickpan47@gmail.com>

Closes #130 from shanthoosh/master

7 months agoSAMZA-1157: Serialization/deserialization throwables should not be suppressed
Yi Pan (Data Infrastructure) [Wed, 19 Apr 2017 00:15:32 +0000 (17:15 -0700)] 
SAMZA-1157: Serialization/deserialization throwables should not be suppressed

…ppressed if user don't configure to drop those errors

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #128 from nickpan47/fix-serde-error-suppressed

7 months agoSAMZA-1215, SAMZA-1216: remove ProcessKiller and change the ZooKeeper connection...
Yi Pan (Data Infrastructure) [Wed, 19 Apr 2017 00:04:29 +0000 (17:04 -0700)] 
SAMZA-1215, SAMZA-1216: remove ProcessKiller and change the ZooKeeper connection string in test

… test to use 127.0.0.1 in connect string

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #129 from nickpan47/fix-some-tests

7 months agoSAMZA-1182 : Disable flaky tests in TestAsyncRunLoop
Shanthoosh Venkataraman [Tue, 18 Apr 2017 22:05:27 +0000 (15:05 -0700)] 
SAMZA-1182 : Disable flaky tests in TestAsyncRunLoop

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #126 from shanthoosh/disable_all_async_run_loop_tests

7 months agoSAMZA-1205; Disable flaky tests in TestJmxServer.
Shanthoosh Venkataraman [Mon, 17 Apr 2017 21:25:11 +0000 (14:25 -0700)] 
SAMZA-1205; Disable flaky tests in TestJmxServer.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org?

Closes #120 from shanthoosh/disable_jmx_server_tests

7 months agoSAMZA-1211; Remove Thread.sleep() from TestJoinOperator tests
Prateek Maheshwari [Mon, 17 Apr 2017 19:09:25 +0000 (12:09 -0700)] 
SAMZA-1211; Remove Thread.sleep() from TestJoinOperator tests

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #123 from prateekm/join-test-no-sleep

7 months agoSAMZA-1145: Provide Ability To Confgure The Default Number Of Changel…
James Lent [Fri, 14 Apr 2017 19:09:07 +0000 (12:09 -0700)] 
SAMZA-1145: Provide Ability To Confgure The Default Number Of Changel…

…og Replicas

Author: James Lent <jlent@nc.rr.com>

Reviewers: Yi Pan <nickpan47@apache.org>, Jagadish <vjagadish1989@gmail.com>

Closes #86 from jwlent55/SAMZA-1145

7 months agoSAMZA-1195 fix bug in Samza application master on Kerberos
Chen Song [Fri, 14 Apr 2017 19:04:08 +0000 (12:04 -0700)] 
SAMZA-1195 fix bug in Samza application master on Kerberos

Author: Chen Song <chens_cs@hotmail.com>

Reviewers: Yi Pan <nickpan47@apache.org>

Closes #119 from garlicbulb-puzhuo/SAMZA-1195

7 months agoSAMZA-1183: Fix flaky tests in TestAsyncRunLoop
Shanthoosh Venkataraman [Thu, 13 Apr 2017 22:44:13 +0000 (15:44 -0700)] 
SAMZA-1183: Fix flaky tests in TestAsyncRunLoop

Changes:

* Moved all states locally within the test methods(no global state, since tests can be run parallely).
* Increased the timeout waiting for all task threads to finish.
* Changed message chooser mock to return null at the end(When EOS message is used a final msg, during some runs task.process gets invoked more than once for it. This is due to test setup, other tests within this test class also do the same.)

How was it verified:

Ran the test class repeatedly to check for failures(there were none) using following script.

`
while [ $i -lt 30 ] do     sleep 3;     i=`expr $i + 1`;     ./gradlew :samza-core:clean :samza-core:test -Dtest.single=TestAsyncRunLoop  done
`

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #108 from shanthoosh/fix-async-run-loop-tests-1

7 months agoSAMZA-1132: LocalApplicationRunner for StreamApplication
Xinyu Liu [Thu, 13 Apr 2017 22:39:23 +0000 (15:39 -0700)] 
SAMZA-1132: LocalApplicationRunner for StreamApplication

LocalApplicationRunner runs the StreamApplication locally on every node that the application is deployed to. LocalRunner.start() is blocking until the process is killed or user invoke LocalRunner.stop() from another thread. It can be configured to use Zk for coordination.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #117 from xinyuiscool/SAMZA-1132

7 months agoSAMZA-1208 - IllegalFormatConversionException in LocalContainerRunner
Navina Ramesh [Thu, 13 Apr 2017 01:40:01 +0000 (18:40 -0700)] 
SAMZA-1208 - IllegalFormatConversionException in LocalContainerRunner

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #121 from navina/SAMZA-1208

7 months agoSAMZA-1170: Integration testing harness for StreamApplications
vjagadish1989 [Wed, 12 Apr 2017 00:56:54 +0000 (17:56 -0700)] 
SAMZA-1170: Integration testing harness for StreamApplications

- Added an integration testing harness for testing `StreamApplication`s. This is convenient for running and interacting with Kafka brokers, Zk servers and Samza components locally.
- Added the following integration tests that use this harness to test actual `StreamApplication`s:
1. Keyed Tumbling Window
2. Keyed Session Window
3. Repartition with Keyed Session Window

Bonus: A couple of additional bug-fixes that were blockers for this test.

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<prateekm@linkedin.com>, Jacob Maes<jmaes@linkedin.com>

Closes #96 from vjagadish1989/integration-tests

7 months agoSAMZA-1198: disable the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVer...
Fred Ji [Mon, 10 Apr 2017 20:53:03 +0000 (13:53 -0700)] 
SAMZA-1198: disable the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

We are seeing the fails sometimes from this test, disabling it for build success first. See details in SAMZA-1198, and the JIRA for fix is in SAMZA-1193.

Author: Fred Ji <fji@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #116 from fredji97/disableFlaky

7 months agoSamza-1187 : TestZkProcessorLatch tests share state causing transient failures in...
Navina Ramesh [Sat, 8 Apr 2017 00:00:28 +0000 (17:00 -0700)] 
Samza-1187 : TestZkProcessorLatch tests share state causing transient failures in CI builds

* Removed testSingleCountdown - didn't understand what was being tested
* The timeout behavior for ZkProcessorLatch has not been documented. Hence, I have fixed testLatchExpires as best I could understand.
* Removed redundant/unused member variables from ZkProcessorLatch

Author: navina <navina@apache.org>

Reviewers: Yi Pan <nickpan47@gmail.com>

Closes #112 from navina/SAMZA-1187