samza.git
18 months agoSAMZA-1291: StandAlone config
Boris Shkolnik [Mon, 22 May 2017 16:36:12 +0000 (09:36 -0700)] 
SAMZA-1291: StandAlone config

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #192 from sborya/StandAloneConfig

18 months agoSAMZA-1193. re-enable the test TestZkBarrierForVersionUpgrade.testZkBarrierForVersion...
Boris Shkolnik [Sat, 20 May 2017 00:25:26 +0000 (17:25 -0700)] 
SAMZA-1193. re-enable the test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

The code changed significantly. I've rerun the test multiple times both with gradle and intelij. It passed every time. I suggest we enable it back.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #181 from sborya/testZkBarrierForVersionUpgrade

18 months agoSAMZA-1258. Integration tests. Happy Path.
Boris Shkolnik [Fri, 19 May 2017 17:03:00 +0000 (10:03 -0700)] 
SAMZA-1258. Integration tests. Happy Path.

This is reincarnation of PR #157
It was not possible to merge/rebase that branch, so I had to create another one and reapply the changes.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #178 from sborya/unitTestHappyPath4

18 months agoSAMZA-1287: Updated instructions for running integration and stability tests.
Prateek Maheshwari [Mon, 15 May 2017 21:34:56 +0000 (14:34 -0700)] 
SAMZA-1287: Updated instructions for running integration and stability tests.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #187 from prateekm/integ-test-documentation

18 months agoSAMZA-1290: Fix javadoc for ApplicationRuner
Jacob Maes [Fri, 12 May 2017 22:22:47 +0000 (15:22 -0700)] 
SAMZA-1290: Fix javadoc for ApplicationRuner

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #190 from jmakes/samza-1290

18 months agoSAMZA-1289: Default id generator if not configured
Xinyu Liu [Fri, 12 May 2017 21:20:15 +0000 (14:20 -0700)] 
SAMZA-1289: Default id generator if not configured

Right now in standalone deployment we require the user to provide an id generator. Since most of the time the users can simply use the UUID generator for id generation, we should default it using that.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish V <vjagadish1989@apache.org>

Closes #189 from xinyuiscool/SAMZA-1289

19 months agoSAMZA-1288: Add null check for sink OutputStream
Xinyu Liu [Fri, 12 May 2017 16:49:04 +0000 (09:49 -0700)] 
SAMZA-1288: Add null check for sink OutputStream

The logic to generate json for Sink operator does not check whether the output stream is null. This causes null pointer exception.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apached.org>

Closes #188 from xinyuiscool/SAMZA-1288

19 months agoIncrease the plan graph size
Xinyu Liu [Thu, 11 May 2017 20:39:48 +0000 (13:39 -0700)] 
Increase the plan graph size

Increase the canvas size to a standard 24 inch resolution and also the scaling factor.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #186 from xinyuiscool/PLAN

19 months agoSAMZA-1174; Profiling state store performance
jmehar2 [Thu, 11 May 2017 19:56:31 +0000 (12:56 -0700)] 
SAMZA-1174; Profiling state store performance

This is s commit for [SAMZA-1174](https://issues.apache.org/jira/browse/SAMZA-1174). This commit involves gathering a log of operations (read, write, delete, etc.) happening on the state and publishing them into a kafka topic. It is names as "Access Log" behaving similar to changelog, but gathering log information.

Author: jmehar2 <jmehar2@illinois.edu>
Author: Jayasi Mehar <jayasi05@gmail.com>
Author: s-noghabi <abdolla2@illinois.edu>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jagadish V <jagadish@apache.org>

Closes #132 from s-noghabi/master

19 months agoSAMZA-1286: Close zk connection in ZkController.stop()
Boris Shkolnik [Thu, 11 May 2017 18:35:55 +0000 (11:35 -0700)] 
SAMZA-1286: Close zk connection in ZkController.stop()

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #185 from sborya/ZkControllerClose

19 months agoSAMZA-871; Heart-beat mechanism between JobCoordinator and all running containers release-0.13.0-rc0
Abhishek Shivanna [Wed, 10 May 2017 23:58:14 +0000 (16:58 -0700)] 
SAMZA-871; Heart-beat mechanism between JobCoordinator and all running containers

Author: Abhishek Shivanna <ashivanna@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>, Jagadish V <jagadish@apache.org>

Closes #163 from abhishekshivanna/master

19 months agoSAMZA-1279: Do not process onProcessorChange if list of children is null.
Boris Shkolnik [Wed, 10 May 2017 23:57:42 +0000 (16:57 -0700)] 
SAMZA-1279: Do not process onProcessorChange if list of children is null.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #180 from sborya/NPEinCaseOfZKException

19 months agoSAMZA-1283: Expose the buffered-message-size metric
Xinyu Liu [Wed, 10 May 2017 23:50:49 +0000 (16:50 -0700)] 
SAMZA-1283: Expose the buffered-message-size metric

Regardless of whether we enable size limit for the consumer buffer, this metric helps to see what's the buffer size and make configuring size limit easier.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish Venkatraman <vjagadish1989@gmail.com>

Closes #184 from xinyuiscool/SAMZA-1283

19 months agoSAMZA-1277: Add a static merge() operator that takes all streams to merge as input
Prateek Maheshwari [Wed, 10 May 2017 22:10:02 +0000 (15:10 -0700)] 
SAMZA-1277: Add a static merge() operator that takes all streams to merge as input

Also updated documentation for join and partitionBy.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #182 from prateekm/documentation-cleanup

19 months agoMinor: Re-word a couple of lines in javadoc with explicit notes about failures
vjagadish1989 [Wed, 10 May 2017 19:03:53 +0000 (12:03 -0700)] 
Minor: Re-word a couple of lines in javadoc with explicit notes about failures

19 months agoRevert "Disable flaky tests."
vjagadish1989 [Wed, 10 May 2017 18:41:17 +0000 (11:41 -0700)] 
Revert "Disable flaky tests."

This reverts commit df5aa6c4eeb9b89a40780b19e08bb6d96b97730b.

19 months agoDisable flaky tests.
Shanthoosh Venkataraman [Wed, 10 May 2017 16:27:57 +0000 (09:27 -0700)] 
Disable flaky tests.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh<navina@apache.org>

Closes #158 from shanthoosh/master

19 months agoSAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer
Navina Ramesh [Wed, 10 May 2017 01:24:09 +0000 (18:24 -0700)] 
SAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer

SamzaContainer is not adding custom reporters along side class-loaded reporters from config. This was missed in [SAMZA-1080](https://issues.apache.org/jira/browse/SAMZA-1080).

Author: Navina Ramesh <navina@apache.org>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jagadish <jagadish@apache.org>

Closes #179 from navina/SAMZA-1276

19 months agoSAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset
Navina Ramesh [Wed, 10 May 2017 01:18:52 +0000 (18:18 -0700)] 
SAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset

* ZkCoordinationUtils has a reset interface that deletes the entire Zk tree. This is not desirable.
* Also, fixed flakiness in unit test by unique barrier name in each of the unit tests. Otherwise, they share the same path on Zk and fail during concurrent test execution

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #173 from navina/SAMZA-1272

19 months agoSAMZA-1275: Kafka throws when users configure replication.factor for …
Jacob Maes [Tue, 9 May 2017 22:58:14 +0000 (15:58 -0700)] 
SAMZA-1275: Kafka throws when users configure replication.factor for …

…Kafka default stream

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Xinyu Liu <xiliu@linkedin.com>

Closes #176 from jmakes/samza-1275

19 months agoSAMZA-1268: More javadoc cleanup for public APIs for 0.13 release
Prateek Maheshwari [Tue, 9 May 2017 22:07:32 +0000 (15:07 -0700)] 
SAMZA-1268: More javadoc cleanup for public APIs for 0.13 release

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #175 from prateekm/documentation-cleanup

19 months agoSAMZA-1274; Update kafka-python and kafka broker version for integration tests
Prateek Maheshwari [Tue, 9 May 2017 17:41:50 +0000 (10:41 -0700)] 
SAMZA-1274; Update kafka-python and kafka broker version for integration tests

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #174 from prateekm/integration-test-updates

19 months agoSAMZA-1273: Make StreamConfig.getStreamIds() public
Xinyu Liu [Tue, 9 May 2017 16:27:28 +0000 (09:27 -0700)] 
SAMZA-1273: Make StreamConfig.getStreamIds() public

Making StreamConfig.getStreamIds() public so config provider can scan through all the configured streams and expand some properties if needed.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #172 from xinyuiscool/SAMZA-1273

19 months agoSAMZA-1268: Javadoc cleanup for public APIs for 0.13 release
Prateek Maheshwari [Tue, 9 May 2017 02:24:03 +0000 (19:24 -0700)] 
SAMZA-1268: Javadoc cleanup for public APIs for 0.13 release

This PR cleans up javadocs for Fluent API classes in the samza-api module.
Also updates the TaskContext (existing) and ContextManager (new) interfaces to add support for setting an pass-through user-defined context.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #169 from prateekm/api-docs-cleanup

19 months agoSAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer
Navina Ramesh [Tue, 9 May 2017 00:58:55 +0000 (17:58 -0700)] 
SAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer

This PR depends on PR #153
* Treats all errors in jobcoordinator as FATAL and shuts-down the streamprocessor
* [Bug] Fixed bug reported in SAMZA-1241
* Introduced a callback to be associated with the timer (same callback for every Runnable failure)

**TBD**: some more unit tests

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@apache.org>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #166 from navina/SAMZA-1150

19 months agoSAMZA-1206; Fix TestJMXServer.
Shanthoosh Venkataraman [Tue, 9 May 2017 00:22:14 +0000 (17:22 -0700)] 
SAMZA-1206; Fix TestJMXServer.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh<navina@apache.org>

Closes #164 from shanthoosh/fix-test-jmx-server-1

19 months agoSAMZA-1196; Fix TestJmxReporter
Shanthoosh Venkataraman [Tue, 9 May 2017 00:19:36 +0000 (17:19 -0700)] 
SAMZA-1196; Fix TestJmxReporter

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #140 from shanthoosh/fix-test-jmx-reporter

19 months agoSAMZA-1232; Log configuration value in RunLoopFactory
Shanthoosh Venkataraman [Tue, 9 May 2017 00:18:38 +0000 (17:18 -0700)] 
SAMZA-1232; Log configuration value in RunLoopFactory

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #137 from shanthoosh/adding_logging_into_asyncrunloop

19 months agoDisabled a few flaky tests and added corresponding tickets to fix.
Prateek Maheshwari [Mon, 8 May 2017 20:29:23 +0000 (13:29 -0700)] 
Disabled a few flaky tests and added corresponding tickets to fix.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #171 from prateekm/disable-flaky-test

19 months agoSAMZA-1267: ApplicationRunner#getLocalRunner returns null
Xinyu Liu [Mon, 8 May 2017 16:39:43 +0000 (09:39 -0700)] 
SAMZA-1267: ApplicationRunner#getLocalRunner returns null

Remove ApplicationRunner#getLocalRunner and clean up any usage examples.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #168 from xinyuiscool/SAMZA-1267

19 months agoSAMZA-1155; Validate users configure window.ms when using the fluent API
vjagadish1989 [Sat, 6 May 2017 02:21:29 +0000 (19:21 -0700)] 
SAMZA-1155; Validate users configure window.ms when using the fluent API

We will compute triggering duration as follows:
- If user configures `task.window.ms` we will honor it as the triggering duration
- If not, we will use the `GCD(windowTriggerDurations, joinTTLs)` as the triggering duration.

Changes in this PR:
- Common Interface for all time based triggers
- Additional APIs in `StreamGraphImpl` to recursively traverse all `OperatorSpec`s
- Recursive computation of `triggerInterval` for each `WindowOperatorSpec`
- Tests for all the above

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jacob Maes <jmaes@linkedin.com>, Xinyu Liu <xinyu@apache.org>

Closes #160 from vjagadish1989/samza-1155

19 months agoSAMZA-1266: Unable to use MetricsSnapshotReporterFactory with fluent API
Jacob Maes [Fri, 5 May 2017 22:48:32 +0000 (15:48 -0700)] 
SAMZA-1266: Unable to use MetricsSnapshotReporterFactory with fluent API

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Navina Ramesh <navina@apache.org>

Closes #167 from jmakes/samza-1266

19 months agoSAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector & ZkController
Navina Ramesh [Fri, 5 May 2017 20:50:08 +0000 (13:50 -0700)] 
SAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector & ZkController

Addresses the following:
* Makes LeaderElectionListener to be explicitly registered by the caller
* Removes debouncetimer dependency from ZkLeaderElector implementation
* [Bug] onBecomeLeader was scheduling a task in timer under "OnBecomeLeader", when it should actually be the same as "OnProcessorChange". Otherwise, it will not cancel when there is a new OnProcessorChange event.
* [Transient Test Failure] `TestScheduleAfterDebounceTime` tests were relying on timing controlled by sleep. Fixed it by using latch

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Boris Shkolnik <boryas@apache.org>

Closes #153 from navina/SAMZA-1251

19 months agoSAMZA-1228 : StreamProcessor should stop JmxServer
Navina Ramesh [Fri, 5 May 2017 20:22:35 +0000 (13:22 -0700)] 
SAMZA-1228 : StreamProcessor should stop JmxServer

This is not the solution posted in SAMZA-1228. For now, we are moving jmxserver lifecycle to be within the container. Ideally, it should be within the Streamprocessor so that the job coordinator can also be associated with the same instance.

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #162 from navina/SAMZA-1228

19 months agoSAMZA-1222: Clean up LocalApplicationRunner
Xinyu Liu [Thu, 4 May 2017 22:17:56 +0000 (15:17 -0700)] 
SAMZA-1222: Clean up LocalApplicationRunner

Clean up the LocalApplicationRunner based on the further feedback. The changes include the following:
1. Remove the processorId from the JobCoordinatorFactory/JobCoordinator interfaces
2. LocalApplicationRunner.run() is non-blocking. Add LocalApplicationRunner.waitForFinish() for blocking for completion
3. Remove the config for CooridnatorServiceFactory, and now the CoordinatorService is created based on the type of JobCoordinator.
4. Clean up the StreamProcessor life cycle listener logic inside LocalApplicationRunner.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #135 from xinyuiscool/SAMZA-1222

19 months agoSAMZA-1263: Samza Fluent: NPE is streams.id.samza.system is missing
Jacob Maes [Thu, 4 May 2017 20:57:50 +0000 (13:57 -0700)] 
SAMZA-1263: Samza Fluent: NPE is streams.id.samza.system is missing

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #165 from jmakes/samza-1263

19 months agoSAMZA-1247: MessageStreamImpl#merge shouldn't mutate input collection
Prateek Maheshwari [Thu, 4 May 2017 19:28:58 +0000 (12:28 -0700)] 
SAMZA-1247: MessageStreamImpl#merge shouldn't mutate input collection

Also fixes
SAMZA-1253: MessageStream.merge operator broken for nested types

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>, Jagadish <jvenkatr@linkedin.com>

Closes #159 from prateekm/merge-fixes

19 months agoSAMZA-1224 : Revert job coordinator factory config to the old format
Navina Ramesh [Thu, 4 May 2017 06:17:59 +0000 (23:17 -0700)] 
SAMZA-1224 : Revert job coordinator factory config to the old format

We didn't release since adding this config. So, it is ok to change the format now.

Author: nramesh <nramesh@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #146 from navina/SAMZA-1224

19 months agoSAMZA-1259: LocalApplicationRunner throws exception when configured with a ProcessorI...
Sean McCauliff [Thu, 4 May 2017 05:58:51 +0000 (22:58 -0700)] 
SAMZA-1259: LocalApplicationRunner throws exception when configured with a ProcessorIdGenerator

Minor fix.  Unit tests were updated and passed.

Author: Sean McCauliff <smccauliff@linkedin.com>

Reviewers: Navina Ramesh <nramesh@linkedin.com>

Closes #161 from smccauliff/samza-1259

19 months agoSAMZA-1257: make sure a dataChange listener in LeaderElection is always initialized.
Boris Shkolnik [Thu, 4 May 2017 00:03:08 +0000 (17:03 -0700)] 
SAMZA-1257: make sure a dataChange listener in LeaderElection is always initialized.

xinyuiscool, navina please review.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #156 from sborya/SAMZA-1257

19 months agoSAMZA-1212 - Refactor interaction between StreamProcessor, JobCoordinator and SamzaCo...
Navina Ramesh [Wed, 3 May 2017 22:10:13 +0000 (15:10 -0700)] 
SAMZA-1212 - Refactor interaction between StreamProcessor, JobCoordinator and SamzaContainer

See SAMZA-1212 for motivation toward this refactoring.
Changes here are:
* Removed awaitStart (blocking) method in StreamProcessor, JobCoordinator and SamzaContainer
* Introduced SamzaContainerListener and JobCoordinatorListener interface implemented by StreamProcessor
* Introduced SamzaContainerStatus to handler failures and lifecycle using Listener interfaces

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #148 from navina/SAMZA-1212

19 months agoSAMZA-1256: Improve trace logging for troubleshooting the fluent API
Jacob Maes [Wed, 3 May 2017 20:54:35 +0000 (13:54 -0700)] 
SAMZA-1256: Improve trace logging for troubleshooting the fluent API

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #155 from jmakes/operator-trace-logging

19 months agoSAMZA-1246: ApplicatonRunner.stats() should include exception in case of failure
Xinyu Liu [Wed, 3 May 2017 19:17:22 +0000 (12:17 -0700)] 
SAMZA-1246: ApplicatonRunner.stats() should include exception in case of failure

Current when ApplicationRunner.stats() only returns the enum representing the status. It also need to include the exception if the status is failure.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #154 from xinyuiscool/SAMZA-1246

19 months agofix HdfsSystemAdmin when staging directory is empty
Hai Lu [Tue, 2 May 2017 19:09:55 +0000 (12:09 -0700)] 
fix HdfsSystemAdmin when staging directory is empty

getSystemStreamMetadata has the potential side effect to persist metadata to a staging directory on hdfs. This could fail if staging directory is empty. This patch addresses the issue with test to cover the scenario.

Author: Hai Lu <halu@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #151 from lhaiesp/master

19 months agoSAMZA-1204: Visualize StreamGraph and ExecutionPlan
Xinyu Liu [Tue, 2 May 2017 17:09:22 +0000 (10:09 -0700)] 
SAMZA-1204: Visualize StreamGraph and ExecutionPlan

Once a Samza application (using fluent API) is deployed, an execution plan will be generated by the ExecutionPlanner. The plan JSON will be written to a file (plan.json) under the ./plan directory, which also contains the plan.html and javscripts (js folder).

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>
Author: xinyuiscool <xinyuliu.us@gmail.com>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #127 from xinyuiscool/SAMZA-1204

19 months agoSAMZA-1248 - Fix StandAlone barrier start list.
Boris Shkolnik [Tue, 2 May 2017 01:32:48 +0000 (18:32 -0700)] 
SAMZA-1248 - Fix StandAlone barrier start list.

19 months agoSAMZA-1249: Fix equality for WindowKey for Non-keyed tumbling windows
vjagadish1989 [Mon, 1 May 2017 20:57:25 +0000 (13:57 -0700)] 
SAMZA-1249: Fix equality for WindowKey for Non-keyed tumbling windows

- Fix a `ClassCastException` and an NPE when using Tumbling window without keys
- Fix equality and hashCode for `WindowKey`
- Refactor the `TestWindowOperator` unit tests using simpler types and a mock `MessageCollector`.

More details in `SAMZA-1249`

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jacob Maes <jmaes@linkedin.com>

Closes #149 from vjagadish1989/samza-1249

19 months agoSAMZA-1214: Bug - system-scoped stream default configs may not be honored
Jacob Maes [Mon, 1 May 2017 20:52:13 +0000 (13:52 -0700)] 
SAMZA-1214: Bug - system-scoped stream default configs may not be honored

* Re-introduced deprecated system-stream configs into config table
* Fixed position of task.consumer.batch.size in config table
* Moved system-scoped defaults from StreamConfig to SystemConfig

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #150 from jmakes/samza-1214

19 months agoSAMZA-1250: JobRunner.kill doesn't terminate cleanly with YarnJob.
Jacob Maes [Mon, 1 May 2017 20:44:54 +0000 (13:44 -0700)] 
SAMZA-1250: JobRunner.kill doesn't terminate cleanly with YarnJob.

1. The ClientHelper now checks inactive application IDs so it can get status for terminated jobs in addition to running jobs
2. JobRunner.kill() waits for any finish, not just successful finish.
3. A killed job is now considered successful.

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #152 from jmakes/samza-1250

19 months agoSAMZA-1219; Add metrics for operator message received and execution times
Prateek Maheshwari [Thu, 27 Apr 2017 22:55:11 +0000 (15:55 -0700)] 
SAMZA-1219; Add metrics for operator message received and execution times

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #142 from prateekm/operator-metrics

19 months agoSamza 1214: Allow users to set a default replication.factor for intermediate topics
Jacob Maes [Thu, 27 Apr 2017 22:05:28 +0000 (15:05 -0700)] 
Samza 1214: Allow users to set a default replication.factor for intermediate topics

* Add a new "systems.sysName.default.stream.*" config structure that allows users to set system-wide defaults for streams.
* More thorough testing of system defaults and stream defaults
* Removed the old migration config from the config table since there's no code to support it.
* Moved 2 kafka-specific config accessors out of JobConfig and into KafkaConfig
* Removed duplicate impl of getChangelogStream()

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>

Closes #141 from jmakes/samza-1214

19 months agoSAMZA-1210: Fixing merge issue and container id generations.
Boris Shkolnik [Thu, 27 Apr 2017 21:37:08 +0000 (14:37 -0700)] 
SAMZA-1210: Fixing merge issue and container id generations.

This PR is just for fixing issues introduced by a merge and changing container id generation.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #122 from sborya/mergeFixes

19 months agoSAMZA-1245: Make stream samza.physical.name config name string public
Xinyu Liu [Thu, 27 Apr 2017 20:21:11 +0000 (13:21 -0700)] 
SAMZA-1245: Make stream samza.physical.name config name string public

For certain system such as hdfs, the physical stream name might need to be finalized during the config generation. In order to do that, we will need to expose the stream samza.physical.name config string.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #145 from xinyuiscool/SAMZA-1245

19 months agoSAMZA-1200: Scala compile for samza-core fails with ambiguous reference error for...
Prateek Maheshwari [Wed, 26 Apr 2017 23:58:55 +0000 (16:58 -0700)] 
SAMZA-1200: Scala compile for samza-core fails with ambiguous reference error for some compiler versions

... for some compiler versions.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #143 from prateekm/logging-compile-fix

19 months agoSAMZA-1026: HDFS System Producer should not have Kafka dependency
Prateek Maheshwari [Wed, 26 Apr 2017 23:55:25 +0000 (16:55 -0700)] 
SAMZA-1026: HDFS System Producer should not have Kafka dependency

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <nramesh@linkedin.com>

Closes #144 from prateekm/hdfs-kafka-dependency

19 months agoSAMZA-1233: Create SystemAdmin only for JobCoordinator System in SamzaTaskProxy
Shanthoosh Venkataraman [Wed, 26 Apr 2017 16:05:41 +0000 (09:05 -0700)] 
SAMZA-1233: Create SystemAdmin only for JobCoordinator System in SamzaTaskProxy

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #138 from shanthoosh/fix_samza_task_proxy

19 months agoSAMZA-1220 : Add thread name to SamzaContainer shutdown hook and prevent shutdown...
Navina Ramesh [Tue, 25 Apr 2017 23:27:02 +0000 (16:27 -0700)] 
SAMZA-1220 : Add thread name to SamzaContainer shutdown hook and prevent shutdown deadlock

* SamzaContainerExceptionHandler is written in Java and used by LocalContainerRunner.java

Author: nramesh <nramesh@linkedin.com>

Reviewers: Jagadish Venkataraman <jvenkatr@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #139 from navina/SAMZA-1220

19 months agoSAMZA-1202; Multiple calls to getInputStream() result in non-deterministic behavior
vjagadish1989 [Tue, 25 Apr 2017 00:15:47 +0000 (17:15 -0700)] 
SAMZA-1202; Multiple calls to getInputStream() result in non-deterministic behavior

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<prateekm@linkedin.com>

Closes #136 from vjagadish1989/samza-1202

19 months agoSAMZA-1229; Disk space monitor should only count data in use by the container
Prateek Maheshwari [Mon, 24 Apr 2017 22:03:00 +0000 (15:03 -0700)] 
SAMZA-1229; Disk space monitor should only count data in use by the container

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #134 from prateekm/disk-space-monitor

19 months agoSAMZA-1226: relax type parameters in MessageStream functions
Yi Pan (Data Infrastructure) [Fri, 21 Apr 2017 23:14:41 +0000 (16:14 -0700)] 
SAMZA-1226: relax type parameters in MessageStream functions

relax the type parameter in user supplied functions in fluent API

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Navina Ramesh <navina@apache.org>, Jacob Maes <jmaes@linkedin.com>

Closes #133 from nickpan47/SAMZA-1226 and squashes the following commits:

b8d3461 [Yi Pan (Data Infrastructure)] SAMZA-1226: cleanup code example in StreamApplication javadoc
93fa471 [Yi Pan (Data Infrastructure)] SAMZA-1226: added more unit tests for type-cast functions
18e1e9f [Yi Pan (Data Infrastructure)] SAMZA-1226: address review feedbacks
b5da53b [Yi Pan (Data Infrastructure)] Merge branch 'master' into SAMZA-1226
7981b83 [Yi Pan (Data Infrastructure)] SAMZA-1226: relax type parameters in MessageStream functions

19 months agoSAMZA-868: support elasticsearch version 2.x
Jiri Humpolicek [Thu, 20 Apr 2017 08:25:31 +0000 (01:25 -0700)] 
SAMZA-868: support elasticsearch version 2.x

19 months agoSAMZA-1176: Intermittent TestJoinOperator unit test failure
Prateek Maheshwari [Wed, 19 Apr 2017 21:26:00 +0000 (14:26 -0700)] 
SAMZA-1176: Intermittent TestJoinOperator unit test failure

Join TTL was set too low (10 ms). `joinRetainsMatchedMessagesReverse` will fail if the execution time between line 176 and 186 is longer than that. Increased TTL to 1 min (will not affect test execution time).

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #131 from prateekm/join-test-fix

19 months agoSAMZA-1213 - StreamProcessorLifeCycleAware interface should not use processorId
Navina Ramesh [Wed, 19 Apr 2017 19:33:37 +0000 (12:33 -0700)] 
SAMZA-1213 - StreamProcessorLifeCycleAware interface should not use processorId

Refactoring LocalApplicationRunner s.t. each processor has its own listener instance, instead of a single listener keeping track of all processors.

Author: Navina Ramesh <navina@apache.org>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Xinyu Liu <xiliu@linkedin.com>

Closes #125 from navina/SAMZA-1213

19 months agoSAMZA-1209; Improve error handling in LocalStoreMonitor
Shanthoosh Venkataraman [Wed, 19 Apr 2017 06:18:13 +0000 (23:18 -0700)] 
SAMZA-1209; Improve error handling in LocalStoreMonitor

Changes:

1. Add opt-in configuration to continue garbage collection of local stores
   when there’s a failure in garbage collecting one local store.
2. Handle failures gracefully. In getJobModel, JobModel is expected as return response.
   Incase of failures, returned error-message Map is deserialized to JobModel
   resulting in ClassCastException. Handle 2xx, failures separately. Log error
   messages returned properly in httpGet call.
3. Fix getTasks url format.
4. Minor code cleanup(Switch to using ',' as seperator for job.status.servers configuration
   instead of '.' for ease of use).
5. Fix disabled tests in TestLocalStoreMonitor.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Closes #124 from shanthoosh/fix_config_format_in_localstore_monitor

19 months agoSAMZA-1217: Fix broken build due to TestCheckstyle problem.
Shanthoosh Venkataraman [Wed, 19 Apr 2017 04:23:39 +0000 (21:23 -0700)] 
SAMZA-1217: Fix broken build due to TestCheckstyle problem.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Yi Pan <nickpan47@gmail.com>

Closes #130 from shanthoosh/master

19 months agoSAMZA-1157: Serialization/deserialization throwables should not be suppressed
Yi Pan (Data Infrastructure) [Wed, 19 Apr 2017 00:15:32 +0000 (17:15 -0700)] 
SAMZA-1157: Serialization/deserialization throwables should not be suppressed

…ppressed if user don't configure to drop those errors

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #128 from nickpan47/fix-serde-error-suppressed

19 months agoSAMZA-1215, SAMZA-1216: remove ProcessKiller and change the ZooKeeper connection...
Yi Pan (Data Infrastructure) [Wed, 19 Apr 2017 00:04:29 +0000 (17:04 -0700)] 
SAMZA-1215, SAMZA-1216: remove ProcessKiller and change the ZooKeeper connection string in test

… test to use 127.0.0.1 in connect string

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #129 from nickpan47/fix-some-tests

19 months agoSAMZA-1182 : Disable flaky tests in TestAsyncRunLoop
Shanthoosh Venkataraman [Tue, 18 Apr 2017 22:05:27 +0000 (15:05 -0700)] 
SAMZA-1182 : Disable flaky tests in TestAsyncRunLoop

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #126 from shanthoosh/disable_all_async_run_loop_tests

19 months agoSAMZA-1205; Disable flaky tests in TestJmxServer.
Shanthoosh Venkataraman [Mon, 17 Apr 2017 21:25:11 +0000 (14:25 -0700)] 
SAMZA-1205; Disable flaky tests in TestJmxServer.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org?

Closes #120 from shanthoosh/disable_jmx_server_tests

19 months agoSAMZA-1211; Remove Thread.sleep() from TestJoinOperator tests
Prateek Maheshwari [Mon, 17 Apr 2017 19:09:25 +0000 (12:09 -0700)] 
SAMZA-1211; Remove Thread.sleep() from TestJoinOperator tests

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #123 from prateekm/join-test-no-sleep

19 months agoSAMZA-1145: Provide Ability To Confgure The Default Number Of Changel…
James Lent [Fri, 14 Apr 2017 19:09:07 +0000 (12:09 -0700)] 
SAMZA-1145: Provide Ability To Confgure The Default Number Of Changel…

…og Replicas

Author: James Lent <jlent@nc.rr.com>

Reviewers: Yi Pan <nickpan47@apache.org>, Jagadish <vjagadish1989@gmail.com>

Closes #86 from jwlent55/SAMZA-1145

19 months agoSAMZA-1195 fix bug in Samza application master on Kerberos
Chen Song [Fri, 14 Apr 2017 19:04:08 +0000 (12:04 -0700)] 
SAMZA-1195 fix bug in Samza application master on Kerberos

Author: Chen Song <chens_cs@hotmail.com>

Reviewers: Yi Pan <nickpan47@apache.org>

Closes #119 from garlicbulb-puzhuo/SAMZA-1195

19 months agoSAMZA-1183: Fix flaky tests in TestAsyncRunLoop
Shanthoosh Venkataraman [Thu, 13 Apr 2017 22:44:13 +0000 (15:44 -0700)] 
SAMZA-1183: Fix flaky tests in TestAsyncRunLoop

Changes:

* Moved all states locally within the test methods(no global state, since tests can be run parallely).
* Increased the timeout waiting for all task threads to finish.
* Changed message chooser mock to return null at the end(When EOS message is used a final msg, during some runs task.process gets invoked more than once for it. This is due to test setup, other tests within this test class also do the same.)

How was it verified:

Ran the test class repeatedly to check for failures(there were none) using following script.

`
while [ $i -lt 30 ] do     sleep 3;     i=`expr $i + 1`;     ./gradlew :samza-core:clean :samza-core:test -Dtest.single=TestAsyncRunLoop  done
`

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #108 from shanthoosh/fix-async-run-loop-tests-1

19 months agoSAMZA-1132: LocalApplicationRunner for StreamApplication
Xinyu Liu [Thu, 13 Apr 2017 22:39:23 +0000 (15:39 -0700)] 
SAMZA-1132: LocalApplicationRunner for StreamApplication

LocalApplicationRunner runs the StreamApplication locally on every node that the application is deployed to. LocalRunner.start() is blocking until the process is killed or user invoke LocalRunner.stop() from another thread. It can be configured to use Zk for coordination.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #117 from xinyuiscool/SAMZA-1132

19 months agoSAMZA-1208 - IllegalFormatConversionException in LocalContainerRunner
Navina Ramesh [Thu, 13 Apr 2017 01:40:01 +0000 (18:40 -0700)] 
SAMZA-1208 - IllegalFormatConversionException in LocalContainerRunner

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #121 from navina/SAMZA-1208

20 months agoSAMZA-1170: Integration testing harness for StreamApplications
vjagadish1989 [Wed, 12 Apr 2017 00:56:54 +0000 (17:56 -0700)] 
SAMZA-1170: Integration testing harness for StreamApplications

- Added an integration testing harness for testing `StreamApplication`s. This is convenient for running and interacting with Kafka brokers, Zk servers and Samza components locally.
- Added the following integration tests that use this harness to test actual `StreamApplication`s:
1. Keyed Tumbling Window
2. Keyed Session Window
3. Repartition with Keyed Session Window

Bonus: A couple of additional bug-fixes that were blockers for this test.

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<prateekm@linkedin.com>, Jacob Maes<jmaes@linkedin.com>

Closes #96 from vjagadish1989/integration-tests

20 months agoSAMZA-1198: disable the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVer...
Fred Ji [Mon, 10 Apr 2017 20:53:03 +0000 (13:53 -0700)] 
SAMZA-1198: disable the flaky test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

We are seeing the fails sometimes from this test, disabling it for build success first. See details in SAMZA-1198, and the JIRA for fix is in SAMZA-1193.

Author: Fred Ji <fji@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #116 from fredji97/disableFlaky

20 months agoSamza-1187 : TestZkProcessorLatch tests share state causing transient failures in...
Navina Ramesh [Sat, 8 Apr 2017 00:00:28 +0000 (17:00 -0700)] 
Samza-1187 : TestZkProcessorLatch tests share state causing transient failures in CI builds

* Removed testSingleCountdown - didn't understand what was being tested
* The timeout behavior for ZkProcessorLatch has not been documented. Hence, I have fixed testLatchExpires as best I could understand.
* Removed redundant/unused member variables from ZkProcessorLatch

Author: navina <navina@apache.org>

Reviewers: Yi Pan <nickpan47@gmail.com>

Closes #112 from navina/SAMZA-1187

20 months agoSAMZA-1126 - Semantics of processorId in Samza
Navina Ramesh [Fri, 7 Apr 2017 22:22:13 +0000 (15:22 -0700)] 
SAMZA-1126 - Semantics of processorId in Samza

Implementation based on [SEP-1](https://cwiki.apache.org/confluence/display/SAMZA/SEP-1%3A+Semantics+of+ProcessorId+in+Samza)

Author: navina <navina@apache.org>

Reviewers: Yi Pan <nickpan47@gmail.com>, Jacob Maes <jmaes@linkedin.com>, Jagadish <jagadish@apache.org>

Closes #103 from navina/SAMZA-1126

20 months agoSAMZA-1194 - Commenting out testJmxReporter flaky test
Shanthoosh Venkataraman [Fri, 7 Apr 2017 22:17:10 +0000 (15:17 -0700)] 
SAMZA-1194 - Commenting out testJmxReporter flaky test

TestJmxReporter brings up an JmxServer and uses it to test the reporter behavior. However, due to failures in bringing up/connecting to JMX server, this fails intermittently. Temporarily disabling it.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #118 from shanthoosh/disable_jmx_reporter_tests

20 months agoSAMZA-1089: Enable YarnJob and ClientHelper to kill a job by name rather than YARN...
Jacob Maes [Fri, 7 Apr 2017 21:32:02 +0000 (14:32 -0700)] 
SAMZA-1089: Enable YarnJob and ClientHelper to kill a job by name rather than YARN ApplicationID

Missed a couple files in the previous commit to enable YarnJob to kill and get status of a Job based on the job name rather than the YARN ApplicationName.

These changes have been manually verified in a Yarn cluster at LI.

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>

Closes #114 from jmakes/samza-1089-3

20 months agoSAMZA-1192: Fixed TestJoinOperator test failure on JDK 1.8.0_05
Prateek Maheshwari [Thu, 6 Apr 2017 22:35:39 +0000 (15:35 -0700)] 
SAMZA-1192: Fixed TestJoinOperator test failure on JDK 1.8.0_05

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #115 from prateekm/join-test-fix

20 months agoSAMZA-1178: Generate JSON from StreamPlan
Xinyu Liu [Thu, 6 Apr 2017 16:58:38 +0000 (09:58 -0700)] 
SAMZA-1178: Generate JSON from StreamPlan

As the first step to visualize the StreamGraph/Plan, this patch generates a json representation of it. For the example StreamGraph in TestPlanJsonGenerator, here is the json that is generated (https://github.com/xinyuiscool/examples/blob/master/example_plan.json). This patch also introduces the immutable StreamPlan and did some code cleanup.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #110 from xinyuiscool/SAMZA-1178

20 months agoSAMZA-1143; Include fs.<scheme>.impl.* subkeys to YarnConfiguration used in YarnJobFa...
Fred Ji [Thu, 6 Apr 2017 05:28:38 +0000 (22:28 -0700)] 
SAMZA-1143; Include fs.<scheme>.impl.* subkeys to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager

SAMZA-1143 Include fs.&lt;scheme&gt;.impl.* subkeys, in addition tofs.&lt;scheme&gt;.impl, to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager.

When there are additional subconfigurations under fs.myScheme.impl, such as fs.myScheme.impl.client, we need to keep the set of configuration completed in  YarnJobFactory and YarnClusterResourceManager. When the context is set for localizing the resource in ClientHelper and YarnContainerRunner, it may rely on this configuration to get the FileStatus information, which may or may not depends on fs.&lt;scheme&gt;.impl and all possible fs.&lt;scheme&gt;.impl.* sub-configuration.

This is an enhanced feature to the PR#90.

Author: Fred Ji <fji@linkedin.com>
Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #97 from fredji97/fsImplSubkeys

20 months agoSAMZA-1191; Fixed flaky test: TestExponentialSleepStrategy testThreadInterruptInRetryLoop
Prateek Maheshwari [Thu, 6 Apr 2017 05:26:09 +0000 (22:26 -0700)] 
SAMZA-1191; Fixed flaky test: TestExponentialSleepStrategy testThreadInterruptInRetryLoop

It's possible that the interruptee thread (see `#interruptedThread`) gets pre-empted before it has a chance to run the operation (increment `iterations`) and then gets interrupted, causing these assertions to fail.

I think these assertions also aren't critical for the tests which I presume want to test interrupt propagation behavior, so removing them in this change.

vjagadish1989 & xinyuiscool, please take a look.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #113 from prateekm/flaky-ess-test-fix

20 months agoSAMZA-1094 SAMZA-1101 SAMZA-1159; Remove MessageEnvelope from public operator APIs...
Prateek Maheshwari [Wed, 5 Apr 2017 23:42:15 +0000 (16:42 -0700)] 
SAMZA-1094 SAMZA-1101 SAMZA-1159; Remove MessageEnvelope from public operator APIs. : Delay the creation of SinkFunction for output streams. : Move StreamSpec from a public API to an internal class.

Removed the MessageEnvelope and OutputStream interfaces from public operator APIs.
Moved the creation of SinkFunction for output streams to SinkOperatorSpec.
Moved StreamSpec from a public API to an internal class.

Additionally,
1. Removed references to StreamGraph in OperatorSpecs. It was being used to getNextOpId(). MessageStreamsImpl now gets the ID and gives it to OperatorSpecs itself.
2. Updated and cleaned up the StreamGraphBuilder examples.
3. Renamed SinkOperatorSpec to OutputOperatorSpec since its used by sink, sendTo and partitionBy.

nickpan47 and xinyuiscool, please take a look.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>, Yi Pan <nickpan47@gmail.com>

Closes #92 from prateekm/message-envelope-removal

20 months agoSAMZA-1189: Fix file system closed issue on hdfs system producer
Jacob Maes [Wed, 5 Apr 2017 03:37:32 +0000 (20:37 -0700)] 
SAMZA-1189: Fix file system closed issue on hdfs system producer

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>

Closes #111 from jmakes/samza-1189

20 months agoSamza 1186: Rename Processor to Job
Xinyu Liu [Tue, 4 Apr 2017 01:24:33 +0000 (18:24 -0700)] 
Samza 1186: Rename Processor to Job

Now we have the top level Samza application, and each stage is called a job, the previous introduced "processor" naming should be renamed as Job. This includes renaming PrcessorGraph to JobGraph, and ProcessorNode to JobNode, etc.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #109 from xinyuiscool/SAMZA-1186

20 months agoSAMZA-1089: Runner should support kill and status commands
Jacob Maes [Mon, 3 Apr 2017 16:50:27 +0000 (09:50 -0700)] 
SAMZA-1089: Runner should support kill and status commands

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>,Xinyu Liu <xiliu@linkedin.com>

Closes #106 from jmakes/samza-1089-2

20 months agoSAMZA-1138; Yarn capability check is broken
Maksim Logvinenko [Sun, 2 Apr 2017 23:18:43 +0000 (16:18 -0700)] 
SAMZA-1138; Yarn capability check is broken

After migration from `yarn.container.*` properties to `cluster-manager.container.*` properties we have to use either of them and `ClusterManagerConfig` provides backward compatibility for these properties. But in `YarnClusterResourceManage`r only old properties are used (from `YarnConfig`), hence if job config migrated to new `cluster-manager.*` properties names then check will be evaluated against default values, not against actual values.

Author: Maksim Logvinenko <mlogvinenko@gmail.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #84 from logarithm/yarn-properties-fix

20 months agoSAMZA-1181: Fix AppMaster hang after submitting jobs to Yarn
Shanthoosh Venkataraman [Fri, 31 Mar 2017 22:58:22 +0000 (15:58 -0700)] 
SAMZA-1181: Fix AppMaster hang after submitting jobs to Yarn

Currently when a job is submitted to Yarn, it is going to hang after AppMaster is created. The log shows that it hangs during bootstrapping from Coordinator stream. Further debugging shows that the jobs hang in the second time of bootstrap while reading locality data from LocalityManager. The sequence is the following:
1. JobModelManager creates CoordinatorStreamConsumer, and bootstrap it,
2. LocalityManager writes locality info into coordinator stream
3. JobModelManager closes CoordinatorStreamConsumer
4. Later localityManager bootstraps CoordinatorStreamConsumer again
Step 3 is the problem here. Since CoordinatorStreamConsumer is still held by LocalityManager, it cannot be closed prematurely. Step 3 is introduced in SAMZA-1154, as a refactoring of JobModelManager for task rest end point. To fix this issue, we will revert this change of step 3.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #104 from shanthoosh/master

20 months agoSAMZA-1176; Make TestJoinOperator unit tests safe for concurrent execution
Prateek Maheshwari [Fri, 31 Mar 2017 21:00:30 +0000 (14:00 -0700)] 
SAMZA-1176; Make TestJoinOperator unit tests safe for concurrent execution

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #105 from prateekm/join-tests

20 months agoSAMZA-1182 - Commenting out some of the flaky tests
Navina Ramesh [Fri, 31 Mar 2017 20:48:35 +0000 (13:48 -0700)] 
SAMZA-1182 - Commenting out some of the flaky tests

Author: navina <navina@apache.org>

Reviewers: Prateek Maheshwari <pmaheshwari@linkedin.com>, Jagadish Venkataraman <vjagadish1989@gmail.com>

Closes #107 from navina/SAMZA-1182

20 months ago SAMZA-1135 - Support scala 2.12
vjagadish1989 [Fri, 31 Mar 2017 17:26:08 +0000 (10:26 -0700)] 
SAMZA-1135 - Support scala 2.12

    Added support for scala 2.12

    Author: Maxim Logvinenko <mlogvinenko@gmail.com>

    Reviewers: Jagadish <jagadish@apache.org>,Prateek Maheshwari <prateekm@linkedin.com>

    Closes #82 from metamx:scala-2.12

20 months agoSAMZA-1175 - Removing CoordinationService from JobCoordinatorFactory interface
Navina Ramesh [Wed, 29 Mar 2017 23:25:16 +0000 (16:25 -0700)] 
SAMZA-1175 - Removing CoordinationService from JobCoordinatorFactory interface

Removing CoordinationService from JobCoordinatorFactory interface

Author: navina <navina@apache.org>

Reviewers: Xinyu Liu <xinyuliu.us@apache.org>,Boris Shkolnik <boryas@apache.org>

Closes #102 from navina/SAMZA-1175

20 months agoSAMZA-1161: Adding metrics into LocalStoreMonitor.
Shanthoosh Venkataraman [Wed, 29 Mar 2017 21:07:34 +0000 (14:07 -0700)] 
SAMZA-1161: Adding metrics into LocalStoreMonitor.

Rocksdb LocalStoreMonitor is responsible for clearing up unused local task partition stores. Metrics are required to understand the behavior of this monitor in production(especially when it's clearing up unused rocksdb local stores).

The following two metrics will be emitted from this monitor:
a) Total disk space cleared in bytes.
b) Total number of rocksdb task partition stores cleared.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #95 from shanthoosh/metrics_into_local_store_monitor

20 months agoSAMZA-1167: New streamId-specific configs do not override equivalent system-scoped...
Jacob Maes [Wed, 29 Mar 2017 20:57:26 +0000 (13:57 -0700)] 
SAMZA-1167: New streamId-specific configs do not override equivalent system-scoped configs

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>

Closes #101 from jmakes/samza-1167

20 months agoFix an import issue in unit test
Xinyu Liu [Wed, 29 Mar 2017 20:43:10 +0000 (13:43 -0700)] 
Fix an import issue in unit test

20 months agoSAMZA-1172: Fix for the topological sort to handle single-node loop
Xinyu Liu [Wed, 29 Mar 2017 17:49:25 +0000 (10:49 -0700)] 
SAMZA-1172: Fix for the topological sort to handle single-node loop

In the processor graph, the topological sort missed adding to the visited set during graph traversal. This caused wrong graph being generated for single-node loop. This is fixed in the patch.

Also fixed the maxPartition method not handling empty collection correctly.

Added a few new unit tests for these. Also adjust the timing of previous async commit unit tests so it can run more reliably. Long term wise we need to fix the timer inside the AsyncRunLoop tests.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jacob Maes <jmakes@apache.org>

Closes #100 from xinyuiscool/SAMZA-1172

20 months agoSAMZA-1151 - Coordination Service
Boris Shkolnik [Wed, 29 Mar 2017 17:44:49 +0000 (10:44 -0700)] 
SAMZA-1151 - Coordination Service

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@bshkolni-ld1.linkedin.biz>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Xinyu Liu <xinyuliu.us@apache.org>, Navina Ramesh <navina@apache.org>

Closes #91 from sborya/CoordinationService