samza.git
3 months agoSAMZA-1387: Unable to Start Samza App Because Regex Check
Jacob Maes [Fri, 11 Aug 2017 16:28:20 +0000 (09:28 -0700)] 
SAMZA-1387: Unable to Start Samza App Because Regex Check

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Fred Ji <fredji97@yahoo.com>

Closes #266 from jmakes/samza-1387

3 months agoSAMZA-1384: Race condition with async commit affects checkpoint correctness
Jacob Maes [Thu, 10 Aug 2017 22:44:42 +0000 (15:44 -0700)] 
SAMZA-1384: Race condition with async commit affects checkpoint correctness

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #263 from jmakes/samza-1384

3 months agoSAMZA-1374: Implement Leader Election using Lease Blob in Azure
Pawas Chhokra [Wed, 9 Aug 2017 01:26:50 +0000 (18:26 -0700)] 
SAMZA-1374: Implement Leader Election using Lease Blob in Azure

PR 1: AzureClient + AzureConfig
PR 2: LeaseBlobManager
PR 3: BlobUtils + JobModelBundle
PR 4: TableUtils + ProcessorEntity
**PR 5: AzureLeaderElector** (current PR)

Author: PawasChhokra <Jaimatadi1$>
Author: PawasChhokra <pawas2703@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>, Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Closes #259 from PawasChhokra/LeaderElection

3 months agoSAMZA-1381: Create Utility Class for interacting with Azure Table Storage
Pawas Chhokra [Tue, 8 Aug 2017 00:17:33 +0000 (17:17 -0700)] 
SAMZA-1381: Create Utility Class for interacting with Azure Table Storage

PR 1: AzureClient + AzureConfig
PR 2: LeaseBlobManager
PR 3: BlobUtils + JobModelBundle
**PR 4: TableUtils + ProcessorEntity** (Current PR)

Author: PawasChhokra <Jaimatadi1$>
Author: PawasChhokra <pawas2703@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #258 from PawasChhokra/TableUtils

3 months agoSAMZA-1380: Create Utility Class for interacting with Azure Blob Storage
Pawas Chhokra [Mon, 7 Aug 2017 23:28:05 +0000 (16:28 -0700)] 
SAMZA-1380: Create Utility Class for interacting with Azure Blob Storage

PR 1: AzureClient + AzureConfig
PR 2: LeaseBlobManager
**PR 3: BlobUtils + JobModelBundle** (current PR)

Author: PawasChhokra <Jaimatadi1$>
Author: PawasChhokra <pawas2703@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #257 from PawasChhokra/BlobUtils

3 months agoSAMZA-1376: Create a leasing utility class for blobs in Azure
Pawas Chhokra [Mon, 7 Aug 2017 21:15:23 +0000 (14:15 -0700)] 
SAMZA-1376: Create a leasing utility class for blobs in Azure

navina
PR 1: AzureClient + AzureConfig
**PR 2: LeaseBlobManager** (current PR)

Author: PawasChhokra <Jaimatadi1$>
Author: PawasChhokra <pawas2703@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>, Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Closes #256 from PawasChhokra/LeaseUtils

3 months agoReenable LocalZkApplicationRunner tests.
Shanthoosh Venkataraman [Fri, 4 Aug 2017 18:49:29 +0000 (11:49 -0700)] 
Reenable LocalZkApplicationRunner tests.

Add back commented out ZkLocalApplicationRunner tests(Was dependent upon error propagation from SamzaContainer to LocalApplicationRunner).

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #250 from shanthoosh/fix_broken_tests

3 months agoSamza-1379: Create Azure Client
Pawas Chhokra [Fri, 4 Aug 2017 01:58:10 +0000 (18:58 -0700)] 
Samza-1379: Create Azure Client

navina
**PR 1: AzureClient + AzureConfig** (current PR)

Author: PawasChhokra <Jaimatadi1$>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #254 from PawasChhokra/AzureStorageClient

3 months agoFix flaky, slow integration tests in TestZkStreamProcessor and TestZkStreamProcessorS...
Shanthoosh Venkataraman [Thu, 3 Aug 2017 22:48:23 +0000 (15:48 -0700)] 
Fix flaky, slow integration tests in TestZkStreamProcessor and TestZkStreamProcessorSession

Fix flaky and slow integration tests in TestZkStreamProcessor and TestZkStreamProcessorSession
Reason for failures:

There’re three configurable wait times in rebalancing phase in samza standalone before consensus is acheived and processing resumes with updated jobModel.

* debounceTime (Specified by `job.debounce.time.ms`. Upon processor change, leader waits for this interval before generating jobModel expecting stabilization in processors group(new arrival, deletion etc)).
* taskShutdownMs (Specified by `task.shutdown.ms`. Wait time for SamzaContainer shutdown in StreamProcessor).
* barrierWaitTimeOutMs (Specified by `job.coordinator.zk.consensus.timeout.ms`. Wait time for all processors in the group to join the barrier after creation).

Above wait times affects rebalancing phase duration. All these wait time have defaults in order of 40-60 seconds and not set to low values.

Flaky tests expects processors to come back up after rebalancing phase and drain message sources(Accomplished by checking a latch.count. RemoteApplicationRunner integration tests does exact same thing by checking if kafka input queue is drained directly with similar logic).

In worst case rebalancing phases can last upto 3-4 minutes(Making these tests sometime take 10 minutes at worst case).

Change:

Set all the above timeouts to 2 seconds(Sufficient for tests and verified by local build).

Benefits:

* Faster build time(Average runtime of these individual tests were reduced from 1m56s to 14s)
* More predicability in assertions(Didn’t fail even once in 30-40 attempts locally).

NOTE: If this doesn’t fix TestZkStreamProcessor and TestZkStreamProcessorSession,
longer term fix should be to use message markers in input source and
shutdown taskCoordinator upon receiving them from TaskImpl(Or use
bounded collection based pluggable InMemorySystemConsumer/InMemorySystemProducer).

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Bharath Kumarasubramanian <codin.martial@gmail.com>, Navina Ramesh <navina@apache.org>

Closes #260 from shanthoosh/FIX_ZK_PROCESSOR_FLAKY_TESTS

3 months agoSAMZA-1365: Calling zkClient.close from zkWatch impl blocks indefinitely.
Shanthoosh Venkataraman [Thu, 3 Aug 2017 21:32:09 +0000 (14:32 -0700)] 
SAMZA-1365: Calling zkClient.close from zkWatch impl blocks indefinitely.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #253 from shanthoosh/SAMZA-1365

3 months agoSAMZA-1366: ScriptRunner should allow callers to control the child pr…
Jacob Maes [Wed, 2 Aug 2017 16:22:51 +0000 (09:22 -0700)] 
SAMZA-1366: ScriptRunner should allow callers to control the child pr…

…ocess environment.

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>, Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Closes #245 from jmakes/script-runner-improvements

3 months agoSAMZA-1361; OperatorImplGraph is using wrong keys to store/retrieve OperatorImpl...
Prateek Maheshwari [Thu, 27 Jul 2017 22:13:33 +0000 (15:13 -0700)] 
SAMZA-1361; OperatorImplGraph is using wrong keys to store/retrieve OperatorImpl in the map

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apachce.org>

Closes #248 from prateekm/operatorimpl-key and squashes the following commits:

e733e9d3 [Prateek Maheshwari] Dummy commit to trigger jenkins build.
5a16f162 [Prateek Maheshwari] Updated with Yi's feedback.
8eb2c5df [Prateek Maheshwari] SAMZA-1361: OperatorImplGraph is using wrong keys to store/retrieve OperatorImpl in the map

3 months agoSAMZA-1370; Memory leak in CachedStore when using ByteBufferSerde as key Serde
Prateek Maheshwari [Thu, 27 Jul 2017 22:10:43 +0000 (15:10 -0700)] 
SAMZA-1370; Memory leak in CachedStore when using ByteBufferSerde as key Serde

ByteBufferSerde uses relative bulk get to serialize the provided ByteBuffer which changes its internal position. ByteBuffer's `equals` and `hashCode` depend upon its remaining elements, i.e. on its position. This means that when using ByteBuffers as keys in the CachedStore, flushing cache contents to the underlying store changes their hashCode. Since the hashCode for the key no longer matches the one used when inserting it into the map, the LinkedHashMap cannot correctly evict or remove these entries, leading to a memory leak.

Changing ByteBufferSerde to duplicate the provided ByteBuffer before copying should fix this issue. Prefer this over using absolute gets since there's no bulk absolute get API for ByteBuffer.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #251 from prateekm/bytebufferserde

3 months agoSamza-1364: Handle ZKExceptions in zkCoordinationUtils.reset.
Shanthoosh Venkataraman [Wed, 26 Jul 2017 00:29:08 +0000 (17:29 -0700)] 
Samza-1364: Handle ZKExceptions in zkCoordinationUtils.reset.

In some cases LocalAppRunner.waitForFinish indefinitely blocks after LocalApplicationRunner.kill. Last step in LocalAppRunner.kill(streamApp) is zkClient.close()[zkClient belongs to ZkCoordinationService].

ApplicationRunner.kill triggers listeners chain and in final listener zkClient.close throws ZkInterruptedException(RuntimeException) & it's swallowed in listeners preventing shutdownLatch update in LocalApplicationRunner(required for proper shutdown).

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Boris Shkolnik <boryas@apache.org>, Navina Ramesh <navina@apache.org>

Closes #246 from shanthoosh/SAMZA-1364

3 months agoFix build failures after master merge
Shanthoosh Venkataraman [Wed, 26 Jul 2017 00:04:53 +0000 (17:04 -0700)] 
Fix build failures after master merge

Changes
* Fix checkstyle errors from #243
* Fix failure after bad merge in #244

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #252 from shanthoosh/fix_NPE_after_master_merge

3 months agoSAMZA-1282: Spinning up more containers than number of tasks.
Shanthoosh Venkataraman [Tue, 25 Jul 2017 22:32:22 +0000 (15:32 -0700)] 
SAMZA-1282: Spinning up more containers than number of tasks.

Changes

* Stop streamProcessor in onNewJobModelAvailable eventHandler(instead of onNewJobModelConfirmed
  eventHandler) when it's not part of the group and prevent it from joining the barrier.
* When numContainerIds > numTaskModels, generate JobModel by choosing lexicographically
  least `x` containerIds(where x = numTaskModels).
* Added unit and integration tests in appropriate classes to verify the expected behavior.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Boris Shkolnik <boryas@apache.org>, Navina Ramesh <navina@apache.org>

Closes #244 from shanthoosh/more_processor_than_tasks

3 months agoSAMZA-1359; Handle phantom container notifications cleanly during an RM fail-over
Jagadish Venkatraman [Tue, 25 Jul 2017 18:18:14 +0000 (11:18 -0700)] 
SAMZA-1359; Handle phantom container notifications cleanly during an RM fail-over

1. Improved our container handling logic to be resilient to phantom notifications.
2. Added a new metric to Samza's ContainerProcessManager module that tracks the number of such invalid notifications.
3. Add a couple of tests that simulate this exact scenario above that we encountered during the cluster upgrade. (container starts -> container fails -> legitimate notification for the failure - container re-start -> RM fail-over -> phantom notification with a different exit code)
4. As an aside, there are a whole bunch of tests in ContainerProcessManager that rely on Thread.sleep to ensure that threads get to run in a certain order. Removed this non-determinism and made them predictable.

Author: Jagadish Venkatraman <jvenkatr@jvenkatr-mn2.linkedin.biz>

Reviewers: Jake Maes <jmaes@linkedin.com>

Closes #243 from vjagadish1989/am-bug

3 months agoFix log messages from StreamProcessor(onJobModelExpired event).
Shanthoosh Venkataraman [Tue, 25 Jul 2017 17:35:31 +0000 (10:35 -0700)] 
Fix log messages from StreamProcessor(onJobModelExpired event).

Log messages published in onJobModelExpired event have `processorId` as null. `processorId` is cached as final var in jobCoordinatorListener method. JLS for final fields/variables states that they're initialized before the constructor. This sets local final variable copy as null(since it relies upon value of instance variable to be set in constructor).
Changes
* Use processorId directly in `createCoordinatorListener` method.
* Remove StreamProcessor.toString since it has no usages.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #249 from shanthoosh/fix_logging_in_stream_processor

4 months agoSAMZA-1368; make sure new job model will be generated in case of barrier timeout.
Boris Shkolnik [Fri, 21 Jul 2017 22:32:58 +0000 (15:32 -0700)] 
SAMZA-1368; make sure new job model will be generated in case of barrier timeout.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Shanthoosh V <svenkata@linkedin.com>

Closes #247 from sborya/onBarrierTimeout1

4 months agoSAMZA-1165; cleanup old zk versions.
Boris Shkolnik [Fri, 21 Jul 2017 20:43:18 +0000 (13:43 -0700)] 
SAMZA-1165; cleanup old zk versions.

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Navina <navina@apache.org>, Shanthoosh V<svenkata@linkedin.com>

Closes #239 from sborya/zkCleanUpBarrier1

4 months agoSAMZA-1336. Session disconnect propagation.
Boris Shkolnik [Mon, 17 Jul 2017 22:57:29 +0000 (15:57 -0700)] 
SAMZA-1336. Session disconnect propagation.

If ZK doesn't receive any communication from a zkClient (including heartbeats), it closes the session with the client. It removes all the ephemeral nodes associated with the client. That's why we need to restore all these nodes - need to re-register.
We are using ZkClient library to connect to zookeeper. This library allows us to get notification when the session is closed and when a new session is created. So when the new session is created we reset all session related state and re-register.
One weird feature of the library/zookeeper is that when a new session is established, it is still possible to receive old notifications. To avoid this we introduced generation number which we pass into each callback. And if the generation number has changed when the callback was invoked, we will ignore this callback.

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@bshkolni-ld1.linkedin.biz>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>, Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Closes #229 from sborya/SessionFailReregister

4 months agoSAMZA-1304: Handling duplicate stream processor registration.
Shanthoosh Venkataraman [Thu, 13 Jul 2017 23:19:56 +0000 (16:19 -0700)] 
SAMZA-1304: Handling duplicate stream processor registration.

When a stream processor registers with same processorId as already existing
processor in processor group, it's registration should fail.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>, Jagadish V <jvenkatr@linkedin.com>

Closes #240 from shanthoosh/standalone_duplicate_processor_fix

4 months agoSAMZA-1324: Fix NullPointerException in ZkUtils api's.
Shanthoosh Venkataraman [Wed, 12 Jul 2017 18:13:09 +0000 (11:13 -0700)] 
SAMZA-1324: Fix NullPointerException in ZkUtils api's.

Problem:
Read/Write api methods in ZkUtils updates counters/timers in `metrics` field. In a ZkUtils constructor this fields is not initialized properly. Java default for uninitialized field is null resulting in NPE.

Fix:
Initialize private fields of ZkUtils class with appropriate defaults.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #235 from shanthoosh/fix_zkutils_api

4 months agoSAMZA-1358: fix the bug in validating task.class empty string when app.class is confi...
Yi Pan (Data Infrastructure) [Tue, 11 Jul 2017 17:55:51 +0000 (10:55 -0700)] 
SAMZA-1358: fix the bug in validating task.class empty string when app.class is configured

…p.class is configured

Another bug due to scala/java differences.

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #242 from nickpan47/SAMZA-1358

4 months agoSAMZA-1296 : Stand alone integration tests.
Shanthoosh Venkataraman [Wed, 28 Jun 2017 18:12:15 +0000 (11:12 -0700)] 
SAMZA-1296 : Stand alone integration tests.

Changes
Brings up a test bed that contains embedded kafka broker and zookeeper to test the following scenarios.

A) Rolling upgrade of stream processors.
B) Reelection of leader upon failures.
C) Registering multiple processors with same processor id.
D) Zookeeper failure before job model regeneration upon leader death should kill all running stream applications.

NOTE:
Some tests are commented out since zookeeper exceptions are swallowed in ZKJobCoordinator/ZKUtils.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>, Jagadish V <jvenkatr@linkedin.com>

Closes #196 from shanthoosh/standalone_happy_integration_tests

4 months agoSAMZA-1340 - StreamProcessor does not propagate container failures from StreamTask
Navina Ramesh [Wed, 28 Jun 2017 18:09:27 +0000 (11:09 -0700)] 
SAMZA-1340 - StreamProcessor does not propagate container failures from StreamTask

Storing the exception seen from the container in the `SamzaContainerListener#onFailure(Throwable)` in the StreamProcessor.
`JobCoordinator#stop` callback inspects this stored exception and invokes the correct callback for StreamProcessorLifecycleListener.
It is pretty difficult to add all test cases. Suggestion welcome for improving code/testing.

Author: Navina Ramesh <navina@apache.org>

Reviewers: Chris Pettitt <cpettitt@linkedin.com>, Boris Shkolnik <boryas@apache.org>

Closes #230 from navina/LISAMZA-5272

4 months agoSAMZA-1347: GroupByContainerIds NPE if containerIds list is null
Jacob Maes [Tue, 27 Jun 2017 18:42:51 +0000 (11:42 -0700)] 
SAMZA-1347: GroupByContainerIds NPE if containerIds list is null

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Boris Shkolnik <boryas@apache.org>

Closes #233 from jmakes/samza-1347

4 months agoAdd a basic .travis.yml file to run on Travis.
Nacho Solis [Tue, 27 Jun 2017 00:25:57 +0000 (17:25 -0700)] 
Add a basic .travis.yml file to run on Travis.

Enable gradle caching
Use Trusty as a platform
Set JDK to 8

Author: Nacho Solis <nsolis@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #234 from isolis/add-travis-yml

4 months agoSAMZA-1324 : Add a metricsreporter lifecycle for JobCoordinator component of StreamPr...
PawasChhokra [Mon, 26 Jun 2017 23:35:26 +0000 (16:35 -0700)] 
SAMZA-1324 : Add a metricsreporter lifecycle for JobCoordinator component of StreamProcessor

Added a metrics class for ZK based job coordinator that reports a few metrics.

Author: PawasChhokra <Jaimatadi1$>
Author: Pawas Chhokra <pchhokra@pchhokra-mn1.linkedin.biz>
Author: PawasChhokra <pawas2703@gmail.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #223 from PawasChhokra/ZkJobCoordinatorMetrics

4 months agoSAMZA-1346: GroupByContainerCount.balance() should guard against null…
Jacob Maes [Mon, 26 Jun 2017 20:43:33 +0000 (13:43 -0700)] 
SAMZA-1346: GroupByContainerCount.balance() should guard against null…

… LocalityManager

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Chris Pettitt <cpettitt@linkedin.com>

Closes #232 from jmakes/samza-1346

4 months agoSAMZA-1337: Use StreamTask with the LocalApplicationRunner
Boris Shkolnik [Mon, 26 Jun 2017 19:25:48 +0000 (12:25 -0700)] 
SAMZA-1337: Use StreamTask with the LocalApplicationRunner

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>, Xinyu Liu <xiliu@apache.org>, Bharath Kumarasubramanian <codin.martial@gmail.com>

Closes #231 from sborya/LocalAppRunnerWithStreamTask

5 months agoSAMZA-1334: fix pre-condition for ContainerAllocator to work properly
Yi Pan (Data Infrastructure) [Tue, 20 Jun 2017 15:35:39 +0000 (08:35 -0700)] 
SAMZA-1334: fix pre-condition for ContainerAllocator to work properly

We have observed issues when the LocalityManager reports the container locality mapping while the host-affinity is disabled in ContainerAllocator, in which the ContainerAllocator failed to release extra containers.

Hence, fix is in the form of make sure the pre-condition is met for the ContainerAllocator w/o host-affinity: the localityMap from the JobModel should contain no preferred host info.

Author: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>

Reviewers: Jagadish <jagadish1989@gmail.com>

Closes #228 from nickpan47/SAMZA-1334 and squashes the following commits:

ad3320f [Yi Pan (Data Infrastructure)] SAMZA-1334: fix the pre-conditions for ContainerAllocator to work properly. Make sure JobModel is generated w/o LocalityManager if host-affinity is disabled
f76fff1 [Yi Pan (Data Infrastructure)] WIP: SAMZA-1334 fix

5 months agoSAMZA-1335: Improve logging for LocalStoreMonitor
Jacob Maes [Thu, 15 Jun 2017 20:40:43 +0000 (13:40 -0700)] 
SAMZA-1335: Improve logging for LocalStoreMonitor

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #226 from jmakes/samza-1335

5 months agoUpdated test build and README versions to next version
Prateek Maheshwari [Wed, 14 Jun 2017 22:18:45 +0000 (15:18 -0700)] 
Updated test build and README versions to next version

Updated the versions used in smoke tests and the README for overnight tests to point to the next release. Using 0.13.1 instead of 0.13.1-SNAPSHOT here since these tests are usually run against the release branch when release testing, where the version doesn't have -SNAPSHOT in it.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #224 from prateekm/version-update

5 months agoSAMZA-1312: Add Control Messages and Intermediate Stream Serde
Xinyu Liu [Tue, 13 Jun 2017 16:46:02 +0000 (09:46 -0700)] 
SAMZA-1312: Add Control Messages and Intermediate Stream Serde

In this patch, we add the control message types which includes:
* EndOfStreamMessage
* WatermarkMessage

To support in-band data and control messages, we provide a wrapper serde (IntermediateMessageSerde) to serialize/deserialize data/control messages based on message type byte (first byte in the intermediate stream message). The format of the message is defined in SAMZA-1312. The patch integrates this serde with SerdeManager.

Tested in example jobs deployed locally and works as expected.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish V <jagadish@apache.org>

Closes #207 from xinyuiscool/SAMZA-1312

5 months agoSAMZA-1330: Stand alone feature preview, known limitation.
Boris Shkolnik [Mon, 12 Jun 2017 21:08:45 +0000 (14:08 -0700)] 
SAMZA-1330: Stand alone feature preview, known limitation.

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #220 from sborya/StandAloneFeaturePreview

5 months agoSAMZA-1327: fail if namespace specified in the connection string does not exist
Boris Shkolnik [Mon, 12 Jun 2017 17:51:27 +0000 (10:51 -0700)] 
SAMZA-1327: fail if namespace specified in the connection string does not exist

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #218 from sborya/zkNameSpaceFail

5 months agoSAMZA-1331; Optimize Snapshot class constructor
Maksim Logvinenko [Fri, 9 Jun 2017 17:33:08 +0000 (10:33 -0700)] 
SAMZA-1331; Optimize Snapshot class constructor

In some of our workloads (where we need to gather samza metrics five times per minute) `SlidingTimeWindowReservoir.getSnapshot()` method takes up to 10% of processor time.

Almost all of `getSnapshot` time is taken by Collections.sort method. So, the complexity of Snapshot constructor is O(NlogN) + iteration through passed values.

This ticket asks to improve the performance of Snapshot constructor but keep the performance of all other methods at least on the same level.

Author: Maksim Logvinenko <mlogvinenko@gmail.com>

Reviewers: Jagadish V<jagadish@apache.org>

Closes #221 from logarithm/fix-snapshot

5 months agoUpdate release instructions to publish artifacts for different scala versions
vjagadish1989 [Fri, 9 Jun 2017 17:32:09 +0000 (10:32 -0700)] 
Update release instructions to publish artifacts for different scala versions

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Jake Maes<jmakes@apache.org>, Prateek Maheshwari<pmaheshw@linkedin.com>

Closes #222 from vjagadish1989/release-instr

5 months agoSAMZA-1329: Switch SamzaTaskProxy to use LocalityManager.
Shanthoosh Venkataraman [Thu, 8 Jun 2017 00:57:54 +0000 (17:57 -0700)] 
SAMZA-1329: Switch SamzaTaskProxy to use LocalityManager.

TasksResource(/tasks/) in samza-rest is relying on `JobModelManager.readJobModel` to get jobmodel from (CoordinatorStream, jobConfig). This created binary dependencies of systemstreams defined in task.inputs of job config into samza-rest. Managing those dependencies is hard and unnecessary.

This PR updates `TasksResource` to use `LocalityManager` to read task to container locality.
In the future, this api will read jobmodel from `SamzaMetadatasystem` store.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #219 from shanthoosh/fix-2

5 months agoSAMZA-1326: Revert to use getGlobalAppId for zk path
Boris Shkolnik [Wed, 7 Jun 2017 21:23:13 +0000 (14:23 -0700)] 
SAMZA-1326: Revert to use getGlobalAppId for zk path

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #216 from sborya/getGlobalId

5 months agoSAMZA-1221, SAMZA-1101: Internal cleanup for High-Level API implementation.
Prateek Maheshwari [Wed, 7 Jun 2017 19:42:11 +0000 (12:42 -0700)] 
SAMZA-1221, SAMZA-1101: Internal cleanup for High-Level API implementation.

SAMZA-1221: Separated the OperatorSpec and MessageStream DAGs so that they're now duals of each other. Users interact with and construct the MessageStream DAG; we create and use the OperatorSpec DAG internally.
Moved common OperatorSpec functionality (getId, getOpCode, getOpName etc.) to the OperatorSpec abstract base class.
Added a new JoinOperatorSpec and PartialJoinOperatorImpls which are created from JoinOperatorSpec in OperatorGraphImpl.
Added a new InputOperatorSpec and InputOperatorImpl (previously RootOperatorImpl). InputOperatorSpec is created when StreamGraph#getInputStream is called.
SAMZA-1101: Added a new OutputOperatorSpec and OutputOperatorImpl for partitionBy and sendTo. These are Separate from SinkOperatorSpec for and SinkOperatorImpl for sink. We don't need to create a sinkFn for partitionBy and sendTo anymore.
Updated most unit tests to use the new classes and avoid reflection.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish V <vjagadish1989@apache.org>

Closes #194 from prateekm/internal-cleanup

5 months agoWebsite updates for 0.13.0 release
Jacob Maes [Wed, 7 Jun 2017 17:50:20 +0000 (10:50 -0700)] 
Website updates for 0.13.0 release

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jagadish <jvenkatr@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #215 from jmakes/samza-0.13.0-website

5 months agoUpdate docs for ClosableFunction
vjagadish1989 [Mon, 5 Jun 2017 17:32:20 +0000 (10:32 -0700)] 
Update docs for ClosableFunction

5 months agoSAMZA-1271; Guarantee predictable, deterministic order for operator initialization...
vjagadish1989 [Mon, 5 Jun 2017 17:27:23 +0000 (10:27 -0700)] 
SAMZA-1271; Guarantee predictable, deterministic order for operator initialization and finalization

Currently, the order of initialization of operators in the Samza high level API is not deterministic. The non-determinism arises from two primary causes:

- No fixed order of iteration for all subscribed `OperatorSpec`s for a given `MessageStream`
- No fixed order of iteration for all the `OperatorImpl`s in the `OperatorImplGraph`

We aim to provide the following 2 guarantees in this patch:
For any 2 operators A, B in the graph, if B consumes the output of A:
- A is initialized before B is initialized
- A is finalized only after B is finalized

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<pmaheshw@linkedin.com>

Closes #211 from vjagadish1989/deterministic_order

5 months agoSAMZA-1317: Changelog validation error for topics with period in the
Jacob Maes [Sat, 3 Jun 2017 03:42:14 +0000 (20:42 -0700)] 
SAMZA-1317: Changelog validation error for topics with period in the

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #213 from jmakes/samza-1317-migration-fix

5 months agoSAMZA-1317: Changelog topic configuration should accept streams.[streamId] configurations
Jacob Maes [Fri, 2 Jun 2017 21:53:31 +0000 (14:53 -0700)] 
SAMZA-1317: Changelog topic configuration should accept streams.[streamId] configurations

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #212 from jmakes/samza-1317-migration-fix

5 months agoUpdate RELEASE documentation to include steps related to integration tests
vjagadish1989 [Fri, 2 Jun 2017 02:13:48 +0000 (19:13 -0700)] 
Update RELEASE documentation to include steps related to integration tests

5 months agoMake run-app.sh executable
Xinyu Liu [Thu, 1 Jun 2017 21:55:17 +0000 (14:55 -0700)] 
Make run-app.sh executable

5 months agoSAMZA-1305. SAMZA-1306. Unit test - zk unavailable.
Boris Shkolnik [Thu, 1 Jun 2017 18:51:22 +0000 (11:51 -0700)] 
SAMZA-1305. SAMZA-1306. Unit test - zk unavailable.

Author: Boris Shkolnik <boryas@apache.org>
Author: Boris Shkolnik <bshkolni@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #198 from sborya/unitTestZkUnavailable2

5 months agoAdding gpg key for Jake Maes
Jacob Maes [Thu, 1 Jun 2017 15:38:36 +0000 (08:38 -0700)] 
Adding gpg key for Jake Maes

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #209 from jmakes/gpg-key

5 months agoSAMZA-1264; Make Operator Functions Closable
vjagadish1989 [Thu, 1 Jun 2017 05:20:14 +0000 (22:20 -0700)] 
SAMZA-1264; Make Operator Functions Closable

- Added `close()` to the lifecycle of `OperatorImpl`s, and all `Function`s.
- Added unit tests to verify calls to `close()`

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari<pmaheshw@linkedin.com>

Closes #208 from vjagadish1989/operator_functions

5 months agoSAMZA-1234: Documentation for 0.13.0 release
Jacob Maes [Thu, 1 Jun 2017 00:23:30 +0000 (17:23 -0700)] 
SAMZA-1234: Documentation for 0.13.0 release

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Yi Pan (Data Infrastructure) <nickpan47@gmail.com>, Boris Shkolnik <boryas@apache.org>

Closes #204 from jmakes/samza-1236-tutorial-1

5 months agoSAMZA-1309; Debounce time config
Boris Shkolnik [Fri, 26 May 2017 23:31:45 +0000 (16:31 -0700)] 
SAMZA-1309; Debounce time config

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #203 from sborya/DebounceConfig

5 months agoSAMZA-1313 : Rename StandaloneJobCoordinator to PassthroughJobCoordinator
Boris Shkolnik [Fri, 26 May 2017 22:45:52 +0000 (15:45 -0700)] 
SAMZA-1313 : Rename StandaloneJobCoordinator to PassthroughJobCoordinator

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #206 from sborya/RenameStandAloneGJobCoordinator

5 months agoUpdating committer page on samza.apache.org
Navina Ramesh [Thu, 25 May 2017 22:47:57 +0000 (15:47 -0700)] 
Updating committer page on samza.apache.org

Giving more screen time for currently active Apache Samza committers (no JIRA)

Author: Navina Ramesh <navina@apache.org>

Reviewers: Yi Pan <nickpan47@gmail.com>

Closes #201 from navina/reOrgCommitters

5 months agoSAMZA-1128 : Remove dependency of debounce timer from the CoordinationUtils
Navina Ramesh [Thu, 25 May 2017 00:46:12 +0000 (17:46 -0700)] 
SAMZA-1128 : Remove dependency of debounce timer from the CoordinationUtils

This patch addresses the following, apart from the main bug that barrier implementation uses a different scheduler that the Jobcoordinator's main thread.
* Removes CoordinationUtils#getBarrier, BarrierForVersionUpgrade interface
* Renamed ZkBarrierForVersionUpgrade to ZkBarrier and introduces a listener ZkBarrierListener
* Simplified the ZkBarrier class and its integration test

Author: Navina Ramesh <navina@apache.org>

Reviewers: Boris Shkolnik <boryas@apache.org>, Bharath Kumarasubramanian <bkumarasubramanian@linkedin.com>

Closes #195 from navina/SAMZA-1128

5 months agoSAMZA-1307 - Fix ZkKeyBuilder null checks for pathPrefix
Navina Ramesh [Wed, 24 May 2017 06:33:09 +0000 (23:33 -0700)] 
SAMZA-1307 - Fix ZkKeyBuilder null checks for pathPrefix

Author: Navina Ramesh <navina@apache.org>

Reviewers: Jagadish V <jagadish@apache.org>

Closes #202 from navina/SAMZA-1307

5 months agoImprove documentation for Resource Localization
vjagadish1989 [Tue, 23 May 2017 00:42:32 +0000 (17:42 -0700)] 
Improve documentation for Resource Localization

This is a follow-up to Fred Ji's original PR : https://github.com/apache/samza/pull/191 .

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #199 from vjagadish1989/doc-improvements

5 months agoSAMZA-1280; document for the general/universal resource localization in YARN
Fred Ji [Mon, 22 May 2017 22:09:32 +0000 (15:09 -0700)] 
SAMZA-1280; document for the general/universal resource localization in YARN

This PR added a MD for localizing resource in Samza on YARN  by configuring path, local.name, local.type and local.visibility, and also updated the configuration table and index table.

Author: Fred Ji <fji@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #191 from fredji97/resourceLocalizationDoc

6 months agoSAMZA-1298; Create zk path.
Boris Shkolnik [Mon, 22 May 2017 18:58:53 +0000 (11:58 -0700)] 
SAMZA-1298; Create zk path.

if ZK path contains extra path at the end, it needs to be created in ZK at first connect.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Jagadish <jagadish@apache.org>

Closes #197 from sborya/createZkPath

6 months agoSAMZA-1291: StandAlone config
Boris Shkolnik [Mon, 22 May 2017 16:36:12 +0000 (09:36 -0700)] 
SAMZA-1291: StandAlone config

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #192 from sborya/StandAloneConfig

6 months agoSAMZA-1193. re-enable the test TestZkBarrierForVersionUpgrade.testZkBarrierForVersion...
Boris Shkolnik [Sat, 20 May 2017 00:25:26 +0000 (17:25 -0700)] 
SAMZA-1193. re-enable the test TestZkBarrierForVersionUpgrade.testZkBarrierForVersionUpgrade

The code changed significantly. I've rerun the test multiple times both with gradle and intelij. It passed every time. I suggest we enable it back.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #181 from sborya/testZkBarrierForVersionUpgrade

6 months agoSAMZA-1258. Integration tests. Happy Path.
Boris Shkolnik [Fri, 19 May 2017 17:03:00 +0000 (10:03 -0700)] 
SAMZA-1258. Integration tests. Happy Path.

This is reincarnation of PR #157
It was not possible to merge/rebase that branch, so I had to create another one and reapply the changes.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #178 from sborya/unitTestHappyPath4

6 months agoSAMZA-1287: Updated instructions for running integration and stability tests.
Prateek Maheshwari [Mon, 15 May 2017 21:34:56 +0000 (14:34 -0700)] 
SAMZA-1287: Updated instructions for running integration and stability tests.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #187 from prateekm/integ-test-documentation

6 months agoSAMZA-1290: Fix javadoc for ApplicationRuner
Jacob Maes [Fri, 12 May 2017 22:22:47 +0000 (15:22 -0700)] 
SAMZA-1290: Fix javadoc for ApplicationRuner

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #190 from jmakes/samza-1290

6 months agoSAMZA-1289: Default id generator if not configured
Xinyu Liu [Fri, 12 May 2017 21:20:15 +0000 (14:20 -0700)] 
SAMZA-1289: Default id generator if not configured

Right now in standalone deployment we require the user to provide an id generator. Since most of the time the users can simply use the UUID generator for id generation, we should default it using that.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish V <vjagadish1989@apache.org>

Closes #189 from xinyuiscool/SAMZA-1289

6 months agoSAMZA-1288: Add null check for sink OutputStream
Xinyu Liu [Fri, 12 May 2017 16:49:04 +0000 (09:49 -0700)] 
SAMZA-1288: Add null check for sink OutputStream

The logic to generate json for Sink operator does not check whether the output stream is null. This causes null pointer exception.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apached.org>

Closes #188 from xinyuiscool/SAMZA-1288

6 months agoIncrease the plan graph size
Xinyu Liu [Thu, 11 May 2017 20:39:48 +0000 (13:39 -0700)] 
Increase the plan graph size

Increase the canvas size to a standard 24 inch resolution and also the scaling factor.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #186 from xinyuiscool/PLAN

6 months agoSAMZA-1174; Profiling state store performance
jmehar2 [Thu, 11 May 2017 19:56:31 +0000 (12:56 -0700)] 
SAMZA-1174; Profiling state store performance

This is s commit for [SAMZA-1174](https://issues.apache.org/jira/browse/SAMZA-1174). This commit involves gathering a log of operations (read, write, delete, etc.) happening on the state and publishing them into a kafka topic. It is names as "Access Log" behaving similar to changelog, but gathering log information.

Author: jmehar2 <jmehar2@illinois.edu>
Author: Jayasi Mehar <jayasi05@gmail.com>
Author: s-noghabi <abdolla2@illinois.edu>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jagadish V <jagadish@apache.org>

Closes #132 from s-noghabi/master

6 months agoSAMZA-1286: Close zk connection in ZkController.stop()
Boris Shkolnik [Thu, 11 May 2017 18:35:55 +0000 (11:35 -0700)] 
SAMZA-1286: Close zk connection in ZkController.stop()

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #185 from sborya/ZkControllerClose

6 months agoSAMZA-871; Heart-beat mechanism between JobCoordinator and all running containers release-0.13.0-rc0
Abhishek Shivanna [Wed, 10 May 2017 23:58:14 +0000 (16:58 -0700)] 
SAMZA-871; Heart-beat mechanism between JobCoordinator and all running containers

Author: Abhishek Shivanna <ashivanna@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>, Jagadish V <jagadish@apache.org>

Closes #163 from abhishekshivanna/master

6 months agoSAMZA-1279: Do not process onProcessorChange if list of children is null.
Boris Shkolnik [Wed, 10 May 2017 23:57:42 +0000 (16:57 -0700)] 
SAMZA-1279: Do not process onProcessorChange if list of children is null.

Author: Boris Shkolnik <boryas@apache.org>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #180 from sborya/NPEinCaseOfZKException

6 months agoSAMZA-1283: Expose the buffered-message-size metric
Xinyu Liu [Wed, 10 May 2017 23:50:49 +0000 (16:50 -0700)] 
SAMZA-1283: Expose the buffered-message-size metric

Regardless of whether we enable size limit for the consumer buffer, this metric helps to see what's the buffer size and make configuring size limit easier.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jagadish Venkatraman <vjagadish1989@gmail.com>

Closes #184 from xinyuiscool/SAMZA-1283

6 months agoSAMZA-1277: Add a static merge() operator that takes all streams to merge as input
Prateek Maheshwari [Wed, 10 May 2017 22:10:02 +0000 (15:10 -0700)] 
SAMZA-1277: Add a static merge() operator that takes all streams to merge as input

Also updated documentation for join and partitionBy.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #182 from prateekm/documentation-cleanup

6 months agoMinor: Re-word a couple of lines in javadoc with explicit notes about failures
vjagadish1989 [Wed, 10 May 2017 19:03:53 +0000 (12:03 -0700)] 
Minor: Re-word a couple of lines in javadoc with explicit notes about failures

6 months agoRevert "Disable flaky tests."
vjagadish1989 [Wed, 10 May 2017 18:41:17 +0000 (11:41 -0700)] 
Revert "Disable flaky tests."

This reverts commit df5aa6c4eeb9b89a40780b19e08bb6d96b97730b.

6 months agoDisable flaky tests.
Shanthoosh Venkataraman [Wed, 10 May 2017 16:27:57 +0000 (09:27 -0700)] 
Disable flaky tests.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh<navina@apache.org>

Closes #158 from shanthoosh/master

6 months agoSAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer
Navina Ramesh [Wed, 10 May 2017 01:24:09 +0000 (18:24 -0700)] 
SAMZA-1276 : Adding customReporters passed by StreamProcessor into SamzaContainer

SamzaContainer is not adding custom reporters along side class-loaded reporters from config. This was missed in [SAMZA-1080](https://issues.apache.org/jira/browse/SAMZA-1080).

Author: Navina Ramesh <navina@apache.org>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jagadish <jagadish@apache.org>

Closes #179 from navina/SAMZA-1276

6 months agoSAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset
Navina Ramesh [Wed, 10 May 2017 01:18:52 +0000 (18:18 -0700)] 
SAMZA-1272 : ZkCoordinationUtils deletes the entire Zk tree on reset

* ZkCoordinationUtils has a reset interface that deletes the entire Zk tree. This is not desirable.
* Also, fixed flakiness in unit test by unique barrier name in each of the unit tests. Otherwise, they share the same path on Zk and fail during concurrent test execution

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #173 from navina/SAMZA-1272

6 months agoSAMZA-1275: Kafka throws when users configure replication.factor for …
Jacob Maes [Tue, 9 May 2017 22:58:14 +0000 (15:58 -0700)] 
SAMZA-1275: Kafka throws when users configure replication.factor for …

…Kafka default stream

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Xinyu Liu <xiliu@linkedin.com>

Closes #176 from jmakes/samza-1275

6 months agoSAMZA-1268: More javadoc cleanup for public APIs for 0.13 release
Prateek Maheshwari [Tue, 9 May 2017 22:07:32 +0000 (15:07 -0700)] 
SAMZA-1268: More javadoc cleanup for public APIs for 0.13 release

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #175 from prateekm/documentation-cleanup

6 months agoSAMZA-1274; Update kafka-python and kafka broker version for integration tests
Prateek Maheshwari [Tue, 9 May 2017 17:41:50 +0000 (10:41 -0700)] 
SAMZA-1274; Update kafka-python and kafka broker version for integration tests

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes #174 from prateekm/integration-test-updates

6 months agoSAMZA-1273: Make StreamConfig.getStreamIds() public
Xinyu Liu [Tue, 9 May 2017 16:27:28 +0000 (09:27 -0700)] 
SAMZA-1273: Make StreamConfig.getStreamIds() public

Making StreamConfig.getStreamIds() public so config provider can scan through all the configured streams and expand some properties if needed.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #172 from xinyuiscool/SAMZA-1273

6 months agoSAMZA-1268: Javadoc cleanup for public APIs for 0.13 release
Prateek Maheshwari [Tue, 9 May 2017 02:24:03 +0000 (19:24 -0700)] 
SAMZA-1268: Javadoc cleanup for public APIs for 0.13 release

This PR cleans up javadocs for Fluent API classes in the samza-api module.
Also updates the TaskContext (existing) and ContextManager (new) interfaces to add support for setting an pass-through user-defined context.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #169 from prateekm/api-docs-cleanup

6 months agoSAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer
Navina Ramesh [Tue, 9 May 2017 00:58:55 +0000 (17:58 -0700)] 
SAMZA-1150 : Handling Error propagation between ZkJobCoordinator & DebounceTimer

This PR depends on PR #153
* Treats all errors in jobcoordinator as FATAL and shuts-down the streamprocessor
* [Bug] Fixed bug reported in SAMZA-1241
* Introduced a callback to be associated with the timer (same callback for every Runnable failure)

**TBD**: some more unit tests

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@apache.org>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #166 from navina/SAMZA-1150

6 months agoSAMZA-1206; Fix TestJMXServer.
Shanthoosh Venkataraman [Tue, 9 May 2017 00:22:14 +0000 (17:22 -0700)] 
SAMZA-1206; Fix TestJMXServer.

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh<navina@apache.org>

Closes #164 from shanthoosh/fix-test-jmx-server-1

6 months agoSAMZA-1196; Fix TestJmxReporter
Shanthoosh Venkataraman [Tue, 9 May 2017 00:19:36 +0000 (17:19 -0700)] 
SAMZA-1196; Fix TestJmxReporter

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #140 from shanthoosh/fix-test-jmx-reporter

6 months agoSAMZA-1232; Log configuration value in RunLoopFactory
Shanthoosh Venkataraman [Tue, 9 May 2017 00:18:38 +0000 (17:18 -0700)] 
SAMZA-1232; Log configuration value in RunLoopFactory

Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com>

Reviewers: Xinyu Liu <xinyu@apache.org>

Closes #137 from shanthoosh/adding_logging_into_asyncrunloop

6 months agoDisabled a few flaky tests and added corresponding tickets to fix.
Prateek Maheshwari [Mon, 8 May 2017 20:29:23 +0000 (13:29 -0700)] 
Disabled a few flaky tests and added corresponding tickets to fix.

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>

Closes #171 from prateekm/disable-flaky-test

6 months agoSAMZA-1267: ApplicationRunner#getLocalRunner returns null
Xinyu Liu [Mon, 8 May 2017 16:39:43 +0000 (09:39 -0700)] 
SAMZA-1267: ApplicationRunner#getLocalRunner returns null

Remove ApplicationRunner#getLocalRunner and clean up any usage examples.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Jake Maes <jmakes@apache.org>

Closes #168 from xinyuiscool/SAMZA-1267

6 months agoSAMZA-1155; Validate users configure window.ms when using the fluent API
vjagadish1989 [Sat, 6 May 2017 02:21:29 +0000 (19:21 -0700)] 
SAMZA-1155; Validate users configure window.ms when using the fluent API

We will compute triggering duration as follows:
- If user configures `task.window.ms` we will honor it as the triggering duration
- If not, we will use the `GCD(windowTriggerDurations, joinTTLs)` as the triggering duration.

Changes in this PR:
- Common Interface for all time based triggers
- Additional APIs in `StreamGraphImpl` to recursively traverse all `OperatorSpec`s
- Recursive computation of `triggerInterval` for each `WindowOperatorSpec`
- Tests for all the above

Author: vjagadish1989 <jvenkatr@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Jacob Maes <jmaes@linkedin.com>, Xinyu Liu <xinyu@apache.org>

Closes #160 from vjagadish1989/samza-1155

6 months agoSAMZA-1266: Unable to use MetricsSnapshotReporterFactory with fluent API
Jacob Maes [Fri, 5 May 2017 22:48:32 +0000 (15:48 -0700)] 
SAMZA-1266: Unable to use MetricsSnapshotReporterFactory with fluent API

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Prateek Maheshwari <pmaheshw@linkedin.com>, Navina Ramesh <navina@apache.org>

Closes #167 from jmakes/samza-1266

6 months agoSAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector & ZkController
Navina Ramesh [Fri, 5 May 2017 20:50:08 +0000 (13:50 -0700)] 
SAMZA-1251 - Remove DebounceTimer dependency from ZkLeaderElector & ZkController

Addresses the following:
* Makes LeaderElectionListener to be explicitly registered by the caller
* Removes debouncetimer dependency from ZkLeaderElector implementation
* [Bug] onBecomeLeader was scheduling a task in timer under "OnBecomeLeader", when it should actually be the same as "OnProcessorChange". Otherwise, it will not cancel when there is a new OnProcessorChange event.
* [Transient Test Failure] `TestScheduleAfterDebounceTime` tests were relying on timing controlled by sleep. Fixed it by using latch

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Boris Shkolnik <boryas@apache.org>

Closes #153 from navina/SAMZA-1251

6 months agoSAMZA-1228 : StreamProcessor should stop JmxServer
Navina Ramesh [Fri, 5 May 2017 20:22:35 +0000 (13:22 -0700)] 
SAMZA-1228 : StreamProcessor should stop JmxServer

This is not the solution posted in SAMZA-1228. For now, we are moving jmxserver lifecycle to be within the container. Ideally, it should be within the Streamprocessor so that the job coordinator can also be associated with the same instance.

Author: Navina Ramesh <navina@apache.org>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #162 from navina/SAMZA-1228

6 months agoSAMZA-1222: Clean up LocalApplicationRunner
Xinyu Liu [Thu, 4 May 2017 22:17:56 +0000 (15:17 -0700)] 
SAMZA-1222: Clean up LocalApplicationRunner

Clean up the LocalApplicationRunner based on the further feedback. The changes include the following:
1. Remove the processorId from the JobCoordinatorFactory/JobCoordinator interfaces
2. LocalApplicationRunner.run() is non-blocking. Add LocalApplicationRunner.waitForFinish() for blocking for completion
3. Remove the config for CooridnatorServiceFactory, and now the CoordinatorService is created based on the type of JobCoordinator.
4. Clean up the StreamProcessor life cycle listener logic inside LocalApplicationRunner.

Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>

Reviewers: Navina Ramesh <navina@apache.org>

Closes #135 from xinyuiscool/SAMZA-1222

6 months agoSAMZA-1263: Samza Fluent: NPE is streams.id.samza.system is missing
Jacob Maes [Thu, 4 May 2017 20:57:50 +0000 (13:57 -0700)] 
SAMZA-1263: Samza Fluent: NPE is streams.id.samza.system is missing

Author: Jacob Maes <jmaes@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #165 from jmakes/samza-1263

6 months agoSAMZA-1247: MessageStreamImpl#merge shouldn't mutate input collection
Prateek Maheshwari [Thu, 4 May 2017 19:28:58 +0000 (12:28 -0700)] 
SAMZA-1247: MessageStreamImpl#merge shouldn't mutate input collection

Also fixes
SAMZA-1253: MessageStream.merge operator broken for nested types

Author: Prateek Maheshwari <pmaheshw@linkedin.com>

Reviewers: Jacob Maes <jmaes@linkedin.com>, Jagadish <jvenkatr@linkedin.com>

Closes #159 from prateekm/merge-fixes

6 months agoSAMZA-1224 : Revert job coordinator factory config to the old format
Navina Ramesh [Thu, 4 May 2017 06:17:59 +0000 (23:17 -0700)] 
SAMZA-1224 : Revert job coordinator factory config to the old format

We didn't release since adding this config. So, it is ok to change the format now.

Author: nramesh <nramesh@linkedin.com>

Reviewers: Xinyu Liu <xiliu@linkedin.com>, Prateek Maheshwari <pmaheshw@linkedin.com>

Closes #146 from navina/SAMZA-1224

6 months agoSAMZA-1259: LocalApplicationRunner throws exception when configured with a ProcessorI...
Sean McCauliff [Thu, 4 May 2017 05:58:51 +0000 (22:58 -0700)] 
SAMZA-1259: LocalApplicationRunner throws exception when configured with a ProcessorIdGenerator

Minor fix.  Unit tests were updated and passed.

Author: Sean McCauliff <smccauliff@linkedin.com>

Reviewers: Navina Ramesh <nramesh@linkedin.com>

Closes #161 from smccauliff/samza-1259