tez.git
2 years agoMerge remote-tracking branch 'origin/branch-0.7' into branch-0.7.1 branch-0.7.1 rel/release-0.7.1 release-0.7.1-rc2
Hitesh Shah [Wed, 4 May 2016 17:09:32 +0000 (10:09 -0700)] 
Merge remote-tracking branch 'origin/branch-0.7' into branch-0.7.1

2 years agoTEZ-3241. Exclude LICENSE files from rat check. (hitesh)
Hitesh Shah [Wed, 4 May 2016 17:08:45 +0000 (10:08 -0700)] 
TEZ-3241. Exclude LICENSE files from rat check. (hitesh)

2 years agoTEZ-3236. Update release date in CHANGES.txt. (hitesh)
Hitesh Shah [Tue, 3 May 2016 21:59:47 +0000 (14:59 -0700)] 
TEZ-3236. Update release date in CHANGES.txt. (hitesh)

2 years agoMerge remote-tracking branch 'origin/branch-0.7' into branch-0.7.1
Hitesh Shah [Tue, 3 May 2016 21:58:44 +0000 (14:58 -0700)] 
Merge remote-tracking branch 'origin/branch-0.7' into branch-0.7.1

2 years agoTEZ-3193. Deadlock in AM during task commit request. (Jason Lowe via hitesh)
Hitesh Shah [Tue, 3 May 2016 21:54:49 +0000 (14:54 -0700)] 
TEZ-3193. Deadlock in AM during task commit request. (Jason Lowe via hitesh)

2 years agoTEZ-3236. Update release date in CHANGES.txt release-0.7.1-rc1
Hitesh Shah [Fri, 29 Apr 2016 02:58:53 +0000 (19:58 -0700)] 
TEZ-3236. Update release date in CHANGES.txt

2 years agoTEZ-3236. Update version to 0.7.1. (hitesh)
Hitesh Shah [Fri, 29 Apr 2016 02:54:29 +0000 (19:54 -0700)] 
TEZ-3236. Update version to 0.7.1. (hitesh)

2 years agoTEZ-3203. DAG hangs when one of the upstream vertices has zero tasks (jlowe)
Jason Lowe [Thu, 28 Apr 2016 13:59:41 +0000 (13:59 +0000)] 
TEZ-3203. DAG hangs when one of the upstream vertices has zero tasks (jlowe)
(cherry picked from commit 7221d386a4fbc1f32aae1854bd25defb4c6d557a)

Conflicts:

CHANGES.txt

2 years agoTEZ-3213. Uncaught exception during vertex recovery leads to invalid state transition...
Hitesh Shah [Wed, 27 Apr 2016 20:13:38 +0000 (13:13 -0700)] 
TEZ-3213. Uncaught exception during vertex recovery leads to invalid state transition loop. (Eric Badger via hitesh)

2 years agoTEZ-3224. User payload is not initialized before creating vertex manager plugin....
Hitesh Shah [Fri, 22 Apr 2016 22:32:06 +0000 (15:32 -0700)] 
TEZ-3224. User payload is not initialized before creating vertex manager plugin. (Zhiyuan Yang via hitesh)

(cherry picked from commit 014d7c4ab4fca8d6119ea6ea993de3632b9f7609)

Conflicts:
CHANGES.txt

2 years agoTEZ-3165. Allow Inputs/Outputs to be initialized serially, control processor initiali...
Jonathan Eagles [Thu, 14 Apr 2016 20:51:47 +0000 (15:51 -0500)] 
TEZ-3165. Allow Inputs/Outputs to be initialized serially, control processor initialization relative to Inputs/Outputs (jeagles)

2 years agoTEZ-3202. Reduce the memory need for jobs with high number of segments (jeagles)
Jonathan Eagles [Thu, 14 Apr 2016 20:30:29 +0000 (15:30 -0500)] 
TEZ-3202. Reduce the memory need for jobs with high number of segments (jeagles)

2 years agoTEZ-3188. Move tez.submit.hosts out of TezConfiguration to
Siddharth Seth [Fri, 8 Apr 2016 17:13:03 +0000 (10:13 -0700)] 
TEZ-3188. Move tez.submit.hosts out of TezConfiguration to
TezConfigurationConstants. (sseth)

2 years agoTEZ-3196. java.lang.InternalError from decompression codec is fatal to a task during...
Jason Lowe [Fri, 8 Apr 2016 14:12:12 +0000 (14:12 +0000)] 
TEZ-3196. java.lang.InternalError from decompression codec is fatal to a task during shuffle (jlowe)

2 years agoTEZ-3201. Tez-UI build broken (sree)
Sreenath Somarajapuram [Thu, 7 Apr 2016 08:24:05 +0000 (13:54 +0530)] 
TEZ-3201. Tez-UI build broken (sree)

2 years agoTEZ-3177. Addendum patch. Non-DAG events should use the session domain or no domain...
Hitesh Shah [Tue, 5 Apr 2016 15:08:45 +0000 (08:08 -0700)] 
TEZ-3177. Addendum patch. Non-DAG events should use the session domain or no domain if the data does not need protection. (hitesh)

2 years agoTEZ-3177. Non-DAG events should use the session domain or no domain if the data does...
Hitesh Shah [Tue, 5 Apr 2016 15:01:42 +0000 (08:01 -0700)] 
TEZ-3177. Non-DAG events should use the session domain or no domain if the data does not need protection. (hitesh)

2 years agoTEZ-3192. IFile#checkState creating unnecessary objects though auto-boxing (jeagles)
Jonathan Eagles [Fri, 1 Apr 2016 00:15:58 +0000 (19:15 -0500)] 
TEZ-3192. IFile#checkState creating unnecessary objects though auto-boxing (jeagles)

(cherry picked from commit 0c7e1c5bb252f2a2631457adad3ebf29c23425e2)

2 years agoTEZ-3189. Pre-warm dags should not be counted in submitted dags count by DAGAppMaster...
Hitesh Shah [Tue, 29 Mar 2016 17:54:49 +0000 (10:54 -0700)] 
TEZ-3189. Pre-warm dags should not be counted in submitted dags count by DAGAppMaster. (Zhiyuan Yang via hitesh)

(cherry picked from commit c547dcc809cfabc7160d6d570efabdce9c4adbdf)

Conflicts:
CHANGES.txt

2 years agoTEZ-2967. Vertex start time should be that of first task start time in UI (jeagles)
Jonathan Eagles [Mon, 21 Mar 2016 15:39:09 +0000 (10:39 -0500)] 
TEZ-2967. Vertex start time should be that of first task start time in UI (jeagles)

(cherry picked from commit 19280cfc656e19e686806bd3ab16a0a7f8610380)

2 years agoTEZ-3175. Add tez client submit host (jeagles)
Jonathan Eagles [Fri, 18 Mar 2016 19:51:10 +0000 (14:51 -0500)] 
TEZ-3175. Add tez client submit host (jeagles)

2 years agoTEZ-3166. Fix a few cases where counters aren't fully updated and sent for failed...
Siddharth Seth [Fri, 18 Mar 2016 00:40:22 +0000 (17:40 -0700)] 
TEZ-3166. Fix a few cases where counters aren't fully updated and sent for failed tasks. Only
adds tests. Contributed by Zhiyuan Yang.

2 years agoTEZ-2958. Recovered TA, whose commit cannot be recovered, should move to killed state...
Jason Lowe [Thu, 17 Mar 2016 19:08:55 +0000 (19:08 +0000)] 
TEZ-2958. Recovered TA, whose commit cannot be recovered, should move to killed state (jlowe)

2 years agoTEZ-3105. TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez...
Hitesh Shah [Mon, 14 Mar 2016 18:17:51 +0000 (11:17 -0700)] 
TEZ-3105. TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures. (Greg Senia via hitesh)

(cherry picked from commit fac2f5fe1e84ed57db045d0467529b3627b91458)

Conflicts:
CHANGES.txt

2 years agoTEZ-2863. Container, node, and logs not available in UI for tasks that fail to launch...
Jonathan Eagles [Tue, 8 Mar 2016 23:33:26 +0000 (17:33 -0600)] 
TEZ-2863. Container, node, and logs not available in UI for tasks that fail to launch (jeagles)

2 years agoTEZ-3140. Reduce AM memory usage during serialization
Rohini Palaniswamy [Mon, 7 Mar 2016 15:57:22 +0000 (07:57 -0800)] 
TEZ-3140. Reduce AM memory usage during serialization

2 years agoTEZ-3156. Tez client keeps trying to talk to RM even if RM does not know about the...
Hitesh Shah [Thu, 3 Mar 2016 20:19:12 +0000 (12:19 -0800)] 
TEZ-3156. Tez client keeps trying to talk to RM even if RM does not know about the application. (hitesh)

(cherry picked from commit 2af886b509015200e1c04527275474cbc771c667)

Conflicts:
CHANGES.txt

2 years agoTEZ-3115. Shuffle string handling adds significant memory overhead (jeagles)
Jonathan Eagles [Wed, 2 Mar 2016 20:05:27 +0000 (14:05 -0600)] 
TEZ-3115. Shuffle string handling adds significant memory overhead (jeagles)

2 years agoTEZ-3149. Tez-tools: Add username in DagInfo (rbalamohan)
Rajesh Balamohan [Mon, 29 Feb 2016 21:59:14 +0000 (03:29 +0530)] 
TEZ-3149. Tez-tools: Add username in DagInfo (rbalamohan)

2 years agoTEZ-2988. DAGAppMaster::shutdownTezAM should return with a no-op if it has been invok...
Hitesh Shah [Mon, 29 Feb 2016 18:52:27 +0000 (10:52 -0800)] 
TEZ-2988. DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier. (Tsuyoshi Ozawa via hitesh)

(cherry picked from commit dd7919061cd64accb08782d1fc3f58635d64fa72)

Conflicts:
CHANGES.txt
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

2 years agoTEZ-3147. Intermediate mem-to-mem: Fix early exit when only one segment can fit into...
Jason Lowe [Mon, 29 Feb 2016 16:11:48 +0000 (16:11 +0000)] 
TEZ-3147. Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory (rbalamohan)
(cherry picked from commit 3e409ae0ee7233b4cf631cac1bc366679a08b7d1)

Conflicts:

CHANGES.txt

2 years agoTEZ-1911. addendum patch. MergeManager's unconditionalReserve() should check for...
Rajesh Balamohan [Sat, 27 Feb 2016 01:33:18 +0000 (07:03 +0530)] 
TEZ-1911. addendum patch. MergeManager's unconditionalReserve() should check for memory limits before allocating memory to IntermediateMemoryToMemoryMerger (rbalamohan)
(cherry picked from commit 12695f3d030b287d10d221fe28d69b6fa5bbc09a)

2 years agoTEZ-1911. MergeManager's unconditionalReserve() should check for memory limits before...
Jason Lowe [Mon, 29 Feb 2016 15:56:35 +0000 (15:56 +0000)] 
TEZ-1911. MergeManager's unconditionalReserve() should check for memory limits before allocating memory to IntermediateMemoryToMemoryMerger (rbalamohan)
(cherry picked from commit 15d7339e9fb64cdd0d995da9832dff721c14eacf)

Conflicts:

CHANGES.txt
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/MergeManager.java

2 years agoTEZ-3141. mapreduce.task.timeout is not translated to container heartbeat timeout...
Jason Lowe [Fri, 26 Feb 2016 21:59:16 +0000 (21:59 +0000)] 
TEZ-3141. mapreduce.task.timeout is not translated to container heartbeat timeout (jlowe)
(cherry picked from commit 41ac55155a59ba60b3da64b6b627c5760d8d6dcf)

Conflicts:

CHANGES.txt

2 years agoTEZ-3128. Avoid stopping containers on the AM shutdown thread. Contributed by Tsuyosh...
Jason Lowe [Fri, 26 Feb 2016 19:51:24 +0000 (19:51 +0000)] 
TEZ-3128. Avoid stopping containers on the AM shutdown thread. Contributed by Tsuyoshi Ozawa.
(cherry picked from commit 980385ab6670abf62d65c5679abee2af3bda0183)

Conflicts:

CHANGES.txt

2 years agoTEZ-3129. Tez task and task attempt UI needs application fails with NotFoundException...
Jonathan Eagles [Fri, 26 Feb 2016 16:38:58 +0000 (10:38 -0600)] 
TEZ-3129. Tez task and task attempt UI needs application fails with NotFoundException (jeagles)

(cherry picked from commit 9710575b152a0b93b1fa213d9017fd21070e09a5)

2 years agoTEZ-3114. Shuffle OOM due to EventMetaData flood (jlowe)
Jason Lowe [Fri, 26 Feb 2016 15:46:29 +0000 (15:46 +0000)] 
TEZ-3114. Shuffle OOM due to EventMetaData flood (jlowe)
(cherry picked from commit 923f7b4e298703658598d5cd3809f38f1231c4ab)

Conflicts:

CHANGES.txt
tez-runtime-internals/src/main/java/org/apache/tez/runtime/LogicalIOProcessorRuntimeTask.java

2 years agoTEZ-3102. Fetch failure of a speculated task causes job hang (jlowe)
Jason Lowe [Thu, 25 Feb 2016 20:23:45 +0000 (20:23 +0000)] 
TEZ-3102. Fetch failure of a speculated task causes job hang (jlowe)
(cherry picked from commit cc9dd2799ff67243017edb9ae5df42dc887032c9)

Conflicts:

CHANGES.txt
tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestTaskImpl.java

2 years agoTEZ-3137. Tez task failed with illegal state exception in recovery (zjffdu)
Jeff Zhang [Thu, 25 Feb 2016 04:11:02 +0000 (12:11 +0800)] 
TEZ-3137. Tez task failed with illegal state exception in recovery (zjffdu)

2 years agoTEZ-3126. Log reason for not reducing parallelism (jeagles)
Jonathan Eagles [Tue, 23 Feb 2016 04:45:23 +0000 (22:45 -0600)] 
TEZ-3126. Log reason for not reducing parallelism (jeagles)

(cherry picked from commit fd75e640396da8d5e1c67ef554d5db1846e08c69)

2 years agoTEZ-3123. Containers can get re-used even with conflicting local resources. (hitesh)
Hitesh Shah [Fri, 19 Feb 2016 22:52:28 +0000 (14:52 -0800)] 
TEZ-3123. Containers can get re-used even with conflicting local resources. (hitesh)

2 years agoTEZ-3117. Deadlock in Edge and Vertex code (bikas)
Bikas Saha [Thu, 18 Feb 2016 01:48:55 +0000 (17:48 -0800)] 
TEZ-3117. Deadlock in Edge and Vertex code (bikas)
(cherry picked from commit de3a0748ff19b5ced87050596d088bdb573cae05)

Conflicts:
CHANGES.txt

2 years agoTEZ-3103. Shuffle can hang when memory to memory merging enabled (jlowe)
Jason Lowe [Fri, 12 Feb 2016 18:34:04 +0000 (18:34 +0000)] 
TEZ-3103. Shuffle can hang when memory to memory merging enabled (jlowe)

Conflicts:

CHANGES.txt
tez-runtime-library/src/test/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/TestMergeManager.java

2 years agoTEZ-3107. tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime >...
Rajesh Balamohan [Fri, 12 Feb 2016 17:59:25 +0000 (09:59 -0800)] 
TEZ-3107. tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime > finishTime) (rbalamohan)

(cherry picked from commit 7e3d5461c3b948ca1c27f386e3e9e3665b8a649e)

2 years agoTEZ-3104. Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier
Jonathan Eagles [Thu, 11 Feb 2016 16:33:52 +0000 (10:33 -0600)] 
TEZ-3104. Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier

(cherry picked from commit 6f57630ee5d5a1a02342a25e4b48a698fd237390)

2 years agoTEZ-3093. addendum patch (rbalamohan)
Rajesh Balamohan [Fri, 5 Feb 2016 03:04:28 +0000 (19:04 -0800)] 
TEZ-3093. addendum patch (rbalamohan)

(cherry picked from commit c7397f5de42bd83f2a479ca1b29ef007cc40cc25)

2 years agoTEZ-3093. CriticalPathAnalyzer should be accessible via zeppelin (rbalamohan)
Rajesh Balamohan [Fri, 5 Feb 2016 00:29:48 +0000 (16:29 -0800)] 
TEZ-3093. CriticalPathAnalyzer should be accessible via zeppelin (rbalamohan)

(cherry picked from commit 12e1aae1eac834a77e3a2b5b3be1cd6d59196d48)

2 years agoTEZ-3089. TaskConcurrencyAnalyzer can return negative task count with very large...
Rajesh Balamohan [Tue, 2 Feb 2016 11:56:46 +0000 (03:56 -0800)] 
TEZ-3089. TaskConcurrencyAnalyzer can return negative task count with very large jobs (rbalamohan)

(cherry picked from commit 89bc6abf6c3caddb6224a19b7b47010b7bca4eff)

2 years agoTEZ-2307. Possible wrong error message when submitting new dag (zjffdu)
Jeff Zhang [Tue, 2 Feb 2016 05:21:45 +0000 (13:21 +0800)] 
TEZ-2307. Possible wrong error message when submitting new dag (zjffdu)

(cherry picked from commit 235841f77ebf88994c8d7af189cf1000aedbd69f)

2 years agoTEZ-3076. Reduce merge memory overhead to support large number of in-memory mapoutput...
Jonathan Eagles [Fri, 29 Jan 2016 19:48:58 +0000 (13:48 -0600)] 
TEZ-3076. Reduce merge memory overhead to support large number of in-memory mapoutputs (jeagles)

2 years agoTEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or...
Jeff Zhang [Fri, 22 Jan 2016 05:32:59 +0000 (21:32 -0800)] 
TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services (zjffdu)

(cherry picked from commit 2bf27de3bfbe7a54fbf2982fa50ab6b32974d6b1)

2 years agoTEZ-3036. Tez AM can hang on startup with no indication of error (jlowe)
Jason Lowe [Thu, 21 Jan 2016 20:30:07 +0000 (20:30 +0000)] 
TEZ-3036. Tez AM can hang on startup with no indication of error (jlowe)
(cherry picked from commit 92def52ff8b02eab7aae38170ca6c9b0caf83ef7)

Conflicts:

CHANGES.txt

2 years agoTEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED (jlowe)
Jason Lowe [Thu, 21 Jan 2016 19:04:17 +0000 (19:04 +0000)] 
TEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED (jlowe)

2 years agoTEZ-2594. Fix LICENSE for missing entries for full and minimal tarballs. (hitesh)
Hitesh Shah [Wed, 20 Jan 2016 21:59:53 +0000 (13:59 -0800)] 
TEZ-2594. Fix LICENSE for missing entries for full and minimal tarballs. (hitesh)

(cherry picked from commit 73e993cba52e25c0d126a59439a76b43af9506bc)

2 years agoTEZ-2595. Add license and notice files into assembled full and minimal tarballs....
Hitesh Shah [Sat, 7 Nov 2015 05:10:19 +0000 (21:10 -0800)] 
TEZ-2595. Add license and notice files into assembled full and minimal tarballs. (hitesh)

(cherry picked from commit 62f8d87eab70bf734e029ade236babf08d72232c)

2 years agoMinor changes:
Hitesh Shah [Tue, 11 Aug 2015 22:58:00 +0000 (15:58 -0700)] 
Minor changes:
  - Moved License text from full into separate license files.
  - Moved stax-api-1.0-2 into CDDL v1.0 section as per http://mvnrepository.com/artifact/javax.xml.stream/stax-api/1.0-2
  - Added SIL OFL license text.

(cherry picked from commit 8e35e498ea14342c72b2aaa77556987f9bc3d006)

2 years agoMove *-full files into dist-files dir
Hitesh Shah [Tue, 11 Aug 2015 22:10:51 +0000 (15:10 -0700)] 
Move *-full files into dist-files dir

(cherry picked from commit 74d1bb7556939551e0e829525d644d220473f1e3)

2 years agoAdditional updates for minimal tarball LICENSE and NOTICE (hitesh)
Hitesh Shah [Fri, 31 Jul 2015 22:49:40 +0000 (15:49 -0700)] 
Additional updates for minimal tarball LICENSE and NOTICE (hitesh)

(cherry picked from commit b37d4254084a19f56557cc0eeb8806e8fade7063)

2 years agoTEZ-2640. Add entry in NOTICE file (zjffdu)
Jeff Zhang [Wed, 29 Jul 2015 06:01:57 +0000 (14:01 +0800)] 
TEZ-2640. Add entry in NOTICE file (zjffdu)

(cherry picked from commit f72fcf8c0c9b05aedcd34eac5a0f3b2609cc7193)

2 years agoTEZ-2640. Update jar licences (zjffdu)
Jeff Zhang [Tue, 28 Jul 2015 07:55:15 +0000 (15:55 +0800)] 
TEZ-2640. Update jar licences (zjffdu)

(cherry picked from commit a60b6a6af2310417e8a2cc7f4d87f74b1ffe9422)

2 years agoReverting the changes related to LICENSE-minimal.txt (rbalamohan)
Rajesh Balamohan [Fri, 24 Jul 2015 01:27:28 +0000 (06:57 +0530)] 
Reverting the changes related to LICENSE-minimal.txt (rbalamohan)

(cherry picked from commit 27e4d6a0575e7642f40eca7ebc2297e30271cd55)

2 years agoUpdated license files for leveldbjni-all, paranamer, log4j, netty, protobuf-java...
Rajesh Balamohan [Fri, 24 Jul 2015 00:01:33 +0000 (05:31 +0530)] 
Updated license files for leveldbjni-all, paranamer, log4j, netty, protobuf-java, servlet-api, slf4j-api, slf4j-log4j, snappy-java, stax-api, xercesImpl, xmlenc, xz, zookeeper (rbalamohan)

(cherry picked from commit 905a3905f6edb5593cb20c7f5c28605384f8cea6)

2 years agoTEZ-2639
Bikas Saha [Thu, 23 Jul 2015 23:37:20 +0000 (16:37 -0700)] 
TEZ-2639

(cherry picked from commit 3f497c397ce0604f6838786ad8fe098985f92829)

2 years agoInitial starting point for TEZ-2592. (hitesh)
Hitesh Shah [Tue, 7 Jul 2015 20:24:54 +0000 (13:24 -0700)] 
Initial starting point for TEZ-2592. (hitesh)

(cherry picked from commit 38d5e84d1fa584023fc713e0cb00393c1dd6253e)

2 years agoTEZ-3046. Compilation issue in tez-runtime-internals of branch-0.7 release-0.7.1-rc0
Jonathan Eagles [Tue, 19 Jan 2016 20:08:50 +0000 (14:08 -0600)] 
TEZ-3046. Compilation issue in tez-runtime-internals of branch-0.7

2 years agoTEZ-2937. Can Processor.close() be called after closing inputs and outputs? (jeagles)
Jonathan Eagles [Fri, 15 Jan 2016 16:58:37 +0000 (10:58 -0600)] 
TEZ-2937. Can Processor.close() be called after closing inputs and outputs? (jeagles)

(cherry picked from commit b0ba133ffcf1339b2c505878e9622084d4290bde)

2 years agoRevert "TEZ-3032. DAG start time getting logged using system time instead of recorded...
Hitesh Shah [Thu, 14 Jan 2016 22:29:25 +0000 (14:29 -0800)] 
Revert "TEZ-3032. DAG start time getting logged using system time instead of recorded time in startTime field. (hitesh)"

This reverts commit 7041fa79510d66577efccdf5dcc33e6242b403d5.

Conflicts:
CHANGES.txt

2 years agoTEZ-3037. History URL should be set regardless of which history logging service is...
Hitesh Shah [Thu, 14 Jan 2016 22:15:42 +0000 (14:15 -0800)] 
TEZ-3037. History URL should be set regardless of which history logging service is enabled. (hitesh)

2 years agoTEZ-3032. DAG start time getting logged using system time instead of recorded time...
Hitesh Shah [Thu, 14 Jan 2016 21:49:17 +0000 (13:49 -0800)] 
TEZ-3032. DAG start time getting logged using system time instead of recorded time in startTime field. (hitesh)

(cherry picked from commit 37e70146e9c6162c048af2388f732a701654cda9)

Conflicts:
CHANGES.txt
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java

2 years agoTEZ-2129. Task and Attempt views should contain links to the logs (jeagles)
Jonathan Eagles [Fri, 8 Jan 2016 20:37:03 +0000 (14:37 -0600)] 
TEZ-2129. Task and Attempt views should contain links to the logs (jeagles)

2 years agoTEZ-3025. InputInitializer creation should use the dag ugi. (sseth)
Siddharth Seth [Fri, 8 Jan 2016 01:39:34 +0000 (17:39 -0800)] 
TEZ-3025. InputInitializer creation should use the dag ugi. (sseth)

2 years agoTEZ-2972. Avoid task rescheduling when a node turns unhealthy (jlowe)
Jason Lowe [Tue, 5 Jan 2016 19:31:35 +0000 (19:31 +0000)] 
TEZ-2972. Avoid task rescheduling when a node turns unhealthy (jlowe)

2 years agoTEZ-3017. HistoryACLManager does not have a close method for cleanup (bikas)
Bikas Saha [Tue, 29 Dec 2015 09:54:00 +0000 (01:54 -0800)] 
TEZ-3017. HistoryACLManager does not have a close method for cleanup (bikas)
(cherry picked from commit 9816a49bba0c89651f9b5bb66ac20f4eb0364ef2)

Conflicts:
CHANGES.txt

2 years agoTEZ-2914. Ability to limit vertex concurrency (bikas)
Bikas Saha [Sat, 26 Dec 2015 03:39:04 +0000 (19:39 -0800)] 
TEZ-2914. Ability to limit vertex concurrency (bikas)
(cherry picked from commit 34eb75d709bbe6e0417f9b4023f4fa1cec81bd8b)

Conflicts:
CHANGES.txt
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskImpl.java
tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerEventHandler.java
tez-dag/src/test/java/org/apache/tez/dag/app/MockDAGAppMaster.java
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestTaskSchedulerEventHandler.java

2 years agofixup CHANGES.txt
Jonathan Eagles [Fri, 18 Dec 2015 07:50:48 +0000 (01:50 -0600)] 
fixup CHANGES.txt

2 years agoTEZ-3011. Link Vertex Name in Dag Tasks/Task Attempts to Vertex (jeagles)
Jonathan Eagles [Fri, 18 Dec 2015 07:44:26 +0000 (01:44 -0600)] 
TEZ-3011. Link Vertex Name in Dag Tasks/Task Attempts to Vertex (jeagles)

(cherry picked from commit c34c54f70214560ed20cbb885da90230346b9e47)

2 years agoTEZ-2538. ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple...
Rajesh Balamohan [Wed, 16 Dec 2015 22:44:18 +0000 (04:14 +0530)] 
TEZ-2538. ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions (rbalamohan)

2 years agoTEZ-3006. Remove unused import in TestHistoryParser (rbalamohan)
Rajesh Balamohan [Wed, 16 Dec 2015 14:37:25 +0000 (20:07 +0530)] 
TEZ-3006. Remove unused import in TestHistoryParser (rbalamohan)

(cherry picked from commit eee485a69c1899824bd9ed14b6d77719ff07d375)

2 years agoTEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition...
Rajesh Balamohan [Tue, 15 Dec 2015 23:18:08 +0000 (04:48 +0530)] 
TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes from the source. Addendum patch. (rbalamohan)

2 years agoTEZ-2918. Make progress notifications in IOs (bikas)
Bikas Saha [Fri, 13 Nov 2015 13:54:09 +0000 (05:54 -0800)] 
TEZ-2918. Make progress notifications in IOs (bikas)
(cherry picked from commit 5ec498d8f5b6cecf4eaee8f995ea2cf9ca2acfcc)

Conflicts:
CHANGES.txt
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/lib/MRReaderMapReduce.java
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/readers/UnorderedKVReader.java
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/MergeManager.java
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/PipelinedSorter.java
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/dflt/DefaultSorter.java
tez-runtime-library/src/test/java/org/apache/tez/runtime/library/common/readers/TestUnorderedKVReader.java
tez-runtime-library/src/test/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/TestMergeManager.java
tez-runtime-library/src/test/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/TestShuffleScheduler.java

2 years agoTEZ-2952. NPE in TestOnFileUnorderedKVOutput (bikas)
Bikas Saha [Thu, 19 Nov 2015 18:17:22 +0000 (10:17 -0800)] 
TEZ-2952. NPE in TestOnFileUnorderedKVOutput (bikas)
(cherry picked from commit 4561b82524ca6ee484910349a6c95703883757b6)

Conflicts:
CHANGES.txt
tez-runtime-internals/src/test/java/org/apache/tez/runtime/api/impl/TestProcessorContext.java

2 years agoTEZ-808. Handle task attempts that are not making progress (bikas)
Bikas Saha [Fri, 30 Oct 2015 05:10:56 +0000 (22:10 -0700)] 
TEZ-808. Handle task attempts that are not making progress (bikas)
(cherry picked from commit 414258e40ff72961c6d1e6fb287b3d2c9732a7a8)

Conflicts:
CHANGES.txt

2 years agoTEZ-2979. FlakyTest: org.apache.tez.history.TestHistoryParser. (rbalamohan)
Rajesh Balamohan [Mon, 14 Dec 2015 06:19:29 +0000 (11:49 +0530)] 
TEZ-2979. FlakyTest: org.apache.tez.history.TestHistoryParser. (rbalamohan)

2 years agoTEZ-2987. TestVertexImpl.testTez2684 fails (bikas)
Bikas Saha [Mon, 14 Dec 2015 03:48:17 +0000 (19:48 -0800)] 
TEZ-2987. TestVertexImpl.testTez2684 fails (bikas)
(cherry picked from commit c9741492056758e3cac1a3d34cd1c7cbd7d9ddd4)

Conflicts:
CHANGES.txt

2 years agoTEZ-2599. Dont send obsoleted data movement events to tasks (bikas)
Bikas Saha [Sat, 4 Jul 2015 08:17:24 +0000 (01:17 -0700)] 
TEZ-2599. Dont send obsoleted data movement events to tasks (bikas)
(cherry picked from commit aca83090ed2580d093a1fec59217d78f2e575a6c)

Conflicts:
CHANGES.txt
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java

2 years agoTEZ-2684. ShuffleVertexManager.parsePartitionStats throws IllegalStateException:...
Rajesh Balamohan [Mon, 14 Dec 2015 02:54:44 +0000 (08:24 +0530)] 
TEZ-2684. ShuffleVertexManager.parsePartitionStats throws IllegalStateException: Stats should be initialized. (rbalamohan)

2 years agoTEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition...
Rajesh Balamohan [Mon, 14 Dec 2015 02:20:22 +0000 (07:50 +0530)] 
TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes from the source. (rbalamohan)

2 years agoTEZ-2995. Timeline primary filter should only be on callerId and not type. (hitesh)
Hitesh Shah [Sat, 12 Dec 2015 04:30:16 +0000 (20:30 -0800)] 
TEZ-2995. Timeline primary filter should only be on callerId and not type. (hitesh)

(cherry picked from commit a34f109650f633a06320238c08884b5817225085)

Conflicts:
CHANGES.txt

2 years agoTEZ-2943. Change shuffle vertex manager to use per vertex data for auto reduce and...
Bikas Saha [Thu, 10 Dec 2015 01:00:17 +0000 (17:00 -0800)] 
TEZ-2943. Change shuffle vertex manager to use per vertex data for auto reduce and slow start (bikas)
(cherry picked from commit 0dd68ad1c92eca98a98b8bd4a0c54b8656573e8d)

Conflicts:
CHANGES.txt
tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java
tez-runtime-library/src/test/java/org/apache/tez/dag/library/vertexmanager/TestShuffleVertexManager.java

2 years agoTEZ-2956. Handle auto-reduce parallelism when the totalNumBipartiteSourceTasks is...
Bikas Saha [Wed, 25 Nov 2015 02:48:23 +0000 (18:48 -0800)] 
TEZ-2956. Handle auto-reduce parallelism when the totalNumBipartiteSourceTasks is 0 (Rajesh Balamohan and Bikas Saha)
(cherry picked from commit c4487f966a81c01ed061d502a397e2cf3b4bce44)

Conflicts:
CHANGES.txt
tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java

2 years agoFix Hadoop 2.2 compat
Bikas Saha [Sat, 12 Dec 2015 01:49:33 +0000 (17:49 -0800)] 
Fix Hadoop 2.2 compat

2 years agoFix build break from 37074a92eda4c61c50582ecb97f83a2781c42ecd
Bikas Saha [Sat, 12 Dec 2015 01:42:21 +0000 (17:42 -0800)] 
Fix build break from 37074a92eda4c61c50582ecb97f83a2781c42ecd

2 years agopostfix
Bikas Saha [Fri, 11 Dec 2015 22:46:53 +0000 (14:46 -0800)] 
postfix

2 years agoTEZ-2633. Allow VertexManagerPlugins to receive and report based on attempts instead...
Bikas Saha [Wed, 5 Aug 2015 18:05:54 +0000 (11:05 -0700)] 
TEZ-2633. Allow VertexManagerPlugins to receive and report based on attempts instead of tasks (bikas)
(cherry picked from commit 7b45e9a142830e7dd8b0263d50dbaaef5fb0da76)

Conflicts:
CHANGES.txt
tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java
tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java
tez-runtime-library/src/test/java/org/apache/tez/dag/library/vertexmanager/TestShuffleVertexManager.java

2 years agoTEZ-2824. Add javadocs for Vertex.setConf and DAG.setConf. (sseth)
Siddharth Seth [Fri, 11 Dec 2015 22:09:30 +0000 (14:09 -0800)] 
TEZ-2824. Add javadocs for Vertex.setConf and DAG.setConf. (sseth)
(cherry picked from commit b748ffe1e7839568e52d05a8bcd83d7dd3aeda29)

Conflicts:
CHANGES.txt

2 years agoTEZ-2963. RecoveryService#handleSummaryEvent exception with HDFS transparent encrypti...
Hitesh Shah [Fri, 11 Dec 2015 18:45:31 +0000 (10:45 -0800)] 
TEZ-2963. RecoveryService#handleSummaryEvent exception with HDFS transparent encryption + kerberos authentication. (Karel Kolman and hitesh via hitesh)

(cherry picked from commit 0c5ccb66afa1a972b2652ecd1d3f98c974f31add)

Conflicts:
CHANGES.txt

2 years agoTEZ-2966. Tez does not honor mapreduce.task.timeout (jeagles)
Jonathan Eagles [Fri, 11 Dec 2015 16:16:25 +0000 (10:16 -0600)] 
TEZ-2966. Tez does not honor mapreduce.task.timeout (jeagles)

(cherry picked from commit 460e0b2a01c00748bb80ad2314b2168dfa1e9615)

2 years agoTEZ-2346. TEZ-UI: Lazy load other info / counter data (Sreenath Somarajapuram via...
Jonathan Eagles [Mon, 7 Dec 2015 23:26:58 +0000 (17:26 -0600)] 
TEZ-2346. TEZ-UI: Lazy load other info / counter data (Sreenath Somarajapuram via jeagles)

(cherry picked from commit 5af06047f03eafbfa97ba0b5213b79bb6e38165e)

2 years agoTEZ-2973. Backport Analyzers to branch-0.7
Jonathan Eagles [Mon, 7 Dec 2015 22:41:00 +0000 (16:41 -0600)] 
TEZ-2973. Backport Analyzers to branch-0.7

2 years agoTEZ-2975. Bump up apache commons dependency. (hitesh)
Hitesh Shah [Mon, 7 Dec 2015 22:23:58 +0000 (14:23 -0800)] 
TEZ-2975. Bump up apache commons dependency. (hitesh)

(cherry picked from commit 69e3de8185d78120dcc6558995e7827a79fc4f48)

Conflicts:
CHANGES.txt