hadoop.git
2 days agoHDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider (#4917) trunk
Navink [Fri, 30 Sep 2022 15:28:12 +0000 (20:58 +0530)] 
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider (#4917)

Contributed-by: navinko <nakumr@cloudera.com>
4 days agoHADOOP-18463. Add an integration test to process data asynchronously during vectored...
Mukund Thakur [Wed, 28 Sep 2022 17:46:47 +0000 (23:16 +0530)] 
HADOOP-18463. Add an integration test to process data asynchronously during vectored read. (#4921)

part of HADOOP-18103.

Contributed by: Mukund Thakur

5 days agoYARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924)
slfan1989 [Tue, 27 Sep 2022 20:31:52 +0000 (04:31 +0800)] 
YARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924)

5 days agoYARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubClu...
slfan1989 [Tue, 27 Sep 2022 20:28:52 +0000 (04:28 +0800)] 
YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster. (#4846)

5 days agoHADOOP-18347. S3A Vectored IO to use bounded thread pool. (#4918) feature-HADOOP-18073-s3a-sdk-upgrade 4944/head
Mukund Thakur [Tue, 27 Sep 2022 15:43:07 +0000 (21:13 +0530)] 
HADOOP-18347. S3A Vectored IO to use bounded thread pool. (#4918)

part of HADOOP-18103.

Also introducing a config fs.s3a.vectored.active.ranged.reads
to configure the maximum number of number of range reads a
single input stream can have active (downloading, or queued)
to the central FileSystem instance's pool of queued operations.
This stops a single stream overloading the shared thread pool.

Contributed by: Mukund Thakur

5 days agoHDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received...
Ashutosh Gupta [Tue, 27 Sep 2022 06:44:25 +0000 (07:44 +0100)] 
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source (#4886)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
5 days agoYARN-11308. Router Page display the db username and password in mask mode. (#4908)
slfan1989 [Mon, 26 Sep 2022 22:54:17 +0000 (06:54 +0800)] 
YARN-11308. Router Page display the db username and password in mask mode. (#4908)

6 days agoYARN-11316. Fix Yarn federation.md table format. (#4935)
slfan1989 [Mon, 26 Sep 2022 22:47:08 +0000 (06:47 +0800)] 
YARN-11316. Fix Yarn federation.md table format. (#4935)

6 days agoYARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897)
slfan1989 [Mon, 26 Sep 2022 22:46:06 +0000 (06:46 +0800)] 
YARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897)

6 days agoHADOOP-18466. Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem...
Viraj Jasani [Mon, 26 Sep 2022 17:56:58 +0000 (10:56 -0700)] 
HADOOP-18466. Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field (#4926)

Follow-on to HADOOP-18455.

Contributed by Viraj Jasani

7 days agoHADOOP-18443. Upgrade snakeyaml to 1.32 (#4906)
Ashutosh Gupta [Sun, 25 Sep 2022 14:49:48 +0000 (15:49 +0100)] 
HADOOP-18443. Upgrade snakeyaml to 1.32 (#4906)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
7 days agoYARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase...
Ashutosh Gupta [Sun, 25 Sep 2022 14:30:58 +0000 (15:30 +0100)] 
YARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-common (#4774)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
7 days agoYARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase...
Ashutosh Gupta [Sun, 25 Sep 2022 14:26:06 +0000 (15:26 +0100)] 
YARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-client (#4773)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
9 days agoHADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToApprop...
Xing Lin [Fri, 23 Sep 2022 17:37:51 +0000 (10:37 -0700)] 
HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash (#4869)

* HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash

Signed-off-by: Xing Lin <xinglin@linkedin.com>
9 days agoHADOOP-18456. NullPointerException in ObjectListingIterator. (#4909)
Steve Loughran [Fri, 23 Sep 2022 08:54:31 +0000 (09:54 +0100)] 
HADOOP-18456. NullPointerException in ObjectListingIterator. (#4909)

This problem surfaced in impala integration tests
   IMPALA-11592. TestLocalCatalogRetries.test_fetch_metadata_retry fails in S3 build
after the change
  HADOOP-17461. Add thread-level IOStatistics Context
The actual GC race condition came with
 HADOOP-18091. S3A auditing leaks memory through ThreadLocal references

The fix for this is, if our hypothesis is correct, in WeakReferenceMap.create()
where a strong reference to the new value is kept in a local variable
*and referred to later* so that the JVM will not GC it.

Along with the fix, extra assertions ensure that if the problem is not fixed,
applications will fail faster/more meaningfully.

Contributed by Steve Loughran.

9 days agoHDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a...
Kidd5368 [Fri, 23 Sep 2022 03:27:56 +0000 (11:27 +0800)] 
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task (#4901)

9 days agoYARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858)
slfan1989 [Thu, 22 Sep 2022 23:40:47 +0000 (07:40 +0800)] 
YARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858)

10 days agoHADOOP-18341: upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9 (#4578)
PJ Fanning [Thu, 22 Sep 2022 00:45:20 +0000 (01:45 +0100)] 
HADOOP-18341: upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9 (#4578)

Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
11 days agoHADOOP-18455. S3A prefetching executor should be closed (#4879)
Viraj Jasani [Wed, 21 Sep 2022 18:52:41 +0000 (11:52 -0700)] 
HADOOP-18455. S3A prefetching executor should be closed (#4879)

follow-on patch to HADOOP-18186.

Contributed by: Viraj Jasani

11 days agoHDFS-16706. ViewFS doc points to wrong mount table name (#4803)
Samrat [Wed, 21 Sep 2022 11:25:20 +0000 (16:55 +0530)] 
HDFS-16706. ViewFS doc points to wrong mount table name (#4803)

11 days agoYARN-11255. Support loading alternative docker client config from system environment...
Ashutosh Gupta [Wed, 21 Sep 2022 11:24:02 +0000 (12:24 +0100)] 
YARN-11255. Support loading alternative docker client config from system environment (#4884)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
11 days agoYARN-11307. Fix Yarn Router Broken Link. (#4905)
slfan1989 [Wed, 21 Sep 2022 03:10:09 +0000 (11:10 +0800)] 
YARN-11307. Fix Yarn Router Broken Link. (#4905)

12 days agoHADOOP-18451. Update hsqldb.version from 2.3.4 to 2.5.2. (#4880)
slfan1989 [Tue, 20 Sep 2022 18:10:51 +0000 (02:10 +0800)] 
HADOOP-18451. Update hsqldb.version from 2.3.4 to 2.5.2. (#4880)

12 days agoHDFS-16341. Fix BlockPlacementPolicy details in hdfs defaults. (#3691). Contributed...
GuoPhilipse [Tue, 20 Sep 2022 17:46:24 +0000 (01:46 +0800)] 
HDFS-16341. Fix BlockPlacementPolicy details in hdfs defaults. (#3691). Contributed by guophilipse.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
12 days agoHADOOP-16674. Fix when TestDNS.testRDNS can fail with ServiceUnavailableException...
Ashutosh Gupta [Tue, 20 Sep 2022 17:39:59 +0000 (18:39 +0100)] 
HADOOP-16674. Fix when TestDNS.testRDNS can fail with ServiceUnavailableException (#4802). Contributed by Ashutosh Gupta.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
12 days agoYARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed...
slfan1989 [Tue, 20 Sep 2022 17:36:24 +0000 (01:36 +0800)] 
YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241. (#4893). Contributed by fanshilun.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
12 days agoYARN-11303. Upgrade jquery ui to 1.13.2 to mitigate CVE-2022-31160 (#4895)
Ashutosh Gupta [Tue, 20 Sep 2022 16:50:03 +0000 (17:50 +0100)] 
YARN-11303. Upgrade jquery ui to 1.13.2 to mitigate CVE-2022-31160 (#4895)

12 days agoHADOOP-18377. hadoop-aws build to add a -prefetch profile to run all tests with prefe...
Viraj Jasani [Tue, 20 Sep 2022 09:26:13 +0000 (02:26 -0700)] 
HADOOP-18377. hadoop-aws build to add a -prefetch profile to run all tests with prefetching (#4914)

Contributed by Viraj Jasani

13 days agoHDFS-16772. refreshHostsReader should use the latest configuration (#4890)
ZanderXu [Mon, 19 Sep 2022 20:27:07 +0000 (04:27 +0800)] 
HDFS-16772. refreshHostsReader should use the latest configuration (#4890)

13 days agoYARN-11283. Fix Typo of NodeManager amrmproxy. (#4899)
slfan1989 [Mon, 19 Sep 2022 20:16:25 +0000 (04:16 +0800)] 
YARN-11283. Fix Typo of NodeManager amrmproxy. (#4899)

13 days agoYARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStor...
slfan1989 [Mon, 19 Sep 2022 20:14:55 +0000 (04:14 +0800)] 
YARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStore. (#4852)

13 days agoHADOOP-18118. [Follow on] Fix test failure in TestHttpServer (#4900)
GuoPhilipse [Mon, 19 Sep 2022 16:10:00 +0000 (00:10 +0800)] 
HADOOP-18118. [Follow on] Fix test failure in TestHttpServer (#4900)

Signed-off-by: Erik Krogen <xkrogen@apache.org>
13 days agoHADOOP-18400. Fix file split duplicating records from a succeeding split when reading...
Ashutosh Gupta [Mon, 19 Sep 2022 04:45:05 +0000 (05:45 +0100)] 
HADOOP-18400. Fix file split duplicating records from a succeeding split when reading BZip2 text files (#4732)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoHADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number...
ZanderXu [Fri, 16 Sep 2022 17:09:01 +0000 (01:09 +0800)] 
HADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number of re-queued RPCs (#4871)

Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2 weeks agoHADOOP-16769. LocalDirAllocator to provide diagnostics when file creation fails ...
Ashutosh Gupta [Fri, 16 Sep 2022 07:47:26 +0000 (08:47 +0100)] 
HADOOP-16769. LocalDirAllocator to provide diagnostics when file creation fails (#4842)

The patch provides detailed diagnostics of file creation failure in LocalDirAllocator.

Contributed by: Ashutosh Gupta

2 weeks agoHDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdEx...
ZanderXu [Thu, 15 Sep 2022 19:44:36 +0000 (03:44 +0800)] 
HDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdException (#4882)

Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2 weeks agoMAPREDUCE-7407. Avoid stopContainer() on dead node (#4779)
Ashutosh Gupta [Thu, 15 Sep 2022 17:30:36 +0000 (18:30 +0100)] 
MAPREDUCE-7407. Avoid stopContainer() on dead node (#4779)

2 weeks agoHDFS-16767: RBF: Support observer node in Router-Based Federation.
Simbarashe Dzinamarira [Tue, 13 Sep 2022 08:13:11 +0000 (01:13 -0700)] 
HDFS-16767: RBF: Support observer node in Router-Based Federation.

Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2 weeks agoHADOOP-18118. Fix KMS Accept Queue Size default value to 500 (#3972)
GuoPhilipse [Wed, 14 Sep 2022 16:53:25 +0000 (00:53 +0800)] 
HADOOP-18118. Fix KMS Accept Queue Size default value to 500 (#3972)

2 weeks agoHADOOP-15072 - Update Apache Kerby to 2.0.2 (#4473)
Colm O hEigeartaigh [Wed, 14 Sep 2022 16:43:25 +0000 (17:43 +0100)] 
HADOOP-15072 - Update Apache Kerby to 2.0.2 (#4473)

2 weeks agoHADOOP-18452. Fix TestKMS#testKMSHAZooKeeperDelegationToken Failed By Hadoop-18427...
slfan1989 [Wed, 14 Sep 2022 16:13:58 +0000 (00:13 +0800)] 
HADOOP-18452. Fix TestKMS#testKMSHAZooKeeperDelegationToken Failed By Hadoop-18427. (#4885)

2 weeks ago[Documentation] RBF: Duplicate statement to be removed for better readabilty (#4881)
Renukaprasad C [Tue, 13 Sep 2022 16:24:05 +0000 (21:54 +0530)] 
[Documentation] RBF: Duplicate statement to be removed for better readabilty (#4881)

2 weeks agoYARN-11301. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactor...
slfan1989 [Tue, 13 Sep 2022 08:08:16 +0000 (16:08 +0800)] 
YARN-11301. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-11269. (#4878)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoYARN-11261. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy (#4777)
Ashutosh Gupta [Tue, 13 Sep 2022 08:05:24 +0000 (09:05 +0100)] 
YARN-11261. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy (#4777)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoYARN-7614. [RESERVATION] Support ListReservation APIs in Federation Router. (#4843)
slfan1989 [Mon, 12 Sep 2022 19:33:21 +0000 (03:33 +0800)] 
YARN-7614. [RESERVATION] Support ListReservation APIs in Federation Router. (#4843)

2 weeks agoYARN-11241. Add uncleaning option for local app log file with log-aggregation enabled...
Ashutosh Gupta [Mon, 12 Sep 2022 14:32:20 +0000 (15:32 +0100)] 
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoHADOOP-18302. Remove WhiteBox in hadoop-common module. (#4457)
slfan1989 [Mon, 12 Sep 2022 14:28:16 +0000 (22:28 +0800)] 
HADOOP-18302. Remove WhiteBox in hadoop-common module. (#4457)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2 weeks agoAllow block reconstruction pending timeout to be refreshable (#4567)
caozhiqiang [Mon, 12 Sep 2022 02:45:01 +0000 (10:45 +0800)] 
Allow block reconstruction pending timeout to be refreshable (#4567)

Reviewed-by: Hiroyuki Adachi <hadachi@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
3 weeks agoYARN-11265. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-sharedcachemanager (...
Ashutosh Gupta [Sat, 10 Sep 2022 18:31:28 +0000 (19:31 +0100)] 
YARN-11265. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-sharedcachemanager (#4772)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
3 weeks agoYARN-11286. Make AsyncDispatcher#printEventDetailsExecutor thread pool parameter...
slfan1989 [Sat, 10 Sep 2022 17:30:50 +0000 (01:30 +0800)] 
YARN-11286. Make AsyncDispatcher#printEventDetailsExecutor thread pool parameter configurable. (#4824). Contributed by fanshilun.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
3 weeks agoYARN-11274. Impove Nodemanager#NodeStatusUpdaterImpl Log. (#4783). Contributed by...
slfan1989 [Sat, 10 Sep 2022 17:27:06 +0000 (01:27 +0800)] 
YARN-11274. Impove Nodemanager#NodeStatusUpdaterImpl Log. (#4783). Contributed by fanshilun.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
3 weeks agoHDFS-13522: Add federated nameservices states to client protocol and propagate it...
Simbarashe Dzinamarira [Wed, 7 Sep 2022 17:54:41 +0000 (10:54 -0700)] 
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.

Fixes #4311

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
3 weeks agoYARN-11297. [Federation] Improve Yarn Router Reservation Submission Code. (#4863)
slfan1989 [Fri, 9 Sep 2022 17:39:00 +0000 (01:39 +0800)] 
YARN-11297. [Federation] Improve Yarn Router Reservation Submission Code. (#4863)

3 weeks agoHADOOP-18439. Fix VectoredIO for LocalFileSystem when checksum is enabled. (#4862)
Mukund Thakur [Fri, 9 Sep 2022 16:16:08 +0000 (21:46 +0530)] 
HADOOP-18439. Fix VectoredIO for LocalFileSystem when checksum is enabled. (#4862)

part of HADOOP-18103.

While merging the ranges in CheckSumFs, they are rounded up based on the
value of checksum bytes size which leads to some ranges crossing the EOF
thus they need to be fixed else it will cause EOFException during actual reads.

Contributed By: Mukund Thakur

3 weeks agoYARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh...
9uapaw [Fri, 9 Sep 2022 12:38:41 +0000 (14:38 +0200)] 
YARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh Gupta.

3 weeks agoHADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting...
Viraj Jasani [Fri, 9 Sep 2022 10:32:20 +0000 (03:32 -0700)] 
HADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting work (#4796)

Contributed by Viraj Jasani

3 weeks agoHDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry...
ZanderXu [Fri, 9 Sep 2022 07:17:36 +0000 (15:17 +0800)] 
HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
3 weeks agoYARN-11298. Improve Yarn Router Junit Test Close MockRM. (#4870)
slfan1989 [Thu, 8 Sep 2022 18:42:36 +0000 (02:42 +0800)] 
YARN-11298. Improve Yarn Router Junit Test Close MockRM. (#4870)

3 weeks agoHADOOP-18427. Improve ZKDelegationTokenSecretManager#startThead With recommended...
slfan1989 [Thu, 8 Sep 2022 18:41:21 +0000 (02:41 +0800)] 
HADOOP-18427. Improve ZKDelegationTokenSecretManager#startThead With recommended methods. (#4812)

3 weeks agoHADOOP-18416. fix ITestS3AIOStatisticsContext test failure (#4806)
Mehakmeet Singh [Thu, 8 Sep 2022 15:33:18 +0000 (21:03 +0530)] 
HADOOP-18416. fix ITestS3AIOStatisticsContext test failure (#4806)

Follow on to HADOOP-17461.

Contributed by: Mehakmeet Singh

3 weeks agoHADOOP-18443. Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 (#4856)
Ashutosh Gupta [Thu, 8 Sep 2022 14:28:38 +0000 (15:28 +0100)] 
HADOOP-18443. Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 (#4856)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Brahma Reddy Battula <brahma@apache.org>
3 weeks agoHADOOP-18441. Remove hadoop custom ServicesResourceTransformer (#4850). Contributed...
PJ Fanning [Wed, 7 Sep 2022 11:41:12 +0000 (12:41 +0100)] 
HADOOP-18441. Remove hadoop custom ServicesResourceTransformer (#4850). Contributed by PJ Fanning.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
3 weeks agoHADOOP-18426. Use weighted calculation for MutableStat mean/variance to fix accuracy...
Erik Krogen [Wed, 7 Sep 2022 05:49:56 +0000 (22:49 -0700)] 
HADOOP-18426. Use weighted calculation for MutableStat mean/variance to fix accuracy. (#4844). Contributed by Erik Krogen.

Co-authored-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
3 weeks agoHADOOP-18388. Allow dynamic groupSearchFilter in LdapGroupsMapping. (#4798)
Ayush Saxena [Tue, 6 Sep 2022 22:38:51 +0000 (04:08 +0530)] 
HADOOP-18388. Allow dynamic groupSearchFilter in LdapGroupsMapping. (#4798)

* HADOOP-18388. Allow dynamic groupSearchFilter in LdapGroupsMapping.

3 weeks agoHDFS-16659. JournalNode should throw NewerTxnIdException when SinceTxId is bigger...
ZanderXu [Tue, 6 Sep 2022 17:12:55 +0000 (01:12 +0800)] 
HDFS-16659. JournalNode should throw NewerTxnIdException when SinceTxId is bigger than HighestWrittenTxId (#4560)

Co-authored-by: Zander Xu <zengqiang.xu@shopee.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
3 weeks agoHADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfs...
Sumangala Patki [Tue, 6 Sep 2022 10:00:52 +0000 (15:30 +0530)] 
HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3699)

Successor for the reverted PR #3341, using the hadoop @VisibleForTesting attribute

Contributed by Sumangala Patki

3 weeks agoHDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id... HDFS-2139
ZanderXu [Mon, 5 Sep 2022 12:16:15 +0000 (20:16 +0800)] 
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF (#4813). Contributed by ZanderXu.

Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
3 weeks agoHDFS-16593. Correct the BlocksRemoved metric on DataNode side (#4353). Contributed...
ZanderXu [Mon, 5 Sep 2022 11:35:48 +0000 (19:35 +0800)] 
HDFS-16593. Correct the BlocksRemoved metric on DataNode side (#4353). Contributed by ZanderXu.

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
4 weeks agoYARN-11289. [Federation] Improve NM FederationInterceptor removeAppFromRegistry....
slfan1989 [Fri, 2 Sep 2022 17:41:31 +0000 (01:41 +0800)] 
YARN-11289. [Federation] Improve NM FederationInterceptor removeAppFromRegistry. (#4836)

4 weeks agoYARN-11273. Federation StateStore: Support storage/retrieval of Reservations With...
slfan1989 [Fri, 2 Sep 2022 17:39:58 +0000 (01:39 +0800)] 
YARN-11273. Federation StateStore: Support storage/retrieval of Reservations With SQL. (#4817)

4 weeks agoYARN-11284. [Federation] Improve UnmanagedAMPoolManager WithoutBlock ServiceStop...
slfan1989 [Fri, 2 Sep 2022 17:28:38 +0000 (01:28 +0800)] 
YARN-11284. [Federation] Improve UnmanagedAMPoolManager WithoutBlock ServiceStop (#4814)

4 weeks agoYARN-6667. Handle containerId duplicate without failing the heartbeat in Federation...
slfan1989 [Fri, 2 Sep 2022 17:25:26 +0000 (01:25 +0800)] 
YARN-6667. Handle containerId duplicate without failing the heartbeat in Federation Interceptor. (#4810)

4 weeks agoHDFS-16750. NameNode should use NameNode.getRemoteUser() to log audit event to avoid...
ZanderXu [Fri, 2 Sep 2022 17:23:03 +0000 (01:23 +0800)] 
HDFS-16750. NameNode should use NameNode.getRemoteUser() to log audit event to avoid possible NPE (#4821)

4 weeks agoHADOOP-18408. ABFS: ITestAbfsManifestCommitProtocol fails on nonHNS configuration...
sreeb-msft [Fri, 2 Sep 2022 11:33:12 +0000 (17:03 +0530)] 
HADOOP-18408. ABFS: ITestAbfsManifestCommitProtocol  fails on nonHNS configuration  (#4758)

ITestAbfsManifestCommitProtocol  to set requireRenameResilience to false for nonHNS configuration  (#4758)

Contributed by Sree Bhattacharyya

4 weeks agoYARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federat...
slfan1989 [Thu, 1 Sep 2022 23:35:20 +0000 (07:35 +0800)] 
YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation. (#4764)

4 weeks agoHADOOP-18339. S3A storage class option only picked up when buffering writes to disk...
monthonk [Thu, 1 Sep 2022 17:14:32 +0000 (18:14 +0100)] 
HADOOP-18339. S3A storage class option only picked up when buffering writes to disk. (#4669)

Follow-up to HADOOP-12020 Support configuration of different S3 storage classes;
S3 storage class is now set when buffering to heap/bytebuffers, and when
creating directory markers

Contributed by Monthon Klongklaew

4 weeks agoHDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected...
Steve Vaughan [Thu, 1 Sep 2022 13:00:15 +0000 (09:00 -0400)] 
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution (#4833)

Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.

Contributed by Steve Vaughan Jr

4 weeks agoYARN-11272. Federation StateStore: Support storage/retrieval of Reservations With...
slfan1989 [Wed, 31 Aug 2022 17:15:15 +0000 (01:15 +0800)] 
YARN-11272. Federation StateStore: Support storage/retrieval of Reservations With Zk. (#4781)

4 weeks agoHADOOP-18391. Improvements in VectoredReadUtils#readVectored() for direct buffers...
Mukund Thakur [Wed, 31 Aug 2022 16:11:41 +0000 (21:41 +0530)] 
HADOOP-18391. Improvements in VectoredReadUtils#readVectored() for direct buffers (#4787)

part of HADOOP-18103.

Contributed By: Mukund Thakur

4 weeks agoMAPREDUCE-7409. Make shuffle key length configurable. Contributed by Ashutosh Gupta.
9uapaw [Wed, 31 Aug 2022 15:30:24 +0000 (17:30 +0200)] 
MAPREDUCE-7409. Make shuffle key length configurable. Contributed by Ashutosh Gupta.

4 weeks agoHADOOP-18410. S3AInputStream.unbuffer() does not release http connections (#4766)
Steve Loughran [Wed, 31 Aug 2022 10:16:52 +0000 (11:16 +0100)] 
HADOOP-18410. S3AInputStream.unbuffer() does not release http connections (#4766)

HADOOP-16202 "Enhance openFile()" added asynchronous draining of the
remaining bytes of an S3 HTTP input stream for those operations
(unbuffer, seek) where it could avoid blocking the active
thread.

This patch fixes the asynchronous stream draining to work and so
return the stream back to the http pool. Without this, whenever
unbuffer() or seek() was called on a stream and an asynchronous
drain triggered, the connection was not returned; eventually
the pool would be empty and subsequent S3 requests would
fail with the message "Timeout waiting for connection from pool"

The root cause was that even though the fields passed in to drain() were
converted to references through the methods, in the lambda expression
passed in to submit, they were direct references

operation = client.submit(
 () -> drain(uri, streamStatistics,
       false, reason, remaining,
       object, wrappedStream));  /* here */

Those fields were only read during the async execution, at which
point they would have been set to null (or even a subsequent read).

A new SDKStreamDrainer class peforms the draining; this is a Callable
and can be submitted directly to the executor pool.

The class is used in both the classic and prefetching s3a input streams.

Also, calling unbuffer() switches the S3AInputStream from adaptive
to random IO mode; that is, it is considered a cue that future
IO will not be sequential, whole-file reads.

Contributed by Steve Loughran.

4 weeks agoHADOOP-18428. Parameterize platform toolset version (#4815)
Gautham B A [Tue, 30 Aug 2022 17:11:03 +0000 (22:41 +0530)] 
HADOOP-18428. Parameterize platform toolset version (#4815)

* This PR adds an option
  use.platformToolsetVersion that
  makes the build systems to use
  this platform toolset version.
* This also makes sure that
  win-vs-upgrade.cmd does not get
  executed when the
  use.platformToolsetVersion
  option is specified.

4 weeks agoYARN-11287. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactor...
slfan1989 [Tue, 30 Aug 2022 11:41:04 +0000 (19:41 +0800)] 
YARN-11287. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-10793 (#4828)

Co-authored-by: slfan1989 <louj1988@@>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
4 weeks agoHADOOP-18375. Fix failure of shelltest for hadoop_add_ldlibpath. (#4652)
Masatake Iwasaki [Tue, 30 Aug 2022 10:33:29 +0000 (19:33 +0900)] 
HADOOP-18375. Fix failure of shelltest for hadoop_add_ldlibpath. (#4652)

4 weeks agoYARN-11245. Upgrade JUnit from 4 to 5 in hadoop-yarn-csi (#4778)
Ashutosh Gupta [Tue, 30 Aug 2022 08:26:06 +0000 (09:26 +0100)] 
YARN-11245. Upgrade JUnit from 4 to 5 in hadoop-yarn-csi (#4778)

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
4 weeks agoYARN-11196. NUMA support in DefaultContainerExecutor (#4742)
Samrat [Tue, 30 Aug 2022 05:09:41 +0000 (10:39 +0530)] 
YARN-11196. NUMA support in DefaultContainerExecutor  (#4742)

4 weeks agoHDFS-16735. Reduce the number of HeartbeatManager loops. (#4780). Contributed by...
zhangshuyan0 [Mon, 29 Aug 2022 03:30:21 +0000 (11:30 +0800)] 
HDFS-16735. Reduce the number of HeartbeatManager loops. (#4780). Contributed by Shuyan Zhang.

Signed-off-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
5 weeks agoYARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager. (#4792)
slfan1989 [Sat, 27 Aug 2022 17:17:00 +0000 (01:17 +0800)] 
YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager. (#4792)

5 weeks agoYARN-8482. [Router] Add cache for fast answers to getApps. (#4769)
slfan1989 [Sat, 27 Aug 2022 17:14:55 +0000 (01:14 +0800)] 
YARN-8482. [Router] Add cache for fast answers to getApps. (#4769)

5 weeks agoYARN-11253. Add Configuration to delegationToken RemoverScanInterval. (#4751)
slfan1989 [Sat, 27 Aug 2022 17:02:59 +0000 (01:02 +0800)] 
YARN-11253. Add Configuration to delegationToken RemoverScanInterval. (#4751)

5 weeks agoHDFS-16734. RBF: fix some bugs when handling getContentSummary RPC (#4763)
ZanderXu [Fri, 26 Aug 2022 23:04:33 +0000 (07:04 +0800)] 
HDFS-16734. RBF: fix some bugs when handling getContentSummary RPC (#4763)

5 weeks agoYARN-11219. [Federation] Add getAppActivities, getAppStatistics REST APIs for Router...
slfan1989 [Fri, 26 Aug 2022 23:01:17 +0000 (07:01 +0800)] 
YARN-11219. [Federation] Add getAppActivities, getAppStatistics REST APIs for Router. (#4757)

5 weeks agoHDFS-16736. Link to Boost library in libhdfspp (#4782)
Gautham B A [Fri, 26 Aug 2022 16:11:44 +0000 (21:41 +0530)] 
HDFS-16736. Link to Boost library in libhdfspp (#4782)

5 weeks agoHDFS-16732. [SBN READ] Avoid get location from observer when the block report is...
zhengchenyu [Thu, 25 Aug 2022 17:37:25 +0000 (01:37 +0800)] 
HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756)

Signed-off-by: Erik Krogen <xkrogen@apache.org>
5 weeks agoHADOOP-18382. AWS SDK v2 upgrade prerequisites (#4698)
ahmarsuhail [Thu, 25 Aug 2022 16:36:48 +0000 (17:36 +0100)] 
HADOOP-18382. AWS SDK v2 upgrade prerequisites (#4698)

This patch prepares the hadoop-aws module for a future
migration to using the v2 AWS SDK (HADOOP-18073)

That upgrade will be incompatible; this patch prepares
for it:
-marks some credential providers and other
 classes and methods as @deprecated.
-updates site documentation
-reduces the visibility of the s3 client;
 other than for testing, it is kept private to
 the S3AFileSystem class.
-logs some warnings when deprecated APIs are used.

The warning messages are printed only once
per JVM's life. To disable them, set the
log level of org.apache.hadoop.fs.s3a.SDKV2Upgrade
to ERROR

Contributed by Ahmar Suhail

5 weeks agoHDFS-16738. Invalid CallerContext caused NullPointerException (#4791)
ZanderXu [Thu, 25 Aug 2022 09:12:27 +0000 (17:12 +0800)] 
HDFS-16738. Invalid CallerContext caused NullPointerException (#4791)

5 weeks agoRevert "HADOOP-18417. Upgrade to M7 of surefire plugin (#4795)"
Ayush Saxena [Wed, 24 Aug 2022 22:14:49 +0000 (03:44 +0530)] 
Revert "HADOOP-18417. Upgrade to M7 of surefire plugin (#4795)"

This reverts commit 1ff121041c647f0a7980b59e06335faac9dc2f45.

5 weeks agoHDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices (#4734)...
ZanderXu [Wed, 24 Aug 2022 14:57:15 +0000 (22:57 +0800)] 
HDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices (#4734). Contributed by ZanderXu.

Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
5 weeks agoYARN-11240. Fix incorrect placeholder in yarn-module. (#4678). Contributed by fanshilun
slfan1989 [Wed, 24 Aug 2022 14:36:36 +0000 (22:36 +0800)] 
YARN-11240. Fix incorrect placeholder in yarn-module. (#4678). Contributed by fanshilun

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
5 weeks agoHADOOP-18361. Update commons-net from 3.6 to 3.8.0. (#4683). Contributed by fanshilun.
slfan1989 [Wed, 24 Aug 2022 14:35:17 +0000 (22:35 +0800)] 
HADOOP-18361. Update commons-net from 3.6 to 3.8.0. (#4683). Contributed by fanshilun.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>