bigtop.git
3 hours agoBIGTOP-3827. Skip dh_strip_nondeterminism for HBase, Phoenix and Zeppelin. (#1024) master
Masatake Iwasaki [Mon, 26 Sep 2022 08:26:31 +0000 (17:26 +0900)] 
BIGTOP-3827. Skip dh_strip_nondeterminism for HBase, Phoenix and Zeppelin. (#1024)

3 days agoBIGTOP-3819. Adding Puppet Recipes for Apache Phoenix (#1018)
Peng Lee [Fri, 23 Sep 2022 03:42:54 +0000 (11:42 +0800)] 
BIGTOP-3819. Adding Puppet Recipes for Apache Phoenix (#1018)

4 days agoBIGTOP-3812: Building bigtop-ambari-mpack fails on Debian/Ubuntu due to unwanted...
Yuqi Gu [Thu, 22 Sep 2022 01:57:51 +0000 (09:57 +0800)] 
BIGTOP-3812: Building bigtop-ambari-mpack fails on Debian/Ubuntu due to unwanted binary files (#1009)

Building bigtop-ambari-mpack fails on Debian/Ubuntu due to unwanted binary files.
We will move 'dev-support' to Bigtop provisioner and rename it to bigtop-stack-provisioner.
It is used for Mpack provisioner which is the test tool for Mpack developers.
After 3.2 release, we it will be used for Ambari UI test framework and Ambair-bigtop-demo.

Change-Id: Ibe310d7ab979c055eacfc8e14daa27ca6aa415e3
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
5 days agoBIGTOP-3818. Phoenix Don't Requre Hadoop and Zookeeper Installed (#1017)
Peng Lee [Tue, 20 Sep 2022 15:17:59 +0000 (23:17 +0800)] 
BIGTOP-3818. Phoenix Don't Requre Hadoop and Zookeeper Installed (#1017)

7 days agoBIGTOP-3815. Fix failure of installing hadoop-httpfs deb due to chagen of httpfs...
Masatake Iwasaki [Mon, 19 Sep 2022 03:50:24 +0000 (12:50 +0900)] 
BIGTOP-3815. Fix failure of installing hadoop-httpfs deb due to chagen of httpfs home dir. (#1011)

10 days agoBIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled (#1000)
timyuer [Fri, 16 Sep 2022 04:21:20 +0000 (12:21 +0800)] 
BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled (#1000)

10 days agoBIGTOP-3804: Add quicklinks for HDFS/YARN/MapReduce (#1013)
yaolei [Fri, 16 Sep 2022 03:15:21 +0000 (11:15 +0800)] 
BIGTOP-3804: Add quicklinks for HDFS/YARN/MapReduce (#1013)

* BIGTOP-3804: Add quicklinks for HDFS/YARN/MapReduce

* Improve QuickLink Label

11 days agoBIGTOP-3623. dh_strip_nondeterminism spends too much time on deb packaging. (#1012)
Masatake Iwasaki [Thu, 15 Sep 2022 10:33:45 +0000 (19:33 +0900)] 
BIGTOP-3623. dh_strip_nondeterminism spends too much time on deb packaging. (#1012)

11 days agoBIGTOP-3814. Deploying Spark ThriftServer fails because it's launched too early....
Kengo Seki [Thu, 15 Sep 2022 07:19:07 +0000 (16:19 +0900)] 
BIGTOP-3814. Deploying Spark ThriftServer fails because it's launched too early. (#1008)

12 days agoBIGTOP-3803. Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when...
Masatake Iwasaki [Wed, 14 Sep 2022 09:19:44 +0000 (18:19 +0900)] 
BIGTOP-3803. Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when kerberos enabled. (#1003)

12 days agoBIGTOP-3803: Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when...
timyuer [Wed, 14 Sep 2022 09:18:53 +0000 (17:18 +0800)] 
BIGTOP-3803: Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when kerberos enabled (#1001)

* BIGTOP-3803: Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when kerberos enabled

* BIGTOP-3803: Update as iwasakims comment

12 days agoBIGTOP-3810: Fix wrong home configuration for HTTPFS on build scripts (#1007)
Zhiguo Wu [Wed, 14 Sep 2022 05:36:15 +0000 (13:36 +0800)] 
BIGTOP-3810: Fix wrong home configuration for HTTPFS on build scripts (#1007)

13 days agoBIGTOP-3792: Mpack solr fail to start when kerberos enabled (#1005)
timyuer [Tue, 13 Sep 2022 06:00:00 +0000 (14:00 +0800)] 
BIGTOP-3792: Mpack solr fail to start when kerberos enabled (#1005)

13 days agoBIGTOP-3799: Support parent directory configuration for Hive rpm build script (#998)
Zhiguo Wu [Tue, 13 Sep 2022 05:57:37 +0000 (13:57 +0800)] 
BIGTOP-3799: Support parent directory configuration for Hive rpm build script (#998)

* BIGTOP-3799: Support parent directory configuration for Hive rpm build script

* remove docdir prefix on deb build script

13 days agoBIGTOP-3798: Support parent directory configuration for Zeppelin rpm build script...
Zhiguo Wu [Tue, 13 Sep 2022 04:17:14 +0000 (12:17 +0800)] 
BIGTOP-3798: Support parent directory configuration for Zeppelin rpm build script (#997)

13 days agoBIGTOP-3809: Fix tez deb package build fails (#1006)
Zhiguo Wu [Tue, 13 Sep 2022 00:57:01 +0000 (08:57 +0800)] 
BIGTOP-3809: Fix tez deb package build fails (#1006)

13 days agoBIGTOP-3806: Support parent directory configuration for Flink rpm build script (...
Zhiguo Wu [Mon, 12 Sep 2022 13:50:03 +0000 (21:50 +0800)] 
BIGTOP-3806: Support parent directory configuration for Flink rpm build script (#1004)

* BIGTOP-3806: Support parent directory configuration for Flink rpm build script

* change hadoop dir

2 weeks agoBIGTOP-3790: Add symlink for Tez conf dir (#990)
Zhiguo Wu [Mon, 12 Sep 2022 02:31:23 +0000 (10:31 +0800)] 
BIGTOP-3790: Add symlink for Tez conf dir (#990)

* BIGTOP-3790: Add symlink for tez conf dir

* doc tez dir changes

2 weeks agoBIGTOP-3797: Support parent directory configuration for Kafka rpm build script (...
Zhiguo Wu [Mon, 12 Sep 2022 01:45:56 +0000 (09:45 +0800)] 
BIGTOP-3797: Support parent directory configuration for Kafka rpm build script (#996)

* BIGTOP-3797: Support parent directory configuration for Kafka rpm build script

* fix conf symlink dir not found

2 weeks agoBIGTOP-3796: Support parent directory configuration for Zookeeper rpm build script...
Zhiguo Wu [Mon, 12 Sep 2022 00:04:02 +0000 (08:04 +0800)] 
BIGTOP-3796: Support parent directory configuration for Zookeeper rpm build script (#995)

* BIGTOP-3796: Support parent directory configuration for Zookeeper rpm build script

* fix conf symlink

2 weeks agoBIGTOP-3756: Add Fedora 36 option (#961)
Leona Yoda [Fri, 9 Sep 2022 14:17:39 +0000 (23:17 +0900)] 
BIGTOP-3756: Add Fedora 36 option (#961)

* Add Fedora 36 support
- use java 8 during build process because on fedora 36/x86, java 17 is also installed and set as default

* opt-out from default build configuration on fedora 36

* add initscripts installation for fedora 36

2 weeks agoBIGTOP-3741:Add Flink support for Bigtop 3.1.0 Mpack (#993)
yaolei [Fri, 9 Sep 2022 10:30:38 +0000 (18:30 +0800)] 
BIGTOP-3741:Add Flink support for Bigtop 3.1.0 Mpack (#993)

BIGTOP-3741:Add Flink support for Bigtop 3.1.0 Mpack
* improve role_command_order.json
* add log4j configuration
* improve flink service check

2 weeks agoBIGTOP-3791: Replace log4j with log4j2 for Hive package (#992)
Zhiguo Wu [Tue, 6 Sep 2022 09:07:29 +0000 (17:07 +0800)] 
BIGTOP-3791: Replace log4j with log4j2 for Hive package (#992)

3 weeks agoBIGTOP-3801. Add maven cache volume for docker provisioner (#999)
Leona Yoda [Sat, 3 Sep 2022 07:49:08 +0000 (16:49 +0900)] 
BIGTOP-3801. Add maven cache volume for docker provisioner (#999)

3 weeks agoBIGTOP-3795: Add parameter to configure parent directory for RPM packages (#994)
Zhiguo Wu [Thu, 1 Sep 2022 12:57:06 +0000 (20:57 +0800)] 
BIGTOP-3795: Add parameter to configure parent directory for RPM packages (#994)

3 weeks agoBIGTOP-3789: Decouple Kerberos services from Ambari Core (#991)
Yuqi Gu [Wed, 31 Aug 2022 04:33:38 +0000 (12:33 +0800)] 
BIGTOP-3789: Decouple Kerberos services from Ambari Core (#991)

Upgrade Kerberos service from 1.10.3-10 to 1.10.3-30.
And fix deployment issues:
https://github.com/apache/bigtop/pull/989#issuecomment-1229959105

Change-Id: I61347c98f47ab109449d52ae84ef7a4352033d8b
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
3 weeks agoBIGTOP-3788: Fix Ambari-HDFS components fail to start when kerberos enabled (#989) 987/head
timyuer [Tue, 30 Aug 2022 08:34:26 +0000 (16:34 +0800)] 
BIGTOP-3788: Fix Ambari-HDFS components fail to start when kerberos enabled (#989)

* BIGTOP-3788: Fix Ambari-HDFS components fail to start when kerberos enabled

* Fix fail to run MapReduce2 service check

* Adding README for Kerberos configuration

Co-authored-by: root <root@bdp92.qa.jndv.org>
4 weeks agoBIGTOP-3787. Add docs in Phoenix .deb package (#988)
Shuhei Yamasaki [Sun, 28 Aug 2022 01:18:06 +0000 (10:18 +0900)] 
BIGTOP-3787. Add docs in Phoenix .deb package (#988)

4 weeks agoBIGTOP-3770: Remove Sqoop from Bigtop stack (#981)
Yuqi Gu [Wed, 24 Aug 2022 03:23:31 +0000 (11:23 +0800)] 
BIGTOP-3770: Remove Sqoop from Bigtop stack (#981)

Change-Id: If91a7b57f59e1d50b9fbb409ccdd2d3654220452
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
5 weeks agoBIGTOP-3777: Add Spark support for Bigtop 3.1.0 Mpack (#985)
timyuer [Mon, 22 Aug 2022 09:41:42 +0000 (17:41 +0800)] 
BIGTOP-3777: Add Spark support for Bigtop 3.1.0 Mpack (#985)

Add Spark support for Bigtop 3.1.0 Mpack
Fix spark_thrift_server in params.py

5 weeks agoBIGTOP-3738: Add Hive support for Bigtop 3.1.0 Mpack (#965)
timyuer [Tue, 16 Aug 2022 06:58:27 +0000 (14:58 +0800)] 
BIGTOP-3738: Add Hive support for Bigtop 3.1.0 Mpack (#965)

* BIGTOP-3738: Add Hive support and deployment script for Bigtop Mpack

Co-authored-by: houyu <houyu@jndv.com>
Co-authored-by: root <root@bdp92.qa.jndv.org>
5 weeks agoBIGTOP-3778: Fix Hive3.1.3 WebHCat service compatible issue with Hadoop3.3.x (#983)
吴治国 [Tue, 16 Aug 2022 06:55:38 +0000 (14:55 +0800)] 
BIGTOP-3778: Fix Hive3.1.3 WebHCat service compatible issue with Hadoop3.3.x (#983)

* BIGTOP-3778: Fix Hive3.1.3 WebHCat service compatible issue with Hadoop3.3.x

Co-authored-by: wuzhiguo <datakun-open@startdt.com>
5 weeks agoBIGTOP-3776. spark-thriftserver should work without external hive-metastore (#982)
Peng Lee [Tue, 16 Aug 2022 04:17:02 +0000 (12:17 +0800)] 
BIGTOP-3776. spark-thriftserver should work without external hive-metastore (#982)

6 weeks agoBIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code (#973)
Peng Lee [Sun, 14 Aug 2022 06:05:03 +0000 (14:05 +0800)] 
BIGTOP-3767. Add Spark Thrift Server Puppet Deploy Code (#973)

6 weeks agoBIGTOP-3775: Failed to build Phoenix on Arm64/powerpc64le (#980)
Yuqi Gu [Fri, 12 Aug 2022 04:53:00 +0000 (12:53 +0800)] 
BIGTOP-3775: Failed to build Phoenix on Arm64/powerpc64le (#980)

Change-Id: Ibe38c50e6cbf83d1e89f3b35c8c4ee500d49d329
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
6 weeks agoBIGTOP-3771. Fix failure of sqlline.py bundled with Phoenix. (#977)
Masatake Iwasaki [Wed, 10 Aug 2022 12:27:42 +0000 (21:27 +0900)] 
BIGTOP-3771. Fix failure of sqlline.py bundled with Phoenix. (#977)

* BIGTOP-3771. Fix failure of sqlline.py bundled with Phoenix.

6 weeks agoBIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner (#978)
Evans Ye [Wed, 10 Aug 2022 08:56:22 +0000 (16:56 +0800)] 
BIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner (#978)

* BIGTOP-3774: Drop Docker Sandbox and Vagrant Provisioner

* BIGTOP-3774: address yuqi's comment

6 weeks agoBIGTOP-3772: Add Solr support for Bigtop 3.1.0 Mpack (#976)
吴治国 [Wed, 10 Aug 2022 07:23:30 +0000 (15:23 +0800)] 
BIGTOP-3772: Add Solr support for Bigtop 3.1.0 Mpack (#976)

6 weeks agoBIGTOP-3740: Add Zeppelin support for Bigtop 3.1.0 Mpack (#954)
吴治国 [Wed, 10 Aug 2022 05:01:01 +0000 (13:01 +0800)] 
BIGTOP-3740: Add Zeppelin support for Bigtop 3.1.0 Mpack (#954)

* BIGTOP-3740: Add Zeppelin support for Bigtop 3.1.0 Mpack

6 weeks agoBIGTOP-3752: WebHDFS api path encode space character (#953)
吴治国 [Wed, 10 Aug 2022 04:44:13 +0000 (12:44 +0800)] 
BIGTOP-3752: WebHDFS api path encode space character (#953)

7 weeks agoBIGTOP-3700: Bump Zeppelin to 0.10.1 (#975)
Yuqi Gu [Mon, 8 Aug 2022 06:26:54 +0000 (14:26 +0800)] 
BIGTOP-3700: Bump Zeppelin to 0.10.1 (#975)

Change-Id: If7f59b4825c6ea32407ca5ae3ba08e7d392d1a77
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
7 weeks agoBIGTOP-3760. addendum. reverting temporary download URL for testing RC.
Masatake Iwasaki [Mon, 8 Aug 2022 04:09:40 +0000 (04:09 +0000)] 
BIGTOP-3760. addendum. reverting temporary download URL for testing RC.

7 weeks agoBigtop 3.1.1 release
Kengo Seki [Sun, 7 Aug 2022 07:15:25 +0000 (16:15 +0900)] 
Bigtop 3.1.1 release

7 weeks agoBigtop 3.1.1 website update
Kengo Seki [Sun, 7 Aug 2022 05:19:35 +0000 (14:19 +0900)] 
Bigtop 3.1.1 website update

7 weeks ago[WIP]BIGTOP-3760. Bump Hadoop to 3.3.4. (#964)
Masatake Iwasaki [Fri, 5 Aug 2022 10:12:00 +0000 (19:12 +0900)] 
[WIP]BIGTOP-3760. Bump Hadoop to 3.3.4. (#964)

* BIGTOP-3760. Bump Hadoop to 3.3.4.

* addressed additional dependency on okhttp in hbase-shaded.

* rebased the patch1-relocate-okhttp.diff to HBase 2.4.13.

* RC0 was canceled and RC1 is now under vote.

7 weeks agoBIGTOP-3769: Upgrade Sqoop ivy version to 2.5.0 (#974)
吴治国 [Fri, 5 Aug 2022 06:54:51 +0000 (14:54 +0800)] 
BIGTOP-3769: Upgrade Sqoop ivy version to 2.5.0 (#974)

* BIGTOP-3769: Upgrade Sqoop ivy version to 2.5.0

* merge patch

Co-authored-by: wuzhiguo <datakun-open@startdt.com>
7 weeks agoBIGTOP-3768. Add Spark User Hdfs Folder for Deploy Spark Thrift Server (#972)
Peng Lee [Wed, 3 Aug 2022 04:44:21 +0000 (00:44 -0400)] 
BIGTOP-3768. Add Spark User Hdfs Folder for Deploy Spark Thrift Server (#972)

8 weeks agoBIGTOP-3727: Add HBase support for Bigtop 3.1.0 Mpack (#947)
yzydhf [Mon, 1 Aug 2022 08:46:58 +0000 (16:46 +0800)] 
BIGTOP-3727: Add HBase support for Bigtop 3.1.0 Mpack (#947)

* BIGTOP-3727: Add HBase support for Bigtop 3.1.0 Mpack

* remove phoenix dependency

Co-authored-by: wangxiangjun <wangxiangjun@datakun-0003.com>
8 weeks agoBIGTOP-3765: Remove cmake conditional for Hadoop rpm spec (#971)
Yuqi Gu [Mon, 1 Aug 2022 03:42:48 +0000 (11:42 +0800)] 
BIGTOP-3765: Remove cmake conditional for Hadoop rpm spec (#971)

Change-Id: I0e7f47512b7717103e97ae51d712bfee8fe716a9
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
8 weeks agoBIGTOP-3761. Fix bigtop_toolchain to enable EPEL before downloading packages (#969)
Shuhei Yamasaki [Sat, 30 Jul 2022 00:08:07 +0000 (09:08 +0900)] 
BIGTOP-3761. Fix bigtop_toolchain to enable EPEL before downloading packages (#969)

Co-authored-by: Shuhei Yamasaki <Shuhei.Yamasaki@oss.nttdata.com>
8 weeks agoBIGTOP-3763: Fix Hive build issues about protobuf 2.5.0 on Arm64 and PPC64le (#970)
Yuqi Gu [Fri, 29 Jul 2022 09:07:36 +0000 (17:07 +0800)] 
BIGTOP-3763: Fix Hive build issues about protobuf 2.5.0 on Arm64 and PPC64le (#970)

Change-Id: I59811ca32e70a8a2c8eb261e55b0e8470590b7ce
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3696. Bump Alluxio to 2.8.0 (#966)
Shuhei Yamasaki [Wed, 27 Jul 2022 02:10:23 +0000 (11:10 +0900)] 
BIGTOP-3696. Bump Alluxio to 2.8.0 (#966)

Co-authored-by: Shuhei Yamasaki <Shuhei.Yamasaki@oss.nttdata.com>
2 months agoBIGTOP-3754. Add Ubuntu 22.04 option (#956)
Leona Yoda [Tue, 26 Jul 2022 09:39:46 +0000 (18:39 +0900)] 
BIGTOP-3754. Add Ubuntu 22.04 option (#956)

2 months agoBIGTOP-3762: Add JVM options for spark build to avoid StackOverflowError (#967)
Leona Yoda [Tue, 26 Jul 2022 05:21:56 +0000 (14:21 +0900)] 
BIGTOP-3762: Add JVM options for spark build to avoid StackOverflowError (#967)

2 months agoBIGTOP-3758. Fix build failure of Tez against Hadoop 3.3. (#963)
Masatake Iwasaki [Mon, 25 Jul 2022 04:58:54 +0000 (13:58 +0900)] 
BIGTOP-3758. Fix build failure of Tez against Hadoop 3.3. (#963)

2 months agoBIGTOP-3755. Bump HBase to 2.4.13. (#960)
Masatake Iwasaki [Mon, 25 Jul 2022 03:57:05 +0000 (12:57 +0900)] 
BIGTOP-3755. Bump HBase to 2.4.13. (#960)

2 months agoBIGTOP-3759: Remove ELK from Bigtop 3.2.0 stack (#962)
吴治国 [Sun, 24 Jul 2022 00:54:41 +0000 (08:54 +0800)] 
BIGTOP-3759: Remove ELK from Bigtop 3.2.0 stack (#962)

2 months agoBIGTOP-3744. Puppet Deploy Hive on Tez add Resource Resizing Configs (#946)
Peng Lee [Sun, 24 Jul 2022 00:32:09 +0000 (20:32 -0400)] 
BIGTOP-3744. Puppet Deploy Hive on Tez add Resource Resizing Configs (#946)

2 months agoUpdate default values in hieradata for release 3.1.1
Kengo Seki [Sat, 23 Jul 2022 06:10:32 +0000 (15:10 +0900)] 
Update default values in hieradata for release 3.1.1

2 months agoBIGTOP-3750: Add Chrony synchronization to fix hostcheck issue in Mpack docker provis...
Yuqi Gu [Thu, 21 Jul 2022 09:47:35 +0000 (17:47 +0800)] 
BIGTOP-3750: Add Chrony synchronization to fix hostcheck issue in Mpack docker provisioner (#952)

Add Chrony synchronization to fix hostcheck issue in Mpack docker provisioner.
Also silence the shellcheck issues: SC2046 and SC2164.

Change-Id: I5e9609bde970d1ec590b13afb915551e4364e377
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3697: Bump HBase to 2.4.12 (#959)
Yuqi Gu [Thu, 21 Jul 2022 09:38:00 +0000 (17:38 +0800)] 
BIGTOP-3697: Bump HBase to 2.4.12 (#959)

Change-Id: I5353c27351006501fe9bec1d1f23f5953857dd91
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3708. Hive build error with Hadoop 3.3.3. (#958)
Masatake Iwasaki [Thu, 21 Jul 2022 08:36:53 +0000 (17:36 +0900)] 
BIGTOP-3708. Hive build error with Hadoop 3.3.3. (#958)

2 months agoBIGTOP-3737: Fix build failure of Sqoop due to unresolved dependency on javax.activat...
吴治国 [Thu, 21 Jul 2022 04:38:55 +0000 (12:38 +0800)] 
BIGTOP-3737: Fix build failure of Sqoop due to unresolved dependency on javax.activation-api (#957)

2 months agoBIGTOP-3753. Bump Solr to 8.11.2 (#955)
Shuhei Yamasaki [Tue, 19 Jul 2022 23:43:04 +0000 (08:43 +0900)] 
BIGTOP-3753. Bump Solr to 8.11.2 (#955)

2 months agoBIGTOP-3747. zeppelin includes plugins directory (#949)
Halim Kim [Tue, 19 Jul 2022 06:49:12 +0000 (15:49 +0900)] 
BIGTOP-3747. zeppelin includes plugins directory (#949)

* BIGTOP-3747. zeppelin includes plugins directory

* BIGTOP-3747 install_zeppelin.sh plugins dir added

Co-authored-by: halim-kim <halim.kim@navercorp.com>
2 months agoBIGTOP-3751. Fix build failure of Zeppelin due to dependency convergence error. ...
Masatake Iwasaki [Tue, 19 Jul 2022 06:47:56 +0000 (15:47 +0900)] 
BIGTOP-3751. Fix build failure of Zeppelin due to dependency convergence error. (#951)

* BIGTOP-3751. Fix build failure of Zeppelin due to dependency convergence error.

* bumped the release number.

2 months agoBIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack (#948)
timyuer [Mon, 18 Jul 2022 15:45:05 +0000 (23:45 +0800)] 
BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack (#948)

* BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Co-authored-by: houyu <houyu@jndv.com>
2 months agoBIGTOP-3742: Add tez.tar.gz to tez lib dir (#945)
吴治国 [Sat, 16 Jul 2022 03:07:35 +0000 (11:07 +0800)] 
BIGTOP-3742: Add tez.tar.gz to tez lib dir (#945)

(cherry picked from commit 3d0a8d4957a6e12105a5e24dcb9f643bd6b7fbba)

2 months agoBIGTOP-3734: Add Sqoop support for Bigtop 3.1.0 Mpack (#943)
timyuer [Thu, 14 Jul 2022 11:36:44 +0000 (19:36 +0800)] 
BIGTOP-3734: Add Sqoop support for Bigtop 3.1.0 Mpack (#943)

Co-authored-by: houyu <houyu@jndv.com>
2 months agoBIGTOP-3735: Fix sqoop cannot found StringUtils from common-lang (#944)
吴治国 [Thu, 14 Jul 2022 03:26:14 +0000 (11:26 +0800)] 
BIGTOP-3735: Fix sqoop cannot found StringUtils from common-lang (#944)

(cherry picked from commit 2b13a53e64b79aaa113c5553577258dfb857c25e)

2 months agoBIGTOP-3728: Docker dev/test env for centos7 (#940)
吴治国 [Tue, 12 Jul 2022 10:13:59 +0000 (18:13 +0800)] 
BIGTOP-3728: Docker dev/test env for centos7 (#940)

* BIGTOP-3728: Docker dev/test env for centos7

2 months agoBIGTOP-3733. Building Logstash fails on Rocky 8 (#941)
Shuhei Yamasaki [Tue, 12 Jul 2022 04:46:02 +0000 (13:46 +0900)] 
BIGTOP-3733. Building Logstash fails on Rocky 8 (#941)

2 months agoBIGTOP-3721: Add Kafka support for Bigtop 3.1.0 Mpack (#930)
吴治国 [Fri, 8 Jul 2022 08:11:50 +0000 (16:11 +0800)] 
BIGTOP-3721: Add Kafka support for Bigtop 3.1.0 Mpack (#930)

* BIGTOP-3689: Add Kafka support for Bigtop 3.1.0 Mpack

2 months agoBIGTOP-3717: Update MapReduce service configuration in Mpack (#933)
Yuqi Gu [Fri, 8 Jul 2022 06:46:49 +0000 (14:46 +0800)] 
BIGTOP-3717: Update MapReduce service configuration in Mpack (#933)

Mapred configuration macros are deprected (HADOOP_MAPRED_ROOT_LOGGER,
HADOOP_JOB_HISTORYSERVER_OPTS, HADOOP_MAPRED_NICENESS, etc.).

Update them in Mpack aligned to
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/bin/mapred-config.sh#L38-L54

Change-Id: Id2f105d712119e8a930927b99e39b23ed74414a1
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3718: Fix MapReduce2 service-check in Amabri Mpack (#932)
Yuqi Gu [Fri, 8 Jul 2022 06:46:02 +0000 (14:46 +0800)] 
BIGTOP-3718: Fix MapReduce2 service-check in Amabri Mpack (#932)

The patch is to fix the issues of checking MapReduce2 service and
"JAR does not exist".

Change-Id: I59ad4d3ecbe88af7bea25bf7177502ebb3e2cf6b
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3724. Increment the release number of ZooKeeper and Kafka. (#938)
Masatake Iwasaki [Fri, 8 Jul 2022 05:22:20 +0000 (14:22 +0900)] 
BIGTOP-3724. Increment the release number of ZooKeeper and Kafka. (#938)

2 months agoBIGTOP-3719: Decouple Mpack HDFS services from Ambari Core (#931)
Yuqi Gu [Fri, 8 Jul 2022 05:11:31 +0000 (13:11 +0800)] 
BIGTOP-3719: Decouple Mpack HDFS services from Ambari Core (#931)

Decouple HDFS services from Ambari Core and fix the issues of HDFS
deployment from Bigtop 3.x repo. It's convenient for us to modify/upgrade
Mpack without re-building and re-deploying Ambari respectively.

Change-Id: I2fc2036c9fd261f082cc5359a827904df53a1d63
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3725. Puppet 3 is removed from EPEL for CentOS 7. (#939)
Kengo Seki [Fri, 8 Jul 2022 03:43:35 +0000 (12:43 +0900)] 
BIGTOP-3725. Puppet 3 is removed from EPEL for CentOS 7. (#939)

2 months agoBIGTOP-3716: Rename the Yarn/MapReduce historyserver PID file (#934)
Yuqi Gu [Wed, 6 Jul 2022 13:34:33 +0000 (21:34 +0800)] 
BIGTOP-3716: Rename the Yarn/MapReduce historyserver PID file (#934)

The naming rule of Yarn/MapReduce historyserver PID file
is deprecated in Hadoop 3.x.

Change-Id: I9170ff6047f1be593279ef9c9f71dcb09614d7ef
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-2971. RPM packaging for Ranger. (#929)
Kengo Seki [Tue, 5 Jul 2022 06:30:26 +0000 (15:30 +0900)] 
BIGTOP-2971. RPM packaging for Ranger. (#929)

This PR is based on #326 submitted by @Xingwd (much appreciated!) and
includes the following updates:

* Upgrade Ranger version to 2.2.0

* Change the download site from ASF to GitHub, because the source
  tarball distributed on the former is unnecessarily large

* Add package dependencies to bigtop.bom

* Remove the assembly:assembly target from the mvn command since it
  doesn't work in recent releases

* Add the version info of other components to the mvn command

* Fix the lib paths in install_ranger.sh and the RPM spec file

* Add the #BIGTOP_PATCH_FILES and #BIGTOP_PATCH_COMMANDS comments to the
  RPM spec so that the patches provided by Bigtop are applied during
  building RPM

* Add patches for addressing RANGER-3373 and RANGER-3459

* Remove symlink to non-existent ojdbc6.jar in install_ranger.sh

* Replace the hard-coded paths to /usr/lib/rpm/redhat in the RPM spec
  with %{_rpmconfigdir} (cf. BIGTOP-1344)

* Define debug_package=nil in the RPM spec to avoid build failure on
  Fedora 35

2 months agoBIGTOP-3715. Add TestWebHDFS to smoke-tests. (#928)
Masatake Iwasaki [Mon, 4 Jul 2022 09:49:36 +0000 (18:49 +0900)] 
BIGTOP-3715. Add TestWebHDFS to smoke-tests. (#928)

* BIGTOP-3715. Add TestWebHDFS to smoke-tests.

* fixed typo.e

2 months agoBIGTOP-3711: Add read/write smoke tests for Kafka (#922)
Leona Yoda [Mon, 4 Jul 2022 09:04:47 +0000 (18:04 +0900)] 
BIGTOP-3711: Add read/write smoke tests for Kafka (#922)

* Add read/write test for a kafka topic

* reorder import statements

* add a timeout option to consumer

2 months agoBIGTOP-3712: Fix Kafka smoke test failure with Zookeeper built with maven (#925)
Leona Yoda [Thu, 30 Jun 2022 11:37:54 +0000 (20:37 +0900)] 
BIGTOP-3712: Fix Kafka smoke test failure with Zookeeper built with maven (#925)

2 months agoBIGTOP-3710. Fix build failure of HBase on aarch64 due to error in check-aggregate...
Masatake Iwasaki [Wed, 29 Jun 2022 08:14:53 +0000 (17:14 +0900)] 
BIGTOP-3710. Fix build failure of HBase on aarch64 due to error in check-aggregate-license. (#921)

2 months agoBIGTOP-3709. Bring protobuf 2.5.0 back for building HBase on aarch64 and ppc64le...
Masatake Iwasaki [Wed, 29 Jun 2022 03:16:18 +0000 (12:16 +0900)] 
BIGTOP-3709. Bring protobuf 2.5.0 back for building HBase on aarch64 and ppc64le. (#920)

2 months agoBIGTOP-3704: Update Yarn service configuration in Mpack (#917)
Yuqi Gu [Wed, 29 Jun 2022 03:11:03 +0000 (11:11 +0800)] 
BIGTOP-3704: Update Yarn service configuration in Mpack (#917)

* BIGTOP-3704: Update Yarn service configuration in Mpack

This patch is to modfify Yarn configuration directory and replace deprecated Macro
when deploying Yarn+mapReduce service by Ambari Mpack.

Change-Id: I75ebf89d376aea0aadd036e903ff275afd4cfa4f
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
* update deprecated Macro of Yarn

Change-Id: If43eaa361c41cbb5ba56cd8a7812573e566ed944
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2 months agoBIGTOP-3699. Bump Spark to 3.2.1. (#919)
Leona Yoda [Mon, 27 Jun 2022 08:06:06 +0000 (17:06 +0900)] 
BIGTOP-3699. Bump Spark to 3.2.1. (#919)

3 months agoBIGTOP-3707: Update latest repo url for Provisioner docker yamls (#918)
Yuqi Gu [Mon, 27 Jun 2022 02:27:02 +0000 (10:27 +0800)] 
BIGTOP-3707: Update latest repo url for Provisioner docker yamls (#918)

Update repo url of 3.1.0 release for provisioner docker Distros yamls

Change-Id: Iac4125c555e83037604aee49e25f14a83c504964
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
3 months agoBIGTOP-3694. Fix Oozie build errors. (#911)
Leona Yoda [Thu, 23 Jun 2022 08:38:27 +0000 (17:38 +0900)] 
BIGTOP-3694. Fix Oozie build errors. (#911)

3 months agoBIGTOP-3706. Bump Hadoop to 3.3.3. (#916)
Masatake Iwasaki [Wed, 22 Jun 2022 12:12:14 +0000 (21:12 +0900)] 
BIGTOP-3706. Bump Hadoop to 3.3.3. (#916)

* BIGTOP-3706. Bump Hadoop to 3.3.3.

* using CapacityScheduler to make smoke-tests work run by root user with the default configuration.

3 months agoBIGTOP-3669. Use maven instead of ant to build ZooKeeper. (#890)
Masatake Iwasaki [Tue, 21 Jun 2022 09:41:33 +0000 (18:41 +0900)] 
BIGTOP-3669. Use maven instead of ant to build ZooKeeper. (#890)

3 months agoBIGTOP-3705: Refine the Ambari RPM spec to fix warning after rpm uninstallation ...
Yuqi Gu [Tue, 21 Jun 2022 04:48:27 +0000 (12:48 +0800)] 
BIGTOP-3705: Refine the Ambari RPM spec to fix warning after rpm uninstallation (#914)

Duplicated deleting operation ('%files' and '%preun%') in Ambari RPM spec
makes some warning issues when uninstalling Ambari rpm.
And '%config' in spec would create 'xxx.rpmsave' which was no longer needed.

Change-Id: Ic0f8a469ce69641e0482a3c7fcc259bd56f33c40
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
3 months agoBIGTOP-3438. Upgrade protobuf-2.5.0 to 3.7.1 for Hadoop-3.3.0. (#915)
Masatake Iwasaki [Fri, 17 Jun 2022 08:41:27 +0000 (17:41 +0900)] 
BIGTOP-3438. Upgrade protobuf-2.5.0 to 3.7.1 for Hadoop-3.3.0. (#915)

3 months agoBIGTOP-3698. Bump Hive to 3.1.3. (#912)
Leona Yoda [Fri, 10 Jun 2022 02:16:12 +0000 (11:16 +0900)] 
BIGTOP-3698. Bump Hive to 3.1.3. (#912)

3 months agoBIGTOP-3692: Upgrade flink version to 1.15.0 (#909)
Leona Yoda [Mon, 6 Jun 2022 23:00:55 +0000 (08:00 +0900)] 
BIGTOP-3692: Upgrade flink version to 1.15.0 (#909)

* upgrade flink version to 1.15.0 and add patches for husky

* use python3 in rpmbuild to avoid bytecompilation errors

3 months agoBIGTOP-3687. Add kubernetes client jar to Spark package. (#907)
Leona Yoda [Sun, 5 Jun 2022 17:21:54 +0000 (02:21 +0900)] 
BIGTOP-3687. Add kubernetes client jar to Spark package. (#907)

4 months agoBigtop 3.1.0 website update
Kengo Seki [Sat, 21 May 2022 12:33:02 +0000 (21:33 +0900)] 
Bigtop 3.1.0 website update

4 months agoBIGTOP-3685: make docker-provisioner to be used both in docker-compose v1 and v2...
Leona Yoda [Mon, 16 May 2022 14:10:46 +0000 (23:10 +0900)] 
BIGTOP-3685: make docker-provisioner to be used both in docker-compose v1 and v2 (#903)

4 months agoBIGTOP-3684: Fix RAT issues in Bigtop source tree (#902) 904/head
Yuqi Gu [Fri, 13 May 2022 05:35:14 +0000 (13:35 +0800)] 
BIGTOP-3684: Fix RAT issues in Bigtop source tree (#902)

Change-Id: Id9f057daa77af1150bd51af319a4519bc390532f
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
4 months agoBIGTOP-3680: Remove jruby version replacement to be updated. (#900)
Leona Yoda [Wed, 11 May 2022 09:59:39 +0000 (18:59 +0900)] 
BIGTOP-3680: Remove jruby version replacement to be updated. (#900)