hbase-filesystem.git
3 weeks agoHBASE-26483. [HBOSS] add support for createFile() and openFile(path) (#35) master
Steve Loughran [Sat, 10 Sep 2022 00:49:33 +0000 (01:49 +0100)] 
HBASE-26483. [HBOSS] add support for createFile() and openFile(path)  (#35)

* HBASE-26483 [HBOSS] add lock around openFile operation

* Rebase onto 3.3 only code
* add cleverness about whether to pass filestatus down to s3
  (that is safety checks except if something other than hive is
   actually wrapping s3a file status entries)
* lots of improvement on passthrough, including of getters, so that tests
  can see what happened.
* test explicitly makes sure that on s3a, non s3a status entries don't get
  passed down; hadoop 3.3.x releases may fail there, whereas on later
  builds they just ignore it (as does ABFS and its status type).
* TestZKLockManagerConfig hardened to work in IDE test runs with
   FS instance caching
* IOStatistics Awareness and passthrough; used in tests for logging.
* Adding the check that the FileStatus path must equal
   that of status.getPath before passdown, again, something
   the early openFile impls checked for (now they only validate
   the final filename element for equality).

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2 months agoHBASE-27196: Enable code coverage reporting to SonarQube in hbase-filesystem (#36)
Horváth Dóra [Fri, 15 Jul 2022 09:18:22 +0000 (11:18 +0200)] 
HBASE-27196: Enable code coverage reporting to SonarQube in hbase-filesystem (#36)

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Reviewed-by: Andor Molnar <andor@apache.org>
3 months agoHBASE-27076. [HBOSS] compile against hadoop 3.3.2+ only. (#34)
Steve Loughran [Wed, 15 Jun 2022 20:55:13 +0000 (21:55 +0100)] 
HBASE-27076. [HBOSS] compile against hadoop 3.3.2+ only. (#34)

Cut out all hadoop 3.2 support from files/build and move the
s3 client binding code back into hadoop-testutils.

The other hadoop profiles and the java test code to query/act
on different versions are retained, so that if a future version
ever needs to be dynamic again, the code is ready.

* removes the hadoop 3.2 profile; retains the 3.3 and default ones
* hadoop.version is set to 3.3.2.
* version checking only handles HadoopVersion = 3.3
* Hadoop33EmbeddedS3ClientFactory is in hadoop-testutils source tree
* the hadoop3-3-testutils and hadoop3-2-testutils modules are gone
* TestUtils.renameToExistingDestinationSupported() is always true; uses
  in assumeTrue() calls cut
* cut hbase-oss/src/test/resources/contract/hadoop-3.2/s3a.xml;
  choice of contract profile is still dynamic.
* travis to only check 3.3 profile
* readme updated

Change-Id: Idc9181537443904004f2e79baefe7a2c0fe67902

Signed-off-by: Andrew Purtell <apurtell@apache.org>
4 months agoHBASE-27042. Remove S3Guard awareness from HBoss
Steve Loughran [Mon, 16 May 2022 13:22:29 +0000 (14:22 +0100)] 
HBASE-27042. Remove S3Guard awareness from HBoss

* Cut HBoss test code which would explicitly disable S3Guard
  -obsolete and breaks the build on forthcoming hadoop releases.
* Also cut all references in docs and XML.
* EmbeddedS3 to implement deleteObject() and so support
  AWS SDKs 1.11.1026+, which expects this (hadoop 3.3.2+).
* Maven test runs to provide full stack traces on failures,
  so runtime failures can be debugged and fixed.

Change-Id: Iab10cfc6780cea064ea116efc1c177332d780514
Signed-off-by: Josh Elser <elserj@apache.org>
7 months agoHBASE-26786 [hboss] Limit synchronization from hot path of HBoss APIs (#32)
Ankit Singhal [Wed, 2 Mar 2022 23:44:08 +0000 (15:44 -0800)] 
HBASE-26786 [hboss] Limit synchronization from hot path of HBoss APIs (#32)

8 months agoHBASE-26236 Simple travis build for hbase-filesystem (#28)
Peter Somogyi [Tue, 25 Jan 2022 01:40:28 +0000 (02:40 +0100)] 
HBASE-26236 Simple travis build for hbase-filesystem (#28)

Co-authored-by: Josh Elser <elserj@apache.org>
9 months agoHBASE-26513 [HBOSS] Update readme to reflect dependencies change introduced by HBASE...
Wellington Ramos Chevreuil [Fri, 10 Dec 2021 00:04:55 +0000 (00:04 +0000)] 
HBASE-26513 [HBOSS] Update readme to reflect dependencies change introduced by HBASE-25900 (#31)

Signed-off-by: Josh Elser <elserj@apache.org>
10 months agoHBASE-26437 Clean up the znodes for the src after a rename.
Josh Elser [Wed, 10 Nov 2021 18:13:32 +0000 (13:13 -0500)] 
HBASE-26437 Clean up the znodes for the src after a rename.

HBOSS was orphaning znodes from the src of a path which is renamed. Over
time, this will result in a very large usage of ZK due to HBOSS.

Add some logging to dump the contents of ZK

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Closes #29

10 months agoHBASE-26453 Correct the behavior of isBeneath to not consider paths which share a...
Josh Elser [Fri, 12 Nov 2021 17:57:07 +0000 (12:57 -0500)] 
HBASE-26453 Correct the behavior of isBeneath to not consider paths which share a name prefix as beneath one another.

The current implementation of isBeneath fails when given paths of the
form '/foo' and '/foobar' (returning that '/foobar' is beneath '/foo').

Because this method returns incorrect values, it causes Curator mutexes
to be removed and znodes to be removed while they were potentially held.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
11 months agoHBASE-26315 Upgrade commons-io to 2.11.0 in hbase-filesystem (#27)
Peter Somogyi [Wed, 6 Oct 2021 09:09:08 +0000 (11:09 +0200)] 
HBASE-26315 Upgrade commons-io to 2.11.0 in hbase-filesystem (#27)

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
11 months agoHBASE-25900 Hadoop 3.2 and 3.3 support (#25)
Josh Elser [Tue, 5 Oct 2021 16:51:58 +0000 (12:51 -0400)] 
HBASE-25900 Hadoop 3.2 and 3.3 support (#25)

Build on top of SteveL's original changes. Added indirection so
that the correct S3ClientFactory can be included at test-time. Provides
s3a.xml contracts for each version to reflect what actually works. Skips
the new tests which we know would fail on the older <Hadoop3.3 releases.

Co-authored-by: Steve Loughran <stevel@apache.org>
Co-authored-by: Peter Somogyi <psomogyi@cloudera.com>
Signed-off-by: Peter Somogyi <psomogyi@cloudera.com>
13 months agoHBASE-26164 Update dependencies and remove duplicative aws-java-sdk definition (#24)
Josh Elser [Tue, 3 Aug 2021 10:40:10 +0000 (06:40 -0400)] 
HBASE-26164 Update dependencies and remove duplicative aws-java-sdk definition (#24)

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
16 months ago HBASE-25964 Introducing hbase metrics to Hboss
BukrosSzabolcs [Wed, 2 Jun 2021 15:36:09 +0000 (17:36 +0200)] 
 HBASE-25964 Introducing hbase metrics to Hboss

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
19 months agoHBASE-25572 add a 'release' profile to do common project release tasks.
Sean Busbey [Sat, 13 Feb 2021 03:26:07 +0000 (21:26 -0600)] 
HBASE-25572 add a 'release' profile to do common project release tasks.

Closes #20

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
19 months agoHBASE-25578 HBase filesystem pom should include nexus staging repo management (#21)
Sean Busbey [Mon, 15 Feb 2021 12:39:11 +0000 (06:39 -0600)] 
HBASE-25578 HBase filesystem pom should include nexus staging repo management (#21)

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
20 months agoHBASE-25477 Purge use of VisibleForTesting (#19)
Peter Somogyi [Thu, 7 Jan 2021 17:25:18 +0000 (18:25 +0100)] 
HBASE-25477 Purge use of VisibleForTesting (#19)

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
22 months agoHBASE-25331 [HBOSS] TreeLockManager should log warnings about lock contention periodi...
Wellington Ramos Chevreuil [Mon, 30 Nov 2020 15:09:17 +0000 (15:09 +0000)] 
HBASE-25331 [HBOSS] TreeLockManager should log warnings about lock contention periodically (#18)

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2 years agoHBASE-25060 [HBOSS] TreeLockManager should not define itself as the default implement...
Wellington Ramos Chevreuil [Fri, 18 Sep 2020 08:31:52 +0000 (09:31 +0100)] 
HBASE-25060 [HBOSS] TreeLockManager should not define itself as the default implementation for fs.hboss.sync.impl (#17)

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2 years agoHBASE-24989 [HBOSS] Some code cleanup (#16)
Wellington Ramos Chevreuil [Thu, 10 Sep 2020 09:12:23 +0000 (10:12 +0100)] 
HBASE-24989 [HBOSS] Some code cleanup (#16)

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2 years agoHBASE-24963 [HBOSS] UTs failing when running against hadoop 3.3. (#15)
Wellington Ramos Chevreuil [Wed, 9 Sep 2020 16:38:27 +0000 (17:38 +0100)] 
HBASE-24963 [HBOSS] UTs failing when running against hadoop 3.3. (#15)

Signed-off-by: Josh Elser <elserj@apache.org>
Singed-off-by: Andrew Purtell <apurtell@apache.org>
2 years agoHBASE-24961 [HBOSS] HBaseObjectStoreSemantics.close should call super.close to make...
Wellington Ramos Chevreuil [Tue, 1 Sep 2020 08:32:51 +0000 (09:32 +0100)] 
HBASE-24961 [HBOSS] HBaseObjectStoreSemantics.close should call super.close to make sure its own instance always get removed from FileSystem.CACHE (#14)

Signed-off-by: Josh Elser <elserj@apache.org>
2 years agoHBASE-24261: Initial version of ASF infra integration configuration
Bharath Vissapragada [Wed, 6 May 2020 00:28:07 +0000 (17:28 -0700)] 
HBASE-24261: Initial version of ASF infra integration configuration

This is an initial version of the yaml config for ASF infra
integration. We might have some hiccups in the beginning but we
can iteratively improve until the old (desired) setup is back
in place.

2 years agoHBASE-24124 hbase-filesystem to use guava from hbase-thirdparty
Wei-Chiu Chuang [Mon, 6 Apr 2020 19:01:13 +0000 (12:01 -0700)] 
HBASE-24124 hbase-filesystem to use guava from hbase-thirdparty

closes #12

Signed-off-by: Tamas Penzes <tamaas@cloudera.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2 years agoHBASE-23314 Make HBaseObjectStoreSemantics FilterFileSystem (#11)
Mingliang Liu [Tue, 19 Nov 2019 09:07:45 +0000 (01:07 -0800)] 
HBASE-23314 Make HBaseObjectStoreSemantics FilterFileSystem (#11)

Signed-off-by: Andrew Putell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2 years agoHBASE-23142 Include zookeeper-jute into hboss-thirdparty deps
Josh Elser [Wed, 9 Oct 2019 16:21:13 +0000 (12:21 -0400)] 
HBASE-23142 Include zookeeper-jute into hboss-thirdparty deps

ZooKeeper 3.5 has a transitive dependency on zookeeper-jute.
This is a no-op for ZK 3.4.

Closes #10

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
3 years agoHBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1 (#9)
Wellington Ramos Chevreuil [Wed, 2 Oct 2019 17:06:08 +0000 (18:06 +0100)] 
HBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1 (#9)

HBASE-23099 - [HBOSS] Raise hadoop dependency version to 3.2.1

There's been some changes on S3Guard that comes with hadoop 3.2.1 and upcoming versions, such as HADOOP-16085, HADOOP-16233 and HADOOP-16380, among other recent commits. Some of these required additional properties or modified S3A expected behaviour for a few FS contract tests.

Signed-off-by: Josh Elser <elserj@apache.org>
3 years agoHBASE-23076 [HBOSS] ZKTreeLockManager shouldn't try to acquire a lock from the InterP...
Wellington Ramos Chevreuil [Thu, 26 Sep 2019 08:13:30 +0000 (09:13 +0100)] 
HBASE-23076 [HBOSS] ZKTreeLockManager shouldn't try to acquire a lock from the InterProcessMutex instance when checking if other processes hold it. #8

Signed-off-by: Josh Elser <elserj@apache.org>
3 years agoHBASE-23032 Upgrade to Curator 4.2.0
Balazs Meszaros [Mon, 23 Sep 2019 10:52:14 +0000 (12:52 +0200)] 
HBASE-23032 Upgrade to Curator 4.2.0

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
3 years agoHBASE-22694 Fallback to hbase.zookeeper.quorum if fs.hboss.sync.zk.connectionString...
Josh Elser [Mon, 15 Jul 2019 19:43:42 +0000 (15:43 -0400)] 
HBASE-22694 Fallback to hbase.zookeeper.quorum if fs.hboss.sync.zk.connectionString is undefined

Simple code change, but some refactoring to add testing for the change.

EmbeddedZK was made to be an object which we instantiate, rather than
static state and static methods. This lets us re-use the same code for
the contract tests without stomping on one-another.

Also adds in .gitignore entries for Eclipse metadata.

Closes #6

Signed-off-by: Sean Busbey <busbey@apache.org>
3 years agoHBASE-22583 [hbase-filesystem] Update readme to describe compatible hbase versions
Wellington Chevreuil [Fri, 21 Jun 2019 10:49:37 +0000 (11:49 +0100)] 
HBASE-22583 [hbase-filesystem] Update readme to describe compatible hbase versions

Signed-off-by: Andrew Putell <apurtell@apache.org>
3 years agoHBASE-22574 hbase-filesystem does not build against HBase 1 (#5)
Andrew Purtell [Mon, 17 Jun 2019 19:06:59 +0000 (12:06 -0700)] 
HBASE-22574 hbase-filesystem does not build against HBase 1 (#5)

Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
3 years agoHBASE-22516 Update version for next dev cycle (#4)
Wellington Ramos Chevreuil [Tue, 11 Jun 2019 15:09:20 +0000 (16:09 +0100)] 
HBASE-22516 Update version for next dev cycle (#4)

3 years agoHBASE-22516 - prepare for 1.0.0-alpha1 RCs (#3) 1.0.0-alpha1-RC0 rel/1.0.0-alpha1
Wellington Ramos Chevreuil [Tue, 4 Jun 2019 20:18:27 +0000 (21:18 +0100)] 
HBASE-22516 - prepare for 1.0.0-alpha1 RCs (#3)

* changing versions values on related pom files
* yetus generated CHANGELOG.md and RELEASENOTES.md

3 years agoHBASE-22515 HBOSS: Document test cases known to fail under Null lock implementation
Wellington Chevreuil [Mon, 3 Jun 2019 15:23:35 +0000 (16:23 +0100)] 
HBASE-22515 HBOSS: Document test cases known to fail under Null lock implementation

Signed-off-by: Sean Busbey <busbey@apache.org>
3 years agoRevert "Document HBOSS test cases known to fail under Null lock implementation"
Sean Busbey [Mon, 3 Jun 2019 16:02:22 +0000 (11:02 -0500)] 
Revert "Document HBOSS test cases known to fail under Null lock implementation"

This reverts commit 522ddafa13ec8936b72a45f102ca4a21e1a6b9a2.

3 years agoDocument HBOSS test cases known to fail under Null lock implementation
Wellington Chevreuil [Mon, 3 Jun 2019 15:23:35 +0000 (16:23 +0100)] 
Document HBOSS test cases known to fail under Null lock implementation

Signed-off-by: Sean Busbey <busbey@apache.org>
3 years agoHBASE-22415 HBOSS: changed log verbosity to debug on some lock colision cases
Wellington Chevreuil [Fri, 31 May 2019 11:37:24 +0000 (12:37 +0100)] 
HBASE-22415 HBOSS: changed log verbosity to debug on some lock colision cases

Signed-off-by: Biju Nair <gs.biju@gmail.com>
Signed-off-by: Sean Mackrory <mackrorysd@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
3 years agoHBASE-22427 HBOSS: TestTreeLockManager fails on non-ZK implementations.
Sean Mackrory [Wed, 15 May 2019 17:47:52 +0000 (11:47 -0600)] 
HBASE-22427 HBOSS: TestTreeLockManager fails on non-ZK implementations.

Signed-off-by: Sean Busbey <busbey@apache.org>
3 years agoHBASE-22393 HBOSS: Shaded external dependencies to avoid conflicts with Hadoop and...
Sean Mackrory [Thu, 16 May 2019 18:58:42 +0000 (19:58 +0100)] 
HBASE-22393 HBOSS: Shaded external dependencies to avoid conflicts with Hadoop and HBase.

Closes #2

Signed-off-by: Josh Elser <elserj@apache.org>
Co-authored-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Co-authored-by: Sean Busbey <busbey@apache.org>
3 years agoHBASE-22493 Documenting supported hadoop versions.
Wellington Chevreuil [Wed, 29 May 2019 10:01:59 +0000 (11:01 +0100)] 
HBASE-22493 Documenting supported hadoop versions.

Signed-off-by: Sean Busbey <busbey@apache.org>
3 years agoHBASE-22437 HBOSS: Add Hadoop 2 / 3 profiles.
Sean Mackrory [Wed, 22 May 2019 16:30:28 +0000 (12:30 -0400)] 
HBASE-22437 HBOSS: Add Hadoop 2 / 3 profiles.

Signed-off-by: wellington <wchevreuil@cloudera.com>
Amending-Author: Josh Elser <elserj@apache.org>

3 years agoHBASE-22416 HBOSS: Wait for curator to connect to ZK before returning.
Josh Elser [Tue, 14 May 2019 15:54:49 +0000 (11:54 -0400)] 
HBASE-22416 HBOSS: Wait for curator to connect to ZK before returning.

We'll wait up to 30s. If we're still not connected, throw an exception.

Signed-off-by: Sean Mackrory <mackrorysd@gmail.com>
3 years agoHBASE-22386 HBOSS: Limit depth that listing locks check for other locks
Sean Mackrory [Tue, 14 May 2019 15:36:18 +0000 (11:36 -0400)] 
HBASE-22386 HBOSS: Limit depth that listing locks check for other locks

Signed-off-by: Josh Elser <elserj@apache.org>
3 years agoHBASE-22149 HBOSS: A FileSystem implementation to provide HBase's required semantics...
Sean Mackrory [Thu, 9 May 2019 14:03:16 +0000 (08:03 -0600)] 
HBASE-22149 HBOSS: A FileSystem implementation to provide HBase's required semantics on object stores.

* Adds top level project structure
* Adds module for hbase-oss with initial implementation of wrapper FileSystem to enforce semantics needed by hbase

Closes #1

3 years agoInitial empty commit.
Sean Busbey [Tue, 30 Apr 2019 12:37:54 +0000 (07:37 -0500)] 
Initial empty commit.