giraph.git
2 years agoGIRAPH-1055: Javadoc fails build with Java 8
Avery Ching [Fri, 22 Apr 2016 23:07:22 +0000 (16:07 -0700)] 
GIRAPH-1055: Javadoc fails build with Java 8

Summary:
Java 8 javadoc has stricter checking, which results in mvn javadoc:javadoc failing:
Example:
100 errors
200 warnings
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent ............................... SUCCESS [ 1.196 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 9.583 s]

Test Plan:
[INFO] --- maven-javadoc-plugin:2.9:javadoc (default-cli) @ giraph-dist ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent ............................... SUCCESS [  1.093 s]
[INFO] Apache Giraph Core ................................. SUCCESS [ 10.934 s]
[INFO] Apache Giraph Blocks Framework ..................... SUCCESS [  3.245 s]
[INFO] Apache Giraph Examples ............................. SUCCESS [  3.841 s]
[INFO] Apache Giraph Accumulo I/O ......................... SUCCESS [  2.048 s]
[INFO] Apache Giraph HBase I/O ............................ SUCCESS [  1.132 s]
[INFO] Apache Giraph HCatalog I/O ......................... SUCCESS [  3.053 s]
[INFO] Apache Giraph Gora I/O ............................. SUCCESS [  3.500 s]
[INFO] Apache Giraph Rexster I/O .......................... SUCCESS [  0.091 s]
[INFO] Apache Giraph Rexster Kibble ....................... SUCCESS [  1.276 s]
[INFO] Apache Giraph Rexster I/O Formats .................. SUCCESS [  3.193 s]
[INFO] Apache Giraph Distribution ......................... SUCCESS [  2.074 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.880 s
[INFO] Finished at: 2016-04-22T16:09:04-07:00
[INFO] Final Memory: 62M/1131M
[INFO] ------------------------------------------------------------------------

Reviewers: maja.kabiljo, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D57105

2 years agoGIRAPH-1054: Separate ThriftService from JobProgressTrackerService on the client
Avery Ching [Wed, 13 Apr 2016 23:03:40 +0000 (16:03 -0700)] 
GIRAPH-1054: Separate ThriftService from JobProgressTrackerService on the client

Summary:
* Moves the job tracker conf options into the GiraphConstants
* Factors out the static GiraphJob#startThriftServer and GiraphJob#stopThriftServer methods from createJobProgressServer
* Allows adding other Thrift services to the ThriftServer

Test Plan: Tried on a cluster

Reviewers: maja.kabiljo, sergey.edunov

Reviewed By: sergey.edunov

Subscribers: sergey.edunov

Differential Revision: https://reviews.facebook.net/D57087

2 years ago[GIRAPH-1053] Log exceptions to command line
Maja Kabiljo [Tue, 19 Apr 2016 00:34:36 +0000 (17:34 -0700)] 
[GIRAPH-1053] Log exceptions to command line

Summary: When we know an exception occurred, log it to command line to make it easier for people running jobs to see what the issue was.

Test Plan: Ran two jobs, one with error in input one with error in compute, verified exception is printed to command line. Also ran a normal job and verified it didn't print anything new to command line

Differential Revision: https://reviews.facebook.net/D56931

2 years ago[GIRAPH-1041] Generate primitive type specific code
Igor Kabiljo [Tue, 15 Mar 2016 21:41:35 +0000 (14:41 -0700)] 
[GIRAPH-1041] Generate primitive type specific code

Summary:
- Use FreeMarker library to generate primitive type specific code.
Initially generating three sets of files:
{TYPE}Consumer, {TYPE}TypeOps and W{TYPE}ArrayList

Right now generation happens manually, and generated files are being committed.
In the future we can move those to a separate project, and have them generated
when maven is compiling and deploying.

Additionally to generation change, BasicArrayList is renamed to WArrayList and
directly extends fastutil implementation, to now serves two purposes:
- generic handling of efficient arrays through TypeOps
- extended fastutil class - to make it writtable, to add useful Java8 methods,
  or anything else we can think of. Since we are just extending it, and there is
  no efficiency penalty, we can always use WLongArrayList instead of LongArrayList.

There is additional WReusableLongArrayList, which when readFields is called,
doesn't size it to exact size, but reuses the old length.

Test Plan:
mvn clean install

There are no changes in logic in this diff. Will send a small separate diff
with some examples of what is now simpler.

Reviewers: sergey.edunov, dionysis.logothetis, spupyrev, maja.kabiljo

Differential Revision: https://reviews.facebook.net/D52515

2 years agoGIRAPH-1052: Fix makeSymmetricUnweighted
Maja Kabiljo [Fri, 8 Apr 2016 21:33:50 +0000 (14:33 -0700)] 
GIRAPH-1052: Fix makeSymmetricUnweighted

Summary: PrepareGraphPieces.makeSymmetricUnweighted is currently very inefficient for skewed degree graphs, because it reuses set objects based on the number of in edges, but also adds all out edges to the set, so sets which should be small can become huge. Since incoming ids are unique anyways, we don't need to add them to the set.

Test Plan: Ran a job without and with the change, verified result is the same but it's much faster now

Reviewers: ikabiljo

Differential Revision: https://reviews.facebook.net/D56481

2 years agoGIRAPH-1050: Add MapperObserver
Maja Kabiljo [Thu, 7 Apr 2016 16:40:27 +0000 (09:40 -0700)] 
GIRAPH-1050: Add MapperObserver

Summary: Add MapperObserver which will be called once per mapper before anything else happens.

Test Plan: Ran a job with MapperObserver set, verified it's called at the right time

Differential Revision: https://reviews.facebook.net/D56373

2 years agoGIRAPH-1046: Add a way to synchronize full GC calls across workers
Maja Kabiljo [Thu, 10 Mar 2016 22:30:11 +0000 (14:30 -0800)] 
GIRAPH-1046: Add a way to synchronize full GC calls across workers

Summary: In applications which use memory more heavily, we can see full GC pauses happening on different workers at different times, and each of these is causing some delay because other workers are often waiting on something from the worker in GC (closing open requests, finishing superstep, etc). Having a way to coordinate when full GCs are called could help them have less effect on job performance.

Test Plan: Ran some memory heavy jobs where I observed overall better performance from using this feature.

Differential Revision: https://reviews.facebook.net/D55347

2 years agoImprove flow control on sender side (pre-requisite for credit-based flow control)
Hassan Eslami [Mon, 4 Apr 2016 23:29:36 +0000 (16:29 -0700)] 
Improve flow control on sender side (pre-requisite for credit-based flow control)

Summary: Currently, a sender worker will keep all open requests (and optionally up to a certain number of total open requests) in its own memory. This behavior may cause high memory usage in sender side. Also, since messages can arrive to a worker at an arbitrary rate, receiver may not have the ability to handle all incoming messages, hence we may see a large memory footprint in receiver as well. This diff addresses the problem by limiting the number of open requests per worker in sender side. Also, it provides a cache of unsent requests on sender in case the sender already sent enough messages to another worker but has not received any response back.

Test Plan: mvn clean verify

Reviewers: avery.ching, sergey.edunov, maja.kabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Subscribers: Alessio

Differential Revision: https://reviews.facebook.net/D43797

2 years agounsafe readers for varints
spupyrev [Fri, 1 Apr 2016 18:06:38 +0000 (11:06 -0700)] 
unsafe readers for varints

Summary:
Varint encdoing (and hence, LongDiffNullArrayEdges) can be much faster if using UnsafeByteInput/Output. In fact, the speed of iterating over LongDiffNullArrayEdges is almost as fast as iterating over LongNullArrayEdges after the change. This difference is less than a few percent for jobs that require a lot of edge iterators, while it is significant (over 20%) without the change.

JIRA: https://issues.apache.org/jira/browse/GIRAPH-1049

Test Plan: mvn clean install

Reviewers: sergey.edunov, maja.kabiljo, dionysis.logothetis, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D56169

2 years agofaster varint
spupyrev [Wed, 23 Mar 2016 17:34:05 +0000 (10:34 -0700)] 
faster varint

Summary:
Varint is improved in two ways:
- faster readLong and readInt
- making sure that negative numbers can be encoded

JIRA: https://issues.apache.org/jira/browse/GIRAPH-1049

Test Plan: TestVarint.java

Reviewers: dionysis.logothetis, maja.kabiljo, sergey.edunov, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D55755

2 years ago[GIRAPH-1041] Generate primitive type specific code for functions
Igor Kabiljo [Tue, 15 Mar 2016 21:40:52 +0000 (14:40 -0700)] 
[GIRAPH-1041] Generate primitive type specific code for functions

Summary:
- Use FreeMarker library to generate primitive type specific code.
Initially generating two sets of files:
{TYPE}Consumer, Obj2{TYPE}Function

Right now generation happens manually, and generated files are being committed.
In the future we can move those to a separate project, and have them generated
when maven is compiling and deploying.

Splitting of D52515 into reviewable pieces

Test Plan:
mvn clean install

There are no changes in logic in this diff.

Reviewers: spupyrev, sergey.edunov, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D55527

2 years agoIncrease info-logging while waiting for straggler workers
Tyler Serdar Bulut [Tue, 15 Mar 2016 19:10:36 +0000 (12:10 -0700)] 
Increase info-logging while waiting for straggler workers

Summary:
Keep logging info messages while waiting for task-time-out

Test Plan:
All unit tests are passing.
Manual tests to ensure desired functionality is observed.

Reviewers: maja.kabiljo

Subscribers: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D55467

2 years agoGIRAPH-1039: Fix stopping jmap histo thread
Maja Kabiljo [Tue, 3 Nov 2015 19:01:22 +0000 (11:01 -0800)] 
GIRAPH-1039: Fix stopping jmap histo thread

Summary: Currently if jmap histo frequency is set to long period we end up stuck in the end of the job for a long time waiting on jmap histo thread

Test Plan: Ran a job with long jmap frequency - verified it gets stuck without this change and finishes fine with it

Differential Revision: https://reviews.facebook.net/D50085

2 years agounsafe byte readers/writers
spupyrev [Tue, 15 Mar 2016 23:47:15 +0000 (16:47 -0700)] 
unsafe byte readers/writers

Summary: using unsafe readers/writers

Test Plan:
tested on PageRank app, and Fanout computation. In both cases, there is a ~20% speedup

JIRA: https://issues.apache.org/jira/browse/GIRAPH-1049

Reviewers: sergey.edunov, maja.kabiljo, dionysis.logothetis, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D55509

2 years agoNew out-of-core infrastructure (first patch including fixed out-of-core mechanism)
Sergey Edunov [Tue, 15 Mar 2016 17:40:20 +0000 (10:40 -0700)] 
New out-of-core infrastructure (first patch including fixed out-of-core mechanism)

Summary: This is a re-design of out-of-core mechanism. The new implementation allows for much more intelligent partition scheduling and IO.

Test Plan:
mvn clean verify

Reviewers: maja.kabiljo, sergey.edunov, avery.ching, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D54549

2 years ago[easy] Log aggregate times per piece at the end
Igor Kabiljo [Fri, 4 Mar 2016 20:33:35 +0000 (12:33 -0800)] 
[easy] Log aggregate times per piece at the end

Test Plan:
It prints:

  16/03/04 12:46:38 INFO internal.BlockMasterLogic: Time sums master:
     count    time %       time name
        34    37.50%   00:00:00 PageRankCheckConvergence
        34    62.50%   00:00:00 PageRankUpdate
        68             00:00:00 total

  16/03/04 12:46:38 INFO internal.BlockMasterLogic: Time sums worker:
     count    time %       time name
        33    50.00%   00:00:00 [receiver=PageRankCheckConvergence,sender=PageRankUpdate]
         1     0.82%   00:00:00 [receiver=PageRankCheckConvergence,sender=null]
        34    47.54%   00:00:00 [receiver=PageRankUpdate,sender=PageRankCheckConvergence]
         1     1.64%   00:00:00 [receiver=null,sender=PageRankUpdate]
        69             00:00:00 total

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo, spupyrev

Reviewed By: spupyrev

Differential Revision: https://reviews.facebook.net/D55113

2 years agoGIRAPH-1044. Update book info in the User Docs / Related Literature page of the site
Roman Shaposhnik [Mon, 7 Mar 2016 01:41:28 +0000 (17:41 -0800)] 
GIRAPH-1044. Update book info in the User Docs / Related Literature page of the site

2 years agoMaking threads in JobProgressService daemons
Sergey Edunov [Fri, 26 Feb 2016 19:48:08 +0000 (11:48 -0800)] 
Making threads in JobProgressService daemons

Summary: We noticed that sometimes job client doesn't finish because threads in JobProgressService are still running. Here we're making them daemons, so that if everything else is done, we will be able to finish application.

Test Plan: run test job

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D53829

2 years agoAdd PrepareGraphPieces.isSymmetricBlock to check for a symmetric graph
Yuri Schimke [Thu, 25 Feb 2016 18:42:21 +0000 (10:42 -0800)] 
Add PrepareGraphPieces.isSymmetricBlock to check for a symmetric graph

Summary:
PrepareGraphPieces.isSymmetricBlock is a reusable factory function
for creating blocks that check if a graph is symmetric by
XOR reducing a preditable hash of the edges pairs (V1, V2)

Test Plan:
Unit Tests for all changed files, testing on demo graphs.
Will run a full test job.

Reviewers: spupyrev, dionysis.logothetis, ikabiljo, maja.kabiljo

Reviewed By: maja.kabiljo

Subscribers: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D54411

2 years agoUse Partitions in LocalBlockRunner
Igor Kabiljo [Tue, 29 Dec 2015 23:14:32 +0000 (15:14 -0800)] 
Use Partitions in LocalBlockRunner

Summary:
Speed up LocalBlockRunner, by not operating on a TestGraph, but on vertices stored in partitions.
With it - deprecate old non-SimplePartitionerFactory way of specifying partitioning.
(and with it renamed SimplePartitionerFactory to old name GraphPartitionerFactory, and changing it to
 GraphPartitionerFactoryInterface)

Test Plan:
Run unit-test for speed:

  testEmptyIterationsSmallGraph
    6.5 -> 6.3
  testEmptyIterationsSyntheticGraphLowDegree()
    42.0 -> 13.8
  testEmptyIterationsSyntheticGraphHighDegree()
    3.6 -> 2.0
  testPageRankSyntheticGraphLowDegree()
    51.0 -> 47.2
  testPageRankSyntheticGraphHighDegree()
    20.3 -> 17.4

Reviewers: maja.kabiljo, sergey.edunov, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D52425

2 years agoImplementation of DirectWritableSerializerCopyTest.copy()
Sergey Edunov [Fri, 15 Jan 2016 17:42:50 +0000 (09:42 -0800)] 
Implementation of DirectWritableSerializerCopyTest.copy()

Summary: Needed in certain application.

Test Plan:
  mvn clean install
  Also run an actual application.

Reviewers: maja.kabiljo, sergey.edunov, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D52149

3 years agoCorrection in interface documentation
Sergey Edunov [Tue, 1 Dec 2015 19:52:29 +0000 (11:52 -0800)] 
Correction in interface documentation

Summary: The description of the arguments in TypoOps.set() is reversed.

Test Plan: n/a

Reviewers: majakabiljo, sergey.edunov, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D43425

3 years agoAdded vldb publication
Sergey Edunov [Tue, 1 Dec 2015 19:50:19 +0000 (11:50 -0800)] 
Added vldb publication

Summary: Added new vldb paper in literature.xml.

Test Plan: n/a

Reviewers: avery.ching, sergey.edunov, maja.kabiljo, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D45399

3 years agoMake IntSupplier extend Serializable
Sergey Edunov [Wed, 18 Nov 2015 19:58:11 +0000 (11:58 -0800)] 
Make IntSupplier extend Serializable

Summary: Lambdas with IntSuppliers don't get serialized.

Test Plan: n/a

Reviewers: ikabiljo, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D50973

3 years ago[GIRAPH-1037] Surface worker index information to computations
Igor Kabiljo [Sat, 24 Oct 2015 00:34:07 +0000 (17:34 -0700)] 
[GIRAPH-1037] Surface worker index information to computations

Summary:
It can be useful for applications to surface:
number of workers
index of a worker particular VertexId is assigned to
index of a current worker.

Test Plan: mvn clean install

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D49401

3 years agoGIRAPH-1036: Allow mappers to fail early on exceptions
Maja Kabiljo [Wed, 21 Oct 2015 01:19:36 +0000 (18:19 -0700)] 
GIRAPH-1036: Allow mappers to fail early on exceptions

Summary:
Often when something fails in a mapper we see it stuck until its timeout passes. Digging through this issue I found two root causes:
- Many threads we are creating were not daemon, preventing process to exit, only main thread should be daemon
- When calling submit on ExecutorService, exceptions are not propagated back to the caller, unless get is called on the future. In ProgressableUtils.getResultsWithNCallables we were calling get on one by one future, causing us to have to wait for previous futures to finish before getting exception which happened in later one.

Test Plan: Run jobs in which I simulated exceptions on some partitions in loading, compute and storing phases, for each verified we exit quickly with exception clearly shown, and without this change we'd wait for timeout and other threads from same ProgressableUtils.getResultsWithNCallables to finish. Run a normal job successfully. mvn clean verify

Differential Revision: https://reviews.facebook.net/D49143

3 years agoGIRAPH-1034 Allow IPs for Worker2Worker communication
Sergey Edunov [Tue, 20 Oct 2015 00:35:58 +0000 (17:35 -0700)] 
GIRAPH-1034 Allow IPs for Worker2Worker communication

Test Plan:
Run several jobs in unreliable DNS environment.  With and without -Dgiraph.preferIP=true
Without this options job fail, but pass otherwise.

Reviewers: dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D48825

3 years agoGIRAPH-1035: Make sure we are able to use all compute threads
Maja Kabiljo [Mon, 19 Oct 2015 17:35:54 +0000 (10:35 -0700)] 
GIRAPH-1035: Make sure we are able to use all compute threads

Summary: The default logic of choosing the number of partitions when we use few workers and a lot of compute threads ends up choosing less partitions than there are threads. Add additional setting to prevent that.

Test Plan: Run a job with a few workers and lot of threads and verified number of partitions is set properly. mvn verify passed.

Differential Revision: https://reviews.facebook.net/D48993

3 years agoGIRAPH-1033: Remove zookeeper from input splits handling
Maja Kabiljo [Mon, 12 Oct 2015 17:56:39 +0000 (10:56 -0700)] 
GIRAPH-1033: Remove zookeeper from input splits handling

Summary: Currently we use zookeeper for handling input splits, by having each worker checking each split, and when a lot of splits are used this becomes very slow. We should have master coordinate input splits allocation instead, making the complexity proportional to #splits instead of #workers*#splits. Master holds all the splits and worker send requests to him asking for splits when they need them.

Test Plan: Run a job with 200 machines and 200k small splits - without this change input superstep takes 30 minutes, and with it less than 2 minutes. Also verified correctness on sample job. mvn clean verify passes.

Differential Revision: https://reviews.facebook.net/D48531

3 years agoMerge seeded and unseeded BFS into a single BFS implementation
Mayank Pundir [Fri, 9 Oct 2015 20:18:37 +0000 (13:18 -0700)] 
Merge seeded and unseeded BFS into a single BFS implementation

Summary: This change creates a general Breadth First Search version which supports the default BFS where distances to one or more seeds are computed. Additionally, this new version also supports assigning vertices to closest seeds for the purpose of clustering the vertices. This change provides a BlockFactory which highlights this functionality in addition to test cases.

Test Plan: Test cases for the new functionality added.

Reviewers: spupyrev, ikabiljo

Reviewed By: ikabiljo

Subscribers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Differential Revision: https://reviews.facebook.net/D47985

3 years ago[GIRAPH-1031] Adding onAllMappersStarted callback
Sergey Edunov [Sat, 19 Sep 2015 00:27:20 +0000 (17:27 -0700)] 
[GIRAPH-1031] Adding onAllMappersStarted callback

3 years agofixing large sets
spupyrev [Tue, 8 Sep 2015 23:05:29 +0000 (16:05 -0700)] 
fixing large sets

Summary:
Sets with a large number of elements (>800000000) are not supported by IntOpenHashSet.
Chaning it to IntOpenHashBigSet.

ArrayLists have the same problem, but we'll postpone the fix untill we have a use case

https://issues.apache.org/jira/browse/GIRAPH-1028

Test Plan:
mvn clean install
see also a new test TestCollections (that needs 32G to run)

Reviewers: sergey.edunov, maja.kabiljo, ikabiljo

Reviewed By: ikabiljo

Differential Revision: https://reviews.facebook.net/D44859

3 years agoAdding Blocks Framework documentation
Igor Kabiljo [Mon, 24 Aug 2015 21:58:09 +0000 (14:58 -0700)] 
Adding Blocks Framework documentation

Summary:
Adding basic documentation - full example walkthrough and migration library, and only minor info about the framework itself.
Will extend framework part more in the future.

Test Plan: Not sure how to test

Reviewers: avery.ching, sergey.edunov, dionysis.logothetis, maja.kabiljo

Reviewed By: dionysis.logothetis, maja.kabiljo

Differential Revision: https://reviews.facebook.net/D45411

3 years ago[GIRAPH-1023] Adding out-of-core messages to previously implemented adaptive out...
Hassan Eslami [Thu, 30 Jul 2015 21:03:12 +0000 (14:03 -0700)] 
[GIRAPH-1023] Adding out-of-core messages to previously implemented adaptive out-of-core mechanism

Summary:
This is the continuation of the previous diff on out-of-core mechanism. This diff completes the last diff by adding out-of-core messages, making the entire out-of-core mechanism a cohesive entity in Giraph.

This diff also improves the API of PartitionStore by some minor refactoring.

Test Plan:
mvn clean verify
running pagerank and turning message combiner off on a large graph with limited memory does not fail

Reviewers: maja.kabiljo, sergey.edunov, avery.ching, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D42897

3 years agoGIRAPH-1024mvn release:prepare not committing changes to pom.xml
Sergey Edunov [Wed, 29 Jul 2015 21:31:52 +0000 (14:31 -0700)] 
GIRAPH-1024mvn release:prepare not committing changes to pom.xml

3 years ago[GIRAPH-1022] Adaptive out-of-core mechanism for input superstep and graph
Hassan Eslami [Mon, 27 Jul 2015 18:59:21 +0000 (11:59 -0700)] 
[GIRAPH-1022] Adaptive out-of-core mechanism for input superstep and graph

Summary: This code adds the ability to adaptively control the out-of-core mechanism for graph data structure at run-time during input/output superstep and computation superstep. Basically, the implemented mechanism monitors the amount of available free memory in a separate thread. If there is not enough memory, the code adjusts the number of partitions in memory, and spills a series of partitions/buffers to disk. Also, if the amount of free memory is more than expected, some of the on-disk partitions are brought back to memory. Additionally, if amount of free memory is marginal, the mechanism mocks the memory usage by gradually bringing partitions to memory.

Test Plan:
mvn clean verify
Unit tests added to giraph-core
End-to-end test added to giraph-example
Running the code on PageRank on a large graph and not getting OOM failures.

Reviewers: maja.kabiljo, sergey.edunov, avery.ching, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D40563

3 years agoGIRAPH-1021: Missing progress report for graph mutations
Hassan Eslami [Thu, 16 Jul 2015 22:48:26 +0000 (15:48 -0700)] 
GIRAPH-1021: Missing progress report for graph mutations
Fix progress report for graph mutations

Summary: Progress report in the new implementation of graph mutation is missing. This can cause lack of progress errors at runtime for mutations on high degree vertices

Test Plan: mvn clean verify

Reviewers: avery.ching, dionysis.logothetis, maja.kabiljo, sergey.edunov

Differential Revision: https://reviews.facebook.net/D42309

3 years ago[GIRAPH-1020] TaskInfo equality condition bug fix
Hassan Eslami [Wed, 15 Jul 2015 19:48:22 +0000 (12:48 -0700)] 
[GIRAPH-1020] TaskInfo equality condition bug fix

Summary: Currently equality is checked based on the raw host-name of an object with lower-case host-name of another object. This is not the right semantic and can cause subsequent bugs in partition assignment and migration.

Test Plan: mvn clean verify

Reviewers: maja.kabiljo, sergey.edunov, dionysis.logothetis, avery.ching

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D42273

3 years ago[GIRAPH-1019] Optimizing and debugging vertex mutation mechanism
Hassan Eslami [Mon, 6 Jul 2015 17:39:53 +0000 (10:39 -0700)] 
[GIRAPH-1019] Optimizing and debugging vertex mutation mechanism

Summary:
The old implementation of vertex mutation mechanism was single-threaded and had some redundant computation. The single threaded behavior is causing a huge performance degradation for out-of-core case, since all the partitions are being read and written sequentially in one thread to apply mutations. Also, in case where a vertex is mutated and has messages at the same time, the current code fails to execute which does not seem to be the expected behavior.

This diff implements an optimized multi-threaded approach for vertex mutations. With this diff, vertex mutation happens at the beginning of processing each partition. Also, parts of partition migration code is modified to migrate mutations as well.

Test Plan: mvn clean verify

Reviewers: avery.ching, maja.kabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D40821

3 years ago[GIRAPH-1013] Adding more libraries, algos and examples
Igor Kabiljo [Tue, 30 Jun 2015 06:20:58 +0000 (23:20 -0700)] 
[GIRAPH-1013] Adding more libraries, algos and examples

Summary:
Adding more libraries, algos and examples

Only changes from our internal state:

New classes:
PairReduce
MaxMessageCombiner
PartitioningStats
TestMessageChain

Change to:
Pieces
SendMessageChain

Test Plan: mvn clean install -Phadoop_facebook

Reviewers: maja.kabiljo, dionysis.logothetis, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D40935

3 years ago[GIRAPH-1013] Adding prepare graph library in Java8
Igor Kabiljo [Thu, 25 Jun 2015 21:13:18 +0000 (14:13 -0700)] 
[GIRAPH-1013] Adding prepare graph library in Java8

Summary:
Adding simple graph preparation:
- symmetric
- removal of isolated edges
- normalizing
- connected components

Creating a new module, only used for Phadoop_facebook, which is written in Java8.

Tests are new/modified from what is in our repo, the rest is identical.

Test Plan: mvn clean install

Reviewers: maja.kabiljo, dionysis.logothetis, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D40719

3 years agoGIRAPH-1018: Improving PartitionStore API to better match its expected behaviour
Hassan Eslami [Mon, 29 Jun 2015 22:47:12 +0000 (15:47 -0700)] 
GIRAPH-1018: Improving PartitionStore API to better match its expected behaviour
(heslami via aching)

Summary: Currently for statistics operations on each partition, entire partition is loaded using getOrCreatePartition method of PartitionStore. This diff improves the API of PartitionStore by adding required methods to only return the statistics.

Test Plan: mvn clean verify

Reviewers: dionysis.logothetis, maja.kabiljo, avery.ching

Reviewed By: avery.ching

Differential Revision: https://reviews.facebook.net/D40731

3 years ago[GIRAPH 1013] Adding TestGraphUtils and NumericTestGraph
Igor Kabiljo [Tue, 23 Jun 2015 04:44:49 +0000 (21:44 -0700)] 
[GIRAPH 1013] Adding TestGraphUtils and NumericTestGraph

Summary:
Adding simplified framework for running application tests.
Code for testing is going to be much shorter, especially
when Java 8 is used.

Only difference compared to our codebase is addition of
SendingMessagesTest to showcase these capabilities

Test Plan: mvn clean install

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D40533

3 years ago[GIRAPH-1013] Adding reducer handle utilities
Igor Kabiljo [Wed, 17 Jun 2015 19:47:52 +0000 (12:47 -0700)] 
[GIRAPH-1013] Adding reducer handle utilities

Summary: And more functional interfaces, and PairWritable

Test Plan: mvn clean install

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D40269

3 years agoGIRAPH-1017: Add support for ImmutableMap in Kryo
Maja Kabiljo [Tue, 23 Jun 2015 23:48:55 +0000 (16:48 -0700)] 
GIRAPH-1017: Add support for ImmutableMap in Kryo

Summary: Trying to serialize ImmutableMap currently throws an exception - we should add a support for it.

Test Plan: Added a test, verified that app which was failing without the change passes now

Reviewers: ikabiljo

Differential Revision: https://reviews.facebook.net/D40575

3 years ago[GIRAPH 1013] Apply @edunov fix for block output
Igor Kabiljo [Thu, 18 Jun 2015 23:52:22 +0000 (16:52 -0700)] 
[GIRAPH 1013] Apply @edunov fix for block output

Summary:
Apply fix:
https://phabricator.fb.com/D2141200

Test Plan: mvn clean install

Reviewers: maja.kabiljo, sergey.edunov, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D40395

3 years agoGIRAPH-1015: Support vertex combiner in TestGraph
Maja Kabiljo [Wed, 17 Jun 2015 00:30:22 +0000 (17:30 -0700)] 
GIRAPH-1015: Support vertex combiner in TestGraph

Summary: TestGraph should use vertex combiner which is specified in the conf passed, instead of replacing the vertex with latest added.

Test Plan: Added a test, mvn clean verify

Reviewers: sergey.edunov, ikabiljo

Differential Revision: https://reviews.facebook.net/D40227

3 years ago[GIRAPH-1013] Add library of common pieces and functions
Igor Kabiljo [Wed, 10 Jun 2015 22:33:35 +0000 (15:33 -0700)] 
[GIRAPH-1013] Add library of common pieces and functions

Summary:
StripingUtils has been modified, to be compiled with Java7, and to
have snippet of MIT lincense for used hash algorithm.

Test Plan: mvn clean install

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D39915

3 years ago[GIRAPH-1013] Add migration library
Igor Kabiljo [Wed, 10 Jun 2015 17:33:04 +0000 (10:33 -0700)] 
[GIRAPH-1013] Add migration library

Summary:
Add library that simplifies migration to Blocks Framework

Copied one of the example tests, that use both master, computation and worker context,
to show it all works without any code change

Test Plan: mvn clean install -Phadoop_facebook

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D39891

3 years ago[GIRAPH-1013] Cleanup use of conf for local testing
Igor Kabiljo [Thu, 11 Jun 2015 23:17:53 +0000 (16:17 -0700)] 
[GIRAPH-1013] Cleanup use of conf for local testing

Summary:
Right now we are creating two immutable conf options, and using them both,
which is unnecessary and confusing.

Change to do it only once, and not need to pass it around (TestGraph has it)

Test Plan: mvn clean install

Reviewers: laxman.dhulipala, maja.kabiljo, dionysis.logothetis, sergey.edunov

Differential Revision: https://reviews.facebook.net/D39987

3 years agoGIRAPH-1014: Decrease number of nifty threads created
Maja Kabiljo [Sat, 13 Jun 2015 02:06:12 +0000 (19:06 -0700)] 
GIRAPH-1014: Decrease number of nifty threads created

Summary: By default, ThriftClientManager creates 2*numProcessors threads, making it harder to look through jstack. We use them just for job progress reporting, so no need to have that many.

Test Plan: Run a job, verified number of threads decreased

Reviewers: ikabiljo, sergey.edunov

Differential Revision: https://reviews.facebook.net/D40125

3 years ago[GIRAPH-1013] Add BlockExecutionTest
Igor Kabiljo [Mon, 8 Jun 2015 23:24:45 +0000 (16:24 -0700)] 
[GIRAPH-1013] Add BlockExecutionTest

Summary:
Add support for executing single blocks, as well as adding a test for core of the framework

Equivalent to internal https://phabricator.fb.com/D2137589 diff.

Test Plan: mvn clean install

Reviewers: maja.kabiljo, dionysis.logothetis, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D39873

3 years ago[GIRAPH-1013] Add local (single machine) implementation
Igor Kabiljo [Mon, 8 Jun 2015 18:48:28 +0000 (11:48 -0700)] 
[GIRAPH-1013] Add local (single machine) implementation

Summary:
This allows you to run application written in Blocks Framework
very efficiently on single machine.

Specifically this is interesting for having fast unit tests.

Test Plan:
mvn clean install -Phadoop_facebook

Making TargetVertexIdIterator public is in addition to just adding classes to open source

Reviewers: maja.kabiljo, dionysis.logothetis, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D39717

3 years agoGIRAPH-1012: Remove giraph-hive
Maja Kabiljo [Fri, 12 Jun 2015 18:48:18 +0000 (11:48 -0700)] 
GIRAPH-1012: Remove giraph-hive

Summary: We are not using hive-io-experimental anymore and we'll be deprecating that project. Since we are not aware of anyone else using it, we are thinking of removing giraph-hive completely from the repository. Please comment if you have any objections.

Test Plan: compile with different profiles

Reviewers: ikabiljo, sergey.edunov

Differential Revision: https://reviews.facebook.net/D40053

3 years ago[GIRAPH-1013] Add core of Blocks Framework
Igor Kabiljo [Fri, 5 Jun 2015 07:19:41 +0000 (00:19 -0700)] 
[GIRAPH-1013] Add core of Blocks Framework

Summary:
Add all classes in the core of the Framework.

This is the full execution engine of the framework.

New module giraph-block-app is created for it, and all framework
classes are going into org.apache.giraph.block_app.framework,
and all non-framework-internal classes are going to go into
subpackages of org.apache.giraph.block_app (i.e. utilities for
writing applications, etc)

Test Plan: mvn clean install

Reviewers: maja.kabiljo, dionysis.logothetis, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D39639

3 years ago[GIRAPH-1010] Add utilities to allow Kryo to serialize objects
Igor Kabiljo [Wed, 3 Jun 2015 20:35:20 +0000 (13:35 -0700)] 
[GIRAPH-1010] Add utilities to allow Kryo to serialize objects

Test Plan: mvn clean install

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D39513

3 years ago[GIRAPH-1011] Update checkstyle to newest version
Igor Kabiljo [Wed, 3 Jun 2015 18:29:56 +0000 (11:29 -0700)] 
[GIRAPH-1011] Update checkstyle to newest version

Summary:
Checkstyle has updated/fixed/added bunch of checks.

For identation, style we are using is different from default, so
changes to Indentation rule are needed, to match current style.

New line after left curly brace wasn't used always, so I removed that check.

RedundantThrows check has been removed.

Test Plan:
mvn clean install -Phadoop_facebook

It is still complaining about:
[ERROR] src/main/resources/org/apache/giraph/versions.properties[1] (header) Header: Line does not match expected header line of '/*'.

should we remove non .java files from checking?

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D39483

3 years ago[GIRAPH-1006] Remove boxing when not necessary for partitioned jobs
Igor Kabiljo [Wed, 6 May 2015 21:44:50 +0000 (14:44 -0700)] 
[GIRAPH-1006] Remove boxing when not necessary for partitioned jobs

Summary:

Test Plan: mvn clean install

Reviewers: maja.kabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Subscribers:

Differential Revision: https://reviews.facebook.net/D38127

3 years agoGets rid of spammy ZooKeeper messages regarding 'lost reservation' at the end of...
Hassan Eslami [Tue, 26 May 2015 19:56:20 +0000 (12:56 -0700)] 
Gets rid of spammy ZooKeeper messages regarding 'lost reservation' at the end of the workers' log

Summary:
Once a Giraph job with more than 1 worker completes running, ZooKeeper usually causes a lot of
unnecessary messages mistakenly informing 'lost reservation' of input splits. This is happening
because one worker closes EPHEMERAL ZooKeeper node and this causes an event on other workers about
each existing input split. Since there is one instance of EPHEMERAL ZooKeeper, once it is closed
it translate as deletion/lost of input splits in other workers. These events are watched by ZooKeeperExt
in other workers causing 'lost reservation' messages, one for each input split, at the end of the worker
log.

A fix to this problem is to avoid watching aforementioned events once computation passes the INPUT_SUPERSTEP.

[JIRA number is GIRAPH-1009]

Test Plan:
Running a Giraph job on more than 1 worker without this patch causes the 'lost reservation' messages at the end
of worker log. Applying this patch eliminates these messages.

'mvn clean verify' returns successfully with this patch.

Reviewers: dionysis.logothetis, avery.ching

Reviewed By: avery.ching

Differential Revision: https://reviews.facebook.net/D38949

3 years agoGIRAPH-1008: Create Computation per thread instead of per partition
Maja Kabiljo [Thu, 21 May 2015 19:08:17 +0000 (12:08 -0700)] 
GIRAPH-1008: Create Computation per thread instead of per partition

Summary: Currently we create Computation per partition, but there can be much more partitions than compute threads, and Computation can hold large objects or pre/post superstep can be expensive.

Test Plan: mvn clean verify, one of the tests was relying on per partition threads so modified that

Reviewers: ikabiljo

Differential Revision: https://reviews.facebook.net/D38793

3 years ago[GIRAPH-1005] Reorder MessageEncodeAndStoreType
Igor Kabiljo [Thu, 30 Apr 2015 23:41:48 +0000 (16:41 -0700)] 
[GIRAPH-1005] Reorder MessageEncodeAndStoreType

Summary: It is useful to have ordering among MessageEncodeAndStoreType, from one with least overhead and least optimized for good partitioning, to one with most overhead and most optimized for good partitioning.

Test Plan: mvn clean install -Phadoop_facebook

Reviewers: maja.kabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D37959

3 years agoGIRAPH-1004
Maja Kabiljo [Thu, 30 Apr 2015 23:23:07 +0000 (16:23 -0700)] 
GIRAPH-1004

Summary: Currently we fix to BspOutputFormat, but for some more fancy things we need to be able to change it and use different commit method.

Test Plan: mvn clean verify, run a job that changes output format

Reviewers: ikabiljo

Differential Revision: https://reviews.facebook.net/D37917

3 years ago[GIRAPH-1003] Support Writable Basic2ObjectMap
Igor Kabiljo [Wed, 15 Apr 2015 21:31:52 +0000 (14:31 -0700)] 
[GIRAPH-1003] Support Writable Basic2ObjectMap

Summary:
Basic2ObjectMap isn't and cannot be writable (since object itself isn't writable), which makes it drastically
less useful - since most of the objects need to be serialized.

But if we don't count no-arg constructor - we can make it Writable - by having valueWriter
passed into through constructor.

Test Plan: mvn clean install

Reviewers: sergey.edunov, dionysis.logothetis, maja.kabiljo

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D36987

3 years ago[GIRAPH-1002] Improve message changing through iters
Igor Kabiljo [Fri, 10 Apr 2015 18:42:28 +0000 (11:42 -0700)] 
[GIRAPH-1002] Improve message changing through iters

Summary:
Add MessageClasses object to hold all information about single
message class (factory, combiner, store type)

Made it so that it can be changed as a whole, allowing
any complete control over messages.

(i.e. message factory couldn't be used when messages where changing)

Incoming message type configs are completely useless. It needs to be the same
as outgoing message type in the previous step. So deleting that completely.

*breaking change* Removed initialize and getValueClass from value factories.
Initialize can be achieved, as with everywhere else, by implementing
GiraphConfigurationSettable. getValueClass was not used anywhere.

This way - value factory has only one function - and lambda can be passed to it.

Test Plan: mvn clean install -Phadoop_facebook

Reviewers: sergey.edunov, dionysis.logothetis, avery.ching, maja.kabiljo

Differential Revision: https://reviews.facebook.net/D36849

3 years agoGIRAPH-998: Close writers in parallel (majaakbiljo)
Maja Kabiljo [Tue, 24 Mar 2015 22:47:44 +0000 (15:47 -0700)] 
GIRAPH-998: Close writers in parallel (majaakbiljo)

3 years agoAdded support for multi-mapping input formats
Dionysios Logothetis [Tue, 24 Mar 2015 22:28:46 +0000 (15:28 -0700)] 
Added support for multi-mapping input formats

Summary:
https://issues.apache.org/jira/browse/GIRAPH-999

Similarly to multi-vertex/edge input formats, I added multi-mapping input formats. The main changes are the addition of a MultiMappingInputFormat and a MappingInputFormatDecription. The rest are some set/get methods in the configuration-related classes.

Test Plan:
Run:
- mvn clean verify
- Run job with 2 mapping input formats (replicated the same mapping input)

Reviewers: sergey.edunov, ikabiljo, maja.kabiljo

Reviewed By: maja.kabiljo

Subscribers: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D35733

3 years agoGIRAPH-997: Upgrade findbugs to 3.0.0 (dlogothetis via majakabiljo)
Maja Kabiljo [Wed, 11 Mar 2015 17:43:39 +0000 (10:43 -0700)] 
GIRAPH-997: Upgrade findbugs to 3.0.0 (dlogothetis via majakabiljo)

3 years agoGIRAPH-996: Large requests degrade performance. Print out warnings. (dlogothetis...
Maja Kabiljo [Wed, 4 Mar 2015 18:39:41 +0000 (10:39 -0800)] 
GIRAPH-996: Large requests degrade performance. Print out warnings. (dlogothetis via majakabiljo)

3 years agoGIRAPH-990. Current trunk will build for hadoop 1.2.0 not 0.20.203 as stated by docum...
Roman Shaposhnik [Tue, 27 Jan 2015 17:45:06 +0000 (09:45 -0800)] 
GIRAPH-990. Current trunk will build for hadoop 1.2.0 not 0.20.203 as stated by documentation

3 years agoGIRAPH-992 Zookeeper logs have too many NodeExists
Sergey Edunov [Thu, 5 Feb 2015 01:26:55 +0000 (17:26 -0800)] 
GIRAPH-992 Zookeeper logs have too many NodeExists

https://phabricator.fb.com/D1827310

3 years agoGIRAPH-991 Update versions of guava and swift
Sergey Edunov [Tue, 3 Feb 2015 22:47:15 +0000 (14:47 -0800)] 
GIRAPH-991 Update versions of guava and swift

Summary: Changes in WorkerProgress are needed because newer version of thrift requires thrift classes to be final.

Test Plan: mvn clean verify
+ run a bunch of test jobs

Reviewers: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D32757

3 years ago[GIRAPH-987] Improve naming for ReduceOperation
Igor Kabiljo [Wed, 28 Jan 2015 18:10:19 +0000 (10:10 -0800)] 
[GIRAPH-987] Improve naming for ReduceOperation

Summary:
reduceSingle/reducePartial can be slightly confusing, changing to reduce/reduceMerge
OnSameReduceOperation => ReduceSameTypeOperation

If you have better suggestions - I am all ears :)

Test Plan: mvn clean install

Reviewers: majakabiljo, sergey.edunov, laxman.dhulipala, maja.kabiljo

Differential Revision: https://reviews.facebook.net/D31785

3 years ago[GIRAPH-986] Add no-arg constructor to BasicSet
Igor Kabiljo [Sat, 24 Jan 2015 00:29:25 +0000 (16:29 -0800)] 
[GIRAPH-986] Add no-arg constructor to BasicSet

Summary:
https://reviews.facebook.net/D31731 added empty constructors for BasicArrayList,
we need them for BasicSet as well (contract of Writable is two methods and no-arg constructor)

Test Plan: mvn clean install

Reviewers: maja.kabiljo, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D32091

3 years agoGIRAPH-985 addition: Making sum counter come first
Igor Kabiljo [Thu, 22 Jan 2015 18:36:32 +0000 (10:36 -0800)] 
GIRAPH-985 addition: Making sum counter come first

Test Plan: mvn clean install, run a job, confirmed it is first

Reviewers: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D31965

3 years agoGIRAPH-985: Add more metrics
Maja Kabiljo [Fri, 16 Jan 2015 17:45:46 +0000 (09:45 -0800)] 
GIRAPH-985: Add more metrics

Summary: When limit number of open requests is on, significant part of computation can be spent in just waiting. So adding metrics for total amount of time compute threads spent in waiting on open requests, and histogram with compute times per partition. These should help detecting why some job is slower than expected (is it from messaging or compute). Also changed the way we wait on open requests, so we continue on different limit than we stop, so we wouldn't have huge number of very tiny pauses.

Test Plan: Run some jobs on the cluster and looked through these metrics.

Reviewers: sergey.edunov, pavanka

Subscribers: ikabiljo

Differential Revision: https://reviews.facebook.net/D31683

3 years ago[GIRAPH-986] Add more stuff to TypeOps
Igor Kabiljo [Sat, 17 Jan 2015 00:30:29 +0000 (16:30 -0800)] 
[GIRAPH-986] Add more stuff to TypeOps

Summary:
Add:
createZero/createOne to NumericTypeOps
unwrap, fill to BasicArrayList
AndReduce and OrReduce

Test Plan: mvn clean install

Reviewers: majakabiljo, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Subscribers: laxman.dhulipala

Differential Revision: https://reviews.facebook.net/D31731

3 years agoGIRAPH-962: TextAggregatorWriter with frequency AT_THE_END writes in every superstep...
Maja Kabiljo [Fri, 16 Jan 2015 18:41:34 +0000 (10:41 -0800)] 
GIRAPH-962: TextAggregatorWriter with frequency AT_THE_END writes in every superstep (mju via majakabiljo)

3 years agoGIRAPH-967: (Strongly Connected Components) bug fix (mhan via majakabiljo)
Maja Kabiljo [Fri, 16 Jan 2015 18:07:01 +0000 (10:07 -0800)] 
GIRAPH-967: (Strongly Connected Components) bug fix (mhan via majakabiljo)

3 years agoGIRAPH-980 Way to disable checkpoints for particular job and on particular supersteps
Sergey Edunov [Wed, 14 Jan 2015 22:06:09 +0000 (14:06 -0800)] 
GIRAPH-980 Way to disable checkpoints for particular job and on particular supersteps

Summary:
Added CheckpointSupportedChecker, using this interface one can specify whether the job is checkpointable or not.
By default all jobs that don't do output during the computation are checkpointable.

Test Plan: Run several jobs with output during the computation enabled and disabled. Also wrote custom CheckpointSupportedChecker to test if I can disable checkpoints on particular superstep

Reviewers: majakabiljo, pavanka, pavanka.26, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D31113

3 years agoGIRAPH-983 Remove checkpoint related error messages from console
Sergey Edunov [Wed, 14 Jan 2015 00:32:04 +0000 (16:32 -0800)] 
GIRAPH-983 Remove checkpoint related error messages from console

3 years agoGIRAPH-978 Giraph-Debugger Test Graphs not working (nishantgandhi99 via edunov)
Sergey Edunov [Thu, 8 Jan 2015 23:08:39 +0000 (15:08 -0800)] 
GIRAPH-978 Giraph-Debugger Test Graphs not working (nishantgandhi99 via edunov)

Summary:
Giraph-Debugger has feature of creating graphs for test input. The GUI panel has options of selecting predefined graph types with number of nodes. The corresponding graph is created on panel and test code is also generated automatically.
Now there are six types graph available in the drop-down list but only first two types of graph are working.
In this patch, I have implemented another two types of Graph.

Test Plan: Manual Testing.

Reviewers: semihsalihoglu, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D31137

3 years agoGIRAPH-979: Add type of input to 'missing input' error message (majakabiljo)
Maja Kabiljo [Wed, 7 Jan 2015 18:59:13 +0000 (10:59 -0800)] 
GIRAPH-979: Add type of input to 'missing input' error message (majakabiljo)

3 years agoGIRAPH-975 In-proc ZooKeeper server with Master process
Sergey Edunov [Wed, 24 Dec 2014 00:04:11 +0000 (16:04 -0800)] 
GIRAPH-975 In-proc ZooKeeper server with Master process

https://reviews.facebook.net/D30693

3 years agoGIRAPH-977: useMessageSizeEncoding is broken
Maja Kabiljo [Wed, 31 Dec 2014 02:51:55 +0000 (18:51 -0800)] 
GIRAPH-977: useMessageSizeEncoding is broken

Summary: When useMessageSizeEncoding is set to true, any application using messaging and no combiner will fail. It looks like writing lengths of messages was lost at some point, so we are just trying to read the lengths without ever writing them.

Test Plan: Added a test, it failed without the change but succeeded with it.

Reviewers: sergey.edunov, pavanka

Differential Revision: https://reviews.facebook.net/D30825

3 years agoGIRAPH-976: More command line logging
Maja Kabiljo [Mon, 29 Dec 2014 19:47:26 +0000 (11:47 -0800)] 
GIRAPH-976: More command line logging

Summary:
Add more logging to command line:
- if a worker fails log which worker failed
- if some worker is low on memory, warn user about it

Test Plan:
Run a job where worker fails, saw:
  FATAL   2014-12-29 11:29:25,879 ******* WORKERS [Worker(hostname=hadoop5247.frc1.facebook.com, MRtaskID=2, port=30002)] FAILED *******
Run a job low on memory, saw:
  INFO    2014-12-29 11:40:55,731 Data from 5 workers - Compute superstep 0: 1040758 out of 1302981 vertices computed; 20 out of 25 partitions computed; min free memory on worker 2 - 3775.79MB, average 94126.79MB, ******* YOUR JOB IS RUNNING LOW ON MEMORY *******
Run normal job verified everything is as before.

Reviewers: pavanka, sergey.edunov

Differential Revision: https://reviews.facebook.net/D30729

3 years agoGIRAPH-973 Edge trimming no longer works before superstep 0
Sergey Edunov [Fri, 19 Dec 2014 01:43:43 +0000 (17:43 -0800)] 
GIRAPH-973 Edge trimming no longer works before superstep 0

3 years agoGIRAPH-972 Race condition in checkpointing
Sergey Edunov [Thu, 18 Dec 2014 18:05:36 +0000 (10:05 -0800)] 
GIRAPH-972 Race condition in checkpointing

4 years agoGIRAPH-905 Giraph Debugger
Sergey Edunov [Mon, 8 Dec 2014 19:21:19 +0000 (11:21 -0800)] 
GIRAPH-905 Giraph Debugger

4 years agoImproving and adding reducers
Igor Kabiljo [Mon, 1 Dec 2014 22:21:12 +0000 (14:21 -0800)] 
Improving and adding reducers

Summary:
- adding common reducers - Max/Min/Sum Reducer. As you can see - no code duplication
- adding NumericTypeOps functions to support the above
- added Varint encoding class
- renaming registerReduce -> registerReducer

Test Plan: mvn install

Reviewers: pavanka, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D28983

4 years agoFix findbugs from GIRAPH-966 commit
Maja Kabiljo [Mon, 1 Dec 2014 20:01:10 +0000 (12:01 -0800)] 
Fix findbugs from GIRAPH-966 commit

4 years agoGIRAPH-966: Add a way to ignore some thread exceptions
Maja Kabiljo [Mon, 1 Dec 2014 19:39:33 +0000 (11:39 -0800)] 
GIRAPH-966: Add a way to ignore some thread exceptions

4 years agoOpening up 1.2.0-SNAPSHOT development now that 1.1.0 has been released
Roman Shaposhnik [Tue, 25 Nov 2014 23:40:18 +0000 (15:40 -0800)] 
Opening up 1.2.0-SNAPSHOT development now that 1.1.0 has been released

4 years agoGIRAPH-964: Remove quotes from output partition specification in hive-io
Maja Kabiljo [Fri, 7 Nov 2014 18:47:20 +0000 (10:47 -0800)] 
GIRAPH-964: Remove quotes from output partition specification in hive-io

4 years agoGIRAPH-963 Aggregators may not be initialized properly
Sergey Edunov [Wed, 5 Nov 2014 05:47:44 +0000 (21:47 -0800)] 
GIRAPH-963 Aggregators may not be initialized properly

Summary: Aggregators may fail with IllegalArgumentException upon deserialization

Test Plan: Run same job

Reviewers: majakabiljo, pavanka, pavanka.26, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D28227

4 years agoFix removal of MasterLoggingAggregator
Igor Kabiljo [Tue, 28 Oct 2014 21:56:08 +0000 (14:56 -0700)] 
Fix removal of MasterLoggingAggregator

Test Plan:
mvn clean install

How should I test this?

Reviewers: pavanka, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D27651

4 years agoAllow vertex/edge/message value to be configurable
Igor Kabiljo [Tue, 28 Oct 2014 17:59:37 +0000 (10:59 -0700)] 
Allow vertex/edge/message value to be configurable

Summary: And throw if WritableUtils.createWritable(clazz) is called with GiraphConfigurationSettable.

Test Plan: mvn install

Reviewers: sergey.edunov, pavanka, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D25353

4 years agoGIRAPH-960: Max task attempts should be 1 and not 0.
Avery Ching [Mon, 27 Oct 2014 23:12:01 +0000 (16:12 -0700)] 
GIRAPH-960: Max task attempts should be 1 and not 0.

Summary: 0 is an invalid number that may not work on some Hadoop distributions

Test Plan: Ran a job and failed a task.

Reviewers: pavanka.26, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D27579

4 years agoGIRAPH-958. .arcconfig needs to be added to rat exclude
Roman Shaposhnik [Sun, 26 Oct 2014 05:08:03 +0000 (22:08 -0700)] 
GIRAPH-958. .arcconfig needs to be added to rat exclude

4 years agoGIRAPH-956. disable tests for a few components in hadoop_2 profile
Roman Shaposhnik [Sat, 25 Oct 2014 23:45:21 +0000 (16:45 -0700)] 
GIRAPH-956. disable tests for a few components in hadoop_2 profile

4 years agoAuto-restart from checkpoint doesn't pick up latest checkpoint
Sergey Edunov [Mon, 20 Oct 2014 17:08:56 +0000 (10:08 -0700)] 
Auto-restart from checkpoint doesn't pick up latest checkpoint

Summary:
While running different jobs with checkpoints enabled I noticed some issues:
1) The way we pick up latest checkpoint is not correct. Current implementation just picks whatever is returned last from FileSystem.list(), which is not necessarily the last checkpoint
2) If job restarts from checkpoint it immediately creates another checkpoint.
3) We need more flexibility in GiraphJobRetryChecker to allow restarts after multiple failures.

Test Plan: Run our production jobs with checkpointing

Reviewers: majakabiljo, pavanka, pavanka.26, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D23913