giraph.git
4 days agoFix checkstyle trunk
Maja Kabiljo [Thu, 18 Oct 2018 22:21:51 +0000 (15:21 -0700)] 
Fix checkstyle

4 days agoGIRAPH-1205
Maja Kabiljo [Thu, 18 Oct 2018 21:29:38 +0000 (14:29 -0700)] 
GIRAPH-1205

closes #88

4 weeks agofix-tests
Maja Kabiljo [Fri, 21 Sep 2018 21:02:42 +0000 (14:02 -0700)] 
fix-tests

closes #86

4 weeks agoGIRAPH-1201
Dionysios Logothetis [Fri, 21 Sep 2018 17:24:28 +0000 (10:24 -0700)] 
GIRAPH-1201

closes #84

4 weeks agoJIRA-1200
Maja Kabiljo [Tue, 18 Sep 2018 19:25:34 +0000 (12:25 -0700)] 
JIRA-1200

closes #83

2 months agoGIRAPH-1199
Swapnil Gandhi [Thu, 2 Aug 2018 02:28:45 +0000 (19:28 -0700)] 
GIRAPH-1199

Closes #78

3 months agoGIRAPH-1197
Swapnil Gandhi [Tue, 10 Jul 2018 16:43:57 +0000 (09:43 -0700)] 
GIRAPH-1197

Closes #77

4 months agoGIRAPH-1195
Dionysios Logothetis [Thu, 31 May 2018 18:07:33 +0000 (11:07 -0700)] 
GIRAPH-1195

closes #76

4 months agoGIRAPH-299
Dionysios Logothetis [Thu, 31 May 2018 17:41:07 +0000 (10:41 -0700)] 
GIRAPH-299

closes #75

4 months agoGIRAPH-1194
Dionysios Logothetis [Sat, 26 May 2018 01:26:33 +0000 (18:26 -0700)] 
GIRAPH-1194

closes #74

4 months agoGIRAPH-1193
Dionysios Logothetis [Sat, 26 May 2018 01:24:00 +0000 (18:24 -0700)] 
GIRAPH-1193

GIRAPH-1176

closes #73

5 months agoGIRAPH-1188
Yuksel Akinci [Tue, 15 May 2018 05:45:14 +0000 (22:45 -0700)] 
GIRAPH-1188

closes #70

5 months agoGIRAPH-1192
Bruno P. Kinoshita [Thu, 3 May 2018 02:42:00 +0000 (19:42 -0700)] 
GIRAPH-1192

closes #45

5 months agoGIRAPH-1191
Gabor Szarnyas [Wed, 2 May 2018 17:55:19 +0000 (10:55 -0700)] 
GIRAPH-1191

closes #31

6 months agoGIRAPH-1185
Maja Kabiljo [Thu, 12 Apr 2018 15:59:13 +0000 (08:59 -0700)] 
GIRAPH-1185

closes #69

6 months agoGIRAPH-1184
Maja Kabiljo [Thu, 5 Apr 2018 19:14:08 +0000 (12:14 -0700)] 
GIRAPH-1184

closes #68

6 months agoGIRAPH-1181
Maja Kabiljo [Wed, 4 Apr 2018 22:22:24 +0000 (15:22 -0700)] 
GIRAPH-1181

closes #65

6 months agoGIRAPH-1183
Maja Kabiljo [Thu, 29 Mar 2018 16:22:59 +0000 (09:22 -0700)] 
GIRAPH-1183

closes #67

7 months agoGIRAPH-1182
Maja Kabiljo [Thu, 22 Mar 2018 19:52:55 +0000 (12:52 -0700)] 
GIRAPH-1182

closes #66

7 months agocloses #64
Dionysios Logothetis [Tue, 13 Mar 2018 23:44:01 +0000 (16:44 -0700)] 
closes #64

7 months agoGIRAPH-1174
Maja Kabiljo [Fri, 9 Mar 2018 22:03:41 +0000 (14:03 -0800)] 
GIRAPH-1174

closes #62

7 months agoGIRAPH-1175
Dionysios Logothetis [Tue, 13 Mar 2018 22:17:12 +0000 (15:17 -0700)] 
GIRAPH-1175

closes #63

8 months agoJIRA-1171
Maja Kabiljo [Wed, 21 Feb 2018 20:45:54 +0000 (12:45 -0800)] 
JIRA-1171

closes #60

8 months agocloses #59
Maja Kabiljo [Tue, 13 Feb 2018 23:42:12 +0000 (15:42 -0800)] 
closes #59

8 months agoJIRA-1169
Maja Kabiljo [Mon, 12 Feb 2018 18:52:38 +0000 (10:52 -0800)] 
JIRA-1169

closes #58

8 months agoGIRAPH-1168
Maja Kabiljo [Thu, 1 Feb 2018 22:08:27 +0000 (14:08 -0800)] 
GIRAPH-1168

closes #57

9 months agoGIRAPH-1167
Dionysios Logothetis [Sat, 13 Jan 2018 05:00:55 +0000 (21:00 -0800)] 
GIRAPH-1167

closes #56

9 months agoJIRA-1166
Maja Kabiljo [Wed, 10 Jan 2018 23:56:49 +0000 (15:56 -0800)] 
JIRA-1166

closes #55

11 months agoGIRAPH-1164
Dionysios Logothetis [Wed, 1 Nov 2017 21:40:08 +0000 (14:40 -0700)] 
GIRAPH-1164

closes #53

12 months agoGIRAPH-1163
Maja Kabiljo [Thu, 12 Oct 2017 20:26:18 +0000 (13:26 -0700)] 
GIRAPH-1163

closes #52

12 months agoGIRAPH-1162
Yuksel Akinci [Mon, 2 Oct 2017 21:41:32 +0000 (14:41 -0700)] 
GIRAPH-1162

closes #51

12 months agoGIRAPH-1161
Jianlong Zhong [Fri, 29 Sep 2017 17:36:27 +0000 (10:36 -0700)] 
GIRAPH-1161

closes #50

13 months agoJIRA-1160
Maja Kabiljo [Thu, 21 Sep 2017 14:35:53 +0000 (07:35 -0700)] 
JIRA-1160

closes #49

13 months agoJIRA-1159
Maja Kabiljo [Tue, 19 Sep 2017 16:01:24 +0000 (09:01 -0700)] 
JIRA-1159

closes #48

13 months agoGIRAPH-1158
Maja Kabiljo [Fri, 15 Sep 2017 18:37:58 +0000 (11:37 -0700)] 
GIRAPH-1158

closes #47

13 months agoJIRA-1157
Maja Kabiljo [Thu, 7 Sep 2017 23:49:44 +0000 (16:49 -0700)] 
JIRA-1157

closes #46

13 months agoGIRAPH-1154
Dionysios Logothetis [Mon, 28 Aug 2017 23:43:57 +0000 (16:43 -0700)] 
GIRAPH-1154

closes #44

13 months agoJIRA-1152
Sagi Bashari [Mon, 14 Aug 2017 08:31:09 +0000 (11:31 +0300)] 
JIRA-1152

closes #41

13 months agoGIRAPH-1139
Nicholas Eggert [Mon, 28 Aug 2017 17:19:34 +0000 (10:19 -0700)] 
GIRAPH-1139

closes #30

13 months agoJIRA-1153
Dionysios Logothetis [Thu, 24 Aug 2017 18:15:53 +0000 (11:15 -0700)] 
JIRA-1153

closes 43

14 months agoJIRA-1151
Yuksel Akinci [Tue, 22 Aug 2017 00:45:20 +0000 (17:45 -0700)] 
JIRA-1151

closes 42

16 months agoJIRA-1149
Dionysios Logothetis [Fri, 2 Jun 2017 18:38:18 +0000 (11:38 -0700)] 
JIRA-1149

closes #40

16 months agoJIRA-1148
Maja Kabiljo [Fri, 2 Jun 2017 17:14:18 +0000 (10:14 -0700)] 
JIRA-1148

closes #39

17 months agoJIRA-1147
Maja Kabiljo [Fri, 19 May 2017 01:08:46 +0000 (18:08 -0700)] 
JIRA-1147

closes #38

17 months agoJIRA-1138
Maja Kabiljo [Wed, 10 May 2017 16:45:14 +0000 (09:45 -0700)] 
JIRA-1138

closes #37

17 months agoJIRA-1146
Maja Kabiljo [Wed, 10 May 2017 14:31:15 +0000 (07:31 -0700)] 
JIRA-1146

closes #36

18 months agoGIRAPH-1143
Avery Ching [Mon, 17 Apr 2017 19:05:50 +0000 (12:05 -0700)] 
GIRAPH-1143

closes 35

18 months agoJIRA-1141
Maja Kabiljo [Tue, 4 Apr 2017 16:38:43 +0000 (09:38 -0700)] 
JIRA-1141

closes #33

18 months agoFix #27
Maja Kabiljo [Fri, 31 Mar 2017 03:43:30 +0000 (20:43 -0700)] 
Fix #27

18 months agoJIRA-1138
Maja Kabiljo [Thu, 30 Mar 2017 20:23:00 +0000 (13:23 -0700)] 
JIRA-1138

closes #27

18 months agocloses #28
Dionysios Logothetis [Thu, 30 Mar 2017 16:19:41 +0000 (11:19 -0500)] 
closes #28

18 months agoFix findbug issue
Hassan Eslami [Thu, 30 Mar 2017 00:30:21 +0000 (19:30 -0500)] 
Fix findbug issue

18 months agocloses #29
Dionysios Logothetis [Wed, 29 Mar 2017 16:56:59 +0000 (09:56 -0700)] 
closes #29

18 months agoJIRA-1137
Hassan Eslami [Mon, 27 Mar 2017 18:22:09 +0000 (13:22 -0500)] 
JIRA-1137

closes #26

19 months agoJIRA-1134
Maja Kabiljo [Fri, 17 Mar 2017 17:40:53 +0000 (10:40 -0700)] 
JIRA-1134

closes #24

19 months agocloses #25
Sergey Edunov [Wed, 15 Mar 2017 17:00:58 +0000 (10:00 -0700)] 
closes #25

19 months agoJIRA-1133
Maja Kabiljo [Tue, 7 Mar 2017 20:50:45 +0000 (12:50 -0800)] 
JIRA-1133

closes #22

19 months agoGIRAPH-1132
Sergey Edunov [Wed, 1 Mar 2017 22:05:54 +0000 (14:05 -0800)] 
GIRAPH-1132

closes #21

19 months agocloses #20
Dionysios Logothetis [Mon, 27 Feb 2017 18:39:56 +0000 (10:39 -0800)] 
closes #20

20 months agoGIRAPH-1130 Fix RepeatUntilBlock
Igor Kabiljo [Fri, 27 Jan 2017 19:34:06 +0000 (11:34 -0800)] 
GIRAPH-1130 Fix RepeatUntilBlock

closes #16

21 months agoGIRAPH-1129
Igor Kabiljo [Fri, 20 Jan 2017 18:51:20 +0000 (10:51 -0800)] 
GIRAPH-1129

closes #15

21 months agoGIRAPH-1129
Igor Kabiljo [Fri, 13 Jan 2017 19:20:12 +0000 (11:20 -0800)] 
GIRAPH-1129

closes #14

21 months agoGIRAPH-1129
Igor Kabiljo [Fri, 13 Jan 2017 18:48:45 +0000 (10:48 -0800)] 
GIRAPH-1129

closes #13

21 months agoGIRAPH-1128. Giraph does not build because of maven-dependency-plugin (patch submitte...
Roman Shaposhnik [Fri, 13 Jan 2017 02:25:50 +0000 (18:25 -0800)] 
GIRAPH-1128. Giraph does not build because of maven-dependency-plugin (patch submitted by Naresh Bafna)

21 months agoAdd missing files for GIRAPH-1125. Closes #12
Sergey Edunov [Tue, 27 Dec 2016 21:49:25 +0000 (13:49 -0800)] 
Add missing files for GIRAPH-1125. Closes #12

21 months agoGIRAPH-1125
Hassan Eslami [Fri, 23 Dec 2016 18:03:37 +0000 (12:03 -0600)] 
GIRAPH-1125

Closes #12

22 months agoFix typo
Sergey Edunov [Tue, 29 Nov 2016 00:23:58 +0000 (16:23 -0800)] 
Fix typo

Author: edunov

Closes #11

22 months agoCorrect typo in word "initialize"
KidEinstein [Tue, 29 Nov 2016 00:16:07 +0000 (16:16 -0800)] 
Correct typo in word "initialize"

Author: KidEinstein

Reviewer: edunov

Closes #10

23 months agoGIRAPH-1124 - Create documentation on how to make Giraph release
Sergey Edunov [Fri, 18 Nov 2016 19:19:16 +0000 (11:19 -0800)] 
GIRAPH-1124 - Create documentation on how to make Giraph release

Test Plan: mvn clean site

Reviewers: rvs, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D65313

23 months agoFix Checkstyle
Sergey Edunov [Mon, 14 Nov 2016 19:58:59 +0000 (11:58 -0800)] 
Fix Checkstyle

Test Plan:
mvn clean site -DskipTests -Phadoop_2 -Ddependency.locations.enabled=false
mvn clean install -Phadoop_2 -Prelease
mvn clean install -Phadoop_1 -Prelease

Reviewers: dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D65499

23 months agoBump Apache Giraph version to 1.3.0-SNAPSHOT
Sergey Edunov [Tue, 25 Oct 2016 00:20:02 +0000 (17:20 -0700)] 
Bump Apache Giraph version to 1.3.0-SNAPSHOT

Test Plan: none

Reviewers: dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D65391

2 years agoGIRAPH-1122 Javadoc generation fails for Giraph 1.2.0
Sergey Edunov [Fri, 14 Oct 2016 20:43:50 +0000 (13:43 -0700)] 
GIRAPH-1122 Javadoc generation fails for Giraph 1.2.0

Test Plan: mvn clean site -DskipTests -Phadoop_2 -Ddependency.locations.enabled=false

Reviewers: majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64995

2 years agoFixing RAT checks for Apache Giraph release
Sergey Edunov [Tue, 11 Oct 2016 23:49:16 +0000 (16:49 -0700)] 
Fixing RAT checks for Apache Giraph release

Test Plan:
mvn apache-rat:check -Phadoop_2
mvn apache-rat:check -Phadoop_1
mvn clean verify -Phadoop_facebook

Reviewers: maja.kabiljo, majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64917

2 years agoGIRAPH-1118 - Giraph-gora and Giraph-rexster test cases fail in release-1.2
Sergey Edunov [Thu, 6 Oct 2016 18:13:49 +0000 (11:13 -0700)] 
GIRAPH-1118 - Giraph-gora and Giraph-rexster test cases fail in release-1.2

Test Plan:
mvn clean verify -Phadoop_facebook
rm -rf ~/.m2/repository/org/apache/giraph
mvn clean install -Phadoop_1
rm -rf ~/.m2/repository/org/apache/giraph
mvn clean install -Phadoop_2

Reviewers: maja.kabiljo, majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64719

2 years ago GIRAPH-1118 - Giraph-gora and Giraph-rexster test cases fail in release-1.2
Sergey Edunov [Wed, 5 Oct 2016 22:05:58 +0000 (15:05 -0700)] 
 GIRAPH-1118 - Giraph-gora and Giraph-rexster test cases fail in release-1.2

Test Plan:
mvn clean verify -Phadoop_facebook
mvn clean install -Phadoop_1
mvn clean install -Phadoop_2

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D64683

2 years ago[GIRAPH-1117] Provide a flexible way to decide whether to create vertex when it is...
Sergey Edunov [Thu, 29 Sep 2016 23:54:18 +0000 (16:54 -0700)] 
[GIRAPH-1117] Provide a flexible way to decide whether to create vertex when it is not present in the input

Test Plan: run hello pagerank with this feature on and off

Reviewers: majakabiljo, maja.kabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64485

2 years agoGIRAPH-1094 remove hbase1 from distribution for hadoop_1
Sergey Edunov [Wed, 21 Sep 2016 21:47:10 +0000 (14:47 -0700)] 
GIRAPH-1094 remove hbase1 from distribution for hadoop_1

Summary: Missed that part in the last diff.

Test Plan:
mvn clean package -Phadoop_2 -fae
then checked that giraph-hbase.jar is in the distribution

mvn clean package -Phadoop_1 -fae
then checked that giraph-hbase.jar is not in the distribution

Reviewers: maja.kabiljo, majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64203

2 years agoGIRAPH-1094 Remove hbase from hadoop_1
Sergey Edunov [Wed, 21 Sep 2016 18:04:34 +0000 (11:04 -0700)] 
GIRAPH-1094 Remove hbase from hadoop_1

Summary: Hadoop_1 and current versions of hbase are incompatible. Removing support for HBASE from Hadoop_1 profile

Test Plan: mvn clean package -Phadoop_1 -fae

Reviewers: majakabiljo, maja.kabiljo, dionysis.logothetis

Reviewed By: maja.kabiljo, dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64197

2 years agoGIRAPH-1114: Expose StatusReporter from workers in blocks framework
Maja Kabiljo [Wed, 14 Sep 2016 23:35:22 +0000 (16:35 -0700)] 
GIRAPH-1114: Expose StatusReporter from workers in blocks framework

Summary: Sometimes we need to call progress or update status from workers, expose this functionality

Test Plan: verify

Differential Revision: https://reviews.facebook.net/D63999

2 years agoGIRAPH-1115: Move UncaughtExceptionHandler setup to GraphTaskManager
Maja Kabiljo [Mon, 19 Sep 2016 19:26:49 +0000 (12:26 -0700)] 
GIRAPH-1115: Move UncaughtExceptionHandler setup to GraphTaskManager

Test Plan: Ran a job which isn't using GraphMapper and verified exception handler was set properly

Differential Revision: https://reviews.facebook.net/D64113

2 years agoGIRAPH-1111 - FileOutputFormat#setOutputPath is not always available
Sergey Edunov [Wed, 14 Sep 2016 17:20:25 +0000 (10:20 -0700)] 
GIRAPH-1111 - FileOutputFormat#setOutputPath is not always available

Test Plan:
mvn clean install
+ run a few jobs

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D63837

2 years agofaster maps
spupyrev [Wed, 31 Aug 2016 00:32:13 +0000 (17:32 -0700)] 
faster maps

Summary:
The idea is to replace HashMap<LongWritable, V> to Long2ObjectOpenHashMap<V> (and Map<Int...> to Int2Object...)
This will save space and speed up some applications.

I changed the type of such a map in TestGraph.java, which gives up to 2x speed up on an
example of page rank computation (see comment below)

JIRA: https://issues.apache.org/jira/browse/GIRAPH-1049

Test Plan: TestBasicCollections.java contain some tests

Reviewers: sergey.edunov, maja.kabiljo, dionysis.logothetis, heslami, ikabiljo

Reviewed By: heslami

Differential Revision: https://reviews.facebook.net/D55587

2 years agoGIRAPH-1108: Allow measuring time spent doing GC in some interval
Maja Kabiljo [Fri, 26 Aug 2016 20:51:37 +0000 (13:51 -0700)] 
GIRAPH-1108: Allow measuring time spent doing GC in some interval

Summary: Sometimes when things are slow, we want to know whether it's because of GC or not. Keep track of last k GC pauses and a way to check how much time since some timestamp was spent doing GC.

Test Plan: Ran a job which periodically prints stats from this and manually verified based on GC logs that it's measuring it correctly

Differential Revision: https://reviews.facebook.net/D62727

2 years agoOut-of-core is logging too aggressively
Tyler Serdar Bulut [Tue, 30 Aug 2016 18:31:13 +0000 (13:31 -0500)] 
Out-of-core is logging too aggressively

Summary:
Example aggressive logging at INFO level:

INFO    <datestamp> [ooc-io-0] org.apache.giraph.ooc.policy.ThresholdBasedOracle  - getNextIOActions: usedMemoryFraction = 0.79
INFO    <datestamp> [ooc-io-0] org.apache.giraph.ooc.OutOfCoreIOCallable  - call: thread 0's next IO command is: LoadPartitionIOCommand: (partitionId = 4676, superstep = 0)
INFO    <datestamp> [ooc-io-0] org.apache.giraph.ooc.OutOfCoreIOCallable  - call: thread 0's command LoadPartitionIOCommand: (partitionId = 4676, superstep = 0) completed: bytes= 0, duration=0, bandwidth=NaN, bandwidth (excluding GC time)=NaN

Test Plan: mvn clean verify -P hadoop_facebook

Reviewers: majakabiljo, maja.kabiljo, sergey.edunov, heslami

Reviewed By: heslami

Subscribers: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D62853

2 years agoGIRAPH-1103: Another try to fix jobs getting stuck after channel failure
Maja Kabiljo [Mon, 8 Aug 2016 18:13:35 +0000 (11:13 -0700)] 
GIRAPH-1103: Another try to fix jobs getting stuck after channel failure

Summary:
With GIRAPH-1087 we see jobs stuck after channel failure less often, but it still happens. There are several additional issues I found: requests failing to send at the first place so they never get retried, callbacks for channel failures not being triggered always.
Added a thread which will periodically check on open requests even when we are not waiting on all open requests (since in many places we don't), remove the check that request wass ent when retrying it, added some thread utils while at it.

Test Plan: Before the change, failure rate of a particular job was about 1 in 50. Had over 200 successful runs with this change.

Differential Revision: https://reviews.facebook.net/D61719

2 years agoGIRAPH-1107: Allow observers to access job counters
Maja Kabiljo [Tue, 23 Aug 2016 18:52:26 +0000 (11:52 -0700)] 
GIRAPH-1107: Allow observers to access job counters

Summary: From mapper/master/worker observer we might want to update some job counters for stats. For that we should allow observers to access job context.

Test Plan: Ran a job which accesses counters from WorkerObserver

Reviewers: sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D62391

2 years agoGIRAPH-1105: Fix number of open requests in FacebookConfiguration
Maja Kabiljo [Fri, 12 Aug 2016 21:57:53 +0000 (14:57 -0700)] 
GIRAPH-1105: Fix number of open requests in FacebookConfiguration

Test Plan: This was significantly better in some experiments, but we can investigate more in the future

Differential Revision: https://reviews.facebook.net/D62019

2 years agoGIRAPH-1104: NegativeArraySize exception in BigDataOutput
Maja Kabiljo [Wed, 10 Aug 2016 19:56:19 +0000 (12:56 -0700)] 
GIRAPH-1104: NegativeArraySize exception in BigDataOutput

Summary:
BigDataIO is not properly handling large byte[] being written to it. Chunk them up when needed to respect the max single data output size.
With D61791 job was still failing with the same exception.

Test Plan: The job which was failing because of large edges now works, added a test

Differential Revision: https://reviews.facebook.net/D61839

2 years agoFixing Giraph pom.xml to reflect new project committers
Hassan Eslami [Tue, 26 Jul 2016 18:27:24 +0000 (11:27 -0700)] 
Fixing Giraph pom.xml to reflect new project committers

Summary:
Fixed the list of project committers. Please review your information and let me know if I should change anything.

This will be the first diff that I'll be committing all by myself, more like a test to see my username is gone through Apache's internal :-)

Test Plan: N/A

Reviewers: ikabiljo, pavanka, avery.ching, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D61197

2 years agoGIRAPH-1098 Job may get stuck if zookeeper port fixed and is in use
Sergey Edunov [Wed, 20 Jul 2016 17:20:36 +0000 (10:20 -0700)] 
GIRAPH-1098 Job may get stuck if zookeeper port fixed and is in use

Test Plan: mvn clean verify -Phadoop_facebook

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D60945

2 years agoGIRAPH-1087: Retry requests after channel failure
Maja Kabiljo [Tue, 12 Jul 2016 17:27:47 +0000 (10:27 -0700)] 
GIRAPH-1087: Retry requests after channel failure

Summary: We currently don't have a callback to retry requests after channel failure, and would either wait for request timeout or not retrying request at all at places where we don't wait for open requests.

Test Plan: Hard to reproduce the issue (ran many jobs but was unable to), we'll see if the problem happens again in prod with this change.

Differential Revision: https://reviews.facebook.net/D60675

2 years agoGIRAPH-1097 Fix TestOutOfCore.testOutOfCoreLocalDiskAccessor
Sergey Edunov [Tue, 19 Jul 2016 00:30:04 +0000 (17:30 -0700)] 
GIRAPH-1097 Fix TestOutOfCore.testOutOfCoreLocalDiskAccessor

Summary:
On my laptop it failed because of an NPE in WorkerSuperstepMetrics.
I tracked it down and found that it is triggered from the branch of code that prints out metrics. We don't normally print out metrics in unit tests, so I'd expect this feature doesn't exist or not functional in hadoop_1. I'll try to disable it, to see how jenkins reacts.

Test Plan:  mvn test -pl giraph-examples -am -Dtest=TestOutOfCore -DfailIfNoTests=false -Phadoop_1

Reviewers: maja.kabiljo, dionysis.logothetis, heslami

Reviewed By: heslami

Differential Revision: https://reviews.facebook.net/D60873

2 years ago[GIRAPH-1095] Performance regression after GIRAPH-1068
Sergey Edunov [Fri, 15 Jul 2016 21:22:59 +0000 (14:22 -0700)] 
[GIRAPH-1095] Performance regression after GIRAPH-1068

Summary: Need to pass some missing parameters to zookeeper

Test Plan: run a few jobs

Reviewers: dionysis.logothetis, heslami, majakabiljo, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D60831

2 years agoGIRAPH-1092 TestCollections.testLargeBasicList fails with OOM
Sergey Edunov [Wed, 13 Jul 2016 21:38:02 +0000 (14:38 -0700)] 
GIRAPH-1092 TestCollections.testLargeBasicList fails with OOM

Summary: This test case requires too much memory to run in Jenkins. Talked to Sergey Pupyrev and we decided to disable it.

Test Plan: none

Reviewers: majakabiljo, maja.kabiljo, spupyrev

Reviewed By: spupyrev

Differential Revision: https://reviews.facebook.net/D60753

2 years ago[GIRAPH-1091] Fix SimpleRangePartitionFactoryTest
Maja Kabiljo [Wed, 13 Jul 2016 18:05:48 +0000 (11:05 -0700)] 
[GIRAPH-1091] Fix SimpleRangePartitionFactoryTest

Summary: SimpleRangePartitionFactoryTest relied on old logic for calculating number of partitions and got broken with GIRAPH-1082.

Test Plan: Ran the test

Differential Revision: https://reviews.facebook.net/D60747

2 years agoGIRAPH-1086: Use pool of byte arrays with InMemoryDataAccessor
Maja Kabiljo [Mon, 11 Jul 2016 18:07:18 +0000 (11:07 -0700)] 
GIRAPH-1086: Use pool of byte arrays with InMemoryDataAccessor

Summary: Have a pool of byte arrays with InMemoryDataAccessor, to save on byte array creation and initialization.

Test Plan: Improved performance of a job using InMemoryDataAccessor

Differential Revision: https://reviews.facebook.net/D60621

2 years ago[GIRAPH-1089] Fix a bug in out-of-core infrastructure
Hassan Eslami [Tue, 12 Jul 2016 18:33:38 +0000 (11:33 -0700)] 
[GIRAPH-1089] Fix a bug in out-of-core infrastructure

Summary: This diff fixes a bug in out-of-core infrastructure that caused user requirement (max number of partitions in memory) for fixed out-of-core strategy get violated. The cause of the problems was the un-clear definition of in-memory partitions. In this diff, we distinguish the partitions that are entirely in memory from those that are partially in memory.

Test Plan:
mvn clean verify

Reviewers: dionysis.logothetis, sergey.edunov, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D60573

2 years agoGIRAPH-1085: Add InMemoryDataAccessor
Maja Kabiljo [Wed, 6 Jul 2016 21:57:33 +0000 (14:57 -0700)] 
GIRAPH-1085: Add InMemoryDataAccessor

Summary: When we deal with graphs which have a lot of vertices with very little total data associated with them (values + edges) we start experiencing memory problems because of too many objects created, since every vertex has multiple objects associated with it. To solve this problem, we should have a serialized partition representation (current ByteArrayPartition just keeps byte[] per vertex, not per partition). We can leverage the out-of-core infrastructure and just add data accessor which won't be backed by disk but in memory buffers.

Test Plan: Successfully ran a job which was failing without this.

Differential Revision: https://reviews.facebook.net/D60435

2 years agoGIRAPH-1082: Remove limit on the number of partitions
Maja Kabiljo [Fri, 1 Jul 2016 14:39:25 +0000 (07:39 -0700)] 
GIRAPH-1082: Remove limit on the number of partitions

Summary: Currently we have a limit on how many partitions we can have because we write all partition information to Zookeeper. We can instead send this information in requests and remove the hard limit.

Test Plan: Ran pagerank for 100 iterations with 500k partitions.

Differential Revision: https://reviews.facebook.net/D60267

2 years agoGIRAPH-1083: Make sure we fail after exception in ooc-io thread happens
Maja Kabiljo [Fri, 1 Jul 2016 20:26:50 +0000 (13:26 -0700)] 
GIRAPH-1083: Make sure we fail after exception in ooc-io thread happens

Summary: Currently if some exception happens in ooc-io thread the job is left running for long time after the exception. We should make sure we fail early.

Test Plan: Ran a job with ooc on where I simulated the failure, without change job hangs for a long time, with the change it fails right after the exception happens, and logs it to command line.

Differential Revision: https://reviews.facebook.net/D60291