incubator-nemo.git
5 days agoNemo Release 0.4 (#337) master
Won Wook SONG [Thu, 22 Sep 2022 05:19:47 +0000 (14:19 +0900)] 
Nemo Release 0.4 (#337)

**Major changes:**
- Bump Apache Nemo version to 0.5-snapshot, due to the release of v0.4

**Minor changes to note:**
- None

**Tests for the changes:**
- Existing tests comply

**Other comments:**
- None

4 weeks agoBump checkstyle from 8.23 to 9.3 (#336)
dependabot[bot] [Tue, 30 Aug 2022 00:33:47 +0000 (09:33 +0900)] 
Bump checkstyle from 8.23 to 9.3 (#336)

* Bump checkstyle from 8.16 to 8.29 to 9.3

Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 8.16 to 8.29. to 9.3.
- [Release notes](https://github.com/checkstyle/checkstyle/releases)
- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-8.16...checkstyle-8.29)

Signed-off-by: dependabot[bot] <support@github.com>
* Bump checkstyle from 8.23 to 8.29 to 9.3

Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 8.23 to 8.29. to 9.3.
- [Release notes](https://github.com/checkstyle/checkstyle/releases)
- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-8.23...checkstyle-8.29)

---
updated-dependencies:
- dependency-name: com.puppycrawl.tools:checkstyle
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
* checkstyle 8.45

* updated comments

* to 9.3

* timeout

* build badge

* lets just give 50 sec

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Won Wook SONG <wsong0512@gmail.com>
4 weeks agoBump jsdom and fabric in /webui (#335)
dependabot[bot] [Mon, 29 Aug 2022 12:32:50 +0000 (21:32 +0900)] 
Bump jsdom and fabric in /webui (#335)

Bumps [jsdom](https://github.com/jsdom/jsdom) and [fabric](https://github.com/fabricjs/fabric.js). These dependencies needed to be updated together.

Updates `jsdom` from 9.12.0 to 19.0.0
- [Release notes](https://github.com/jsdom/jsdom/releases)
- [Changelog](https://github.com/jsdom/jsdom/blob/master/Changelog.md)
- [Commits](https://github.com/jsdom/jsdom/compare/9.12.0...19.0.0)

Updates `fabric` from 2.7.0 to 5.2.4
- [Release notes](https://github.com/fabricjs/fabric.js/releases)
- [Changelog](https://github.com/fabricjs/fabric.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/fabricjs/fabric.js/compare/v2.7.0...v5.2.4)

---
updated-dependencies:
- dependency-name: jsdom
  dependency-type: indirect
- dependency-name: fabric
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
4 weeks agoBump node-fetch and nuxt in /webui (#334)
dependabot[bot] [Mon, 29 Aug 2022 07:14:13 +0000 (16:14 +0900)] 
Bump node-fetch and nuxt in /webui (#334)

Bumps [node-fetch](https://github.com/node-fetch/node-fetch) and [nuxt](https://github.com/nuxt/nuxt.js). These dependencies needed to be updated together.

Updates `node-fetch` from 1.6.3 to 2.6.7
- [Release notes](https://github.com/node-fetch/node-fetch/releases)
- [Commits](https://github.com/node-fetch/node-fetch/compare/v1.6.3...v2.6.7)

Updates `nuxt` from 1.4.5 to 2.15.8
- [Release notes](https://github.com/nuxt/nuxt.js/releases)
- [Changelog](https://github.com/nuxt/nuxt.js/blob/dev/RELEASE_PLAN.md)
- [Commits](https://github.com/nuxt/nuxt.js/compare/v1.4.5...v2.15.8)

---
updated-dependencies:
- dependency-name: node-fetch
  dependency-type: indirect
- dependency-name: nuxt
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
4 weeks ago[MINOR] Upgrade travis build VM from trusty (v14) to bionic (v18) (#332)
Won Wook SONG [Mon, 29 Aug 2022 04:08:20 +0000 (13:08 +0900)] 
[MINOR] Upgrade travis build VM from trusty (v14) to bionic (v18) (#332)

**Major changes:**
- None

**Minor changes to note:**
- This upgrades the travis build environment from trusty (v14) to bionic (v18) (ubuntu)

**Tests for the changes:**
- Existing tests comply

**Other comments:**
- None

Closes #332

8 weeks ago[NEMO-299] Fix heisenbug regarding stream pipe channels (#330)
Won Wook SONG [Tue, 2 Aug 2022 06:18:06 +0000 (15:18 +0900)] 
[NEMO-299] Fix heisenbug regarding stream pipe channels (#330)

JIRA: [NEMO-299: WindowedWordCountITCase Hangs (Heisenbug)](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-299)

**Major changes:**
- Fixes the concurrency issue in the encoder/decoder that lead to the heisenbug of existing stream pipe channels

**Minor changes to note:**
- Logs are added to display messages when the errors occur

**Tests for the changes:**
- Existing tests run without the heisenbug now.

**Other comments:**
- None

Closes #330

8 weeks ago[NEMO-481] Add Stream examples (#314)
Won Wook SONG [Mon, 1 Aug 2022 08:14:59 +0000 (17:14 +0900)] 
[NEMO-481] Add Stream examples (#314)

JIRA: [NEMO-481: Add Stream examples](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-481)

**Major changes:**
- Adds stream processing example workloads
  - Average document size (global mean)
  - Document fetch success rate (mean per key)
  - Requests by CIK (id) (sum per key)
  - Top 10 bad referer docs (mean per key + top k)
  - Top 10 documents (global sum + top k)

**Minor changes to note:**
- Minor fixes and improvements on streaming code for stream workloads
- Minor revision on the metrics code to ensure metric flushing

**Tests for the changes:**
- The IT Case tests for the new examples

**Other comments:**
- None

Closes #314

8 weeks ago[MINOR] Expose Latency metrics (#324)
Won Wook SONG [Mon, 1 Aug 2022 06:24:16 +0000 (15:24 +0900)] 
[MINOR] Expose Latency metrics (#324)

**Major changes:**
- Fix for exposing latency metrics on the metrics log

**Minor changes to note:**
-

**Tests for the changes:**
- Existing tests pass.

**Other comments:**
-

Closes #324

8 weeks ago[MINOR] Bump Library Versions (#323)
Won Wook SONG [Mon, 1 Aug 2022 05:38:14 +0000 (14:38 +0900)] 
[MINOR] Bump Library Versions (#323)

**Major changes:**
-  None

**Minor changes to note:**
-  Bump library versions to keep everything up to date

**Tests for the changes:**
- Existing tests comply.

**Other comments:**
- None

Closes #323

8 weeks ago[MINOR] Fix Travis script to run with LTS node version (#331)
Won Wook SONG [Mon, 1 Aug 2022 01:58:02 +0000 (10:58 +0900)] 
[MINOR] Fix Travis script to run with LTS node version (#331)

**Major changes:**
- None

**Minor changes to note:**
- It adds installation of node dependencies with LTS version

**Tests for the changes:**
- Existing tests comply

**Other comments:**
- Node on versions 18 newly requires manual installation of other libraries on Ubuntu servers

Closes #331

11 months ago[NEMO-483] Record Metrics associated with stream processing (#317)
Lemarais [Mon, 18 Oct 2021 12:15:13 +0000 (21:15 +0900)] 
[NEMO-483] Record Metrics associated with stream processing (#317)

JIRA: [NEMO-483: Record Metrics associated with stream processing](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-483)

**Major changes:**
-

Two types of metric were added. It can degrade performance, So It is recommended to use only for debugging purpose.

Task periodically record how many data has received
- Record the following metric:
1. the number of processed tuples
2. the size of read serialized bytes
- The recording period can be adjusted through "stream_metric_period" argument. the unit is milli second.
- If stream_metric_period is not set, it is disabled.

For each task, record traversal time from source vertex.
Source vertex sends latencymark from source vertex
- The latencymark has the following data:
1. Timestamp when it was create.
2. Task id where it created.
3. Task id where it delivered from. task id of upstream task.
- When a latencymark is reached  at any task, It is recorded with the timestamp when it reached at task.
- The period for creating latencymarks can be adjusted through "latencymark_period" argument. the unit is milli second.
- If latencymark_period is not set, it is disabled.

**Minor changes to note:**
-
**Tests for the changes:**
-
- Tested on ubuntu machine

**Other comments:**
-
- There is a gap between when the number of processed tuples increased and when the size of read serialized bytes is increased. Because the size of read serialized bytes is increased when the data are received, the number of processed tuples is increased just before it is processed, and data fetchers are working on the multithread environment.
- When increasing the size of read serialized bytes, It is not possible to distinguish whether it is the data to be processed or the watermark, So When the number of processed tuples is 0, the size of read serialized bytes can be not 0.
- If a task read data from local block, because upstream task is in the same node, the size of read serialized data does not increase because the data is not serialized. Instead, isReadNotSerializedData field of StreamMetric indicates whether the task read not serialized data or not.

Closes #317

13 months ago[MINOR] Network Profiling Scripts (#315)
Won Wook SONG [Mon, 16 Aug 2021 10:28:13 +0000 (19:28 +0900)] 
[MINOR] Network Profiling Scripts (#315)

**Major changes:**
- Adds network profiling scripts in the bin directory

**Minor changes to note:**
- None

**Tests for the changes:**
- Example result files show profiling results

**Other comments:**
- None

Closes #315

13 months ago[NEMO-482] Keep library versions updated (#316)
Won Wook SONG [Mon, 16 Aug 2021 09:10:31 +0000 (18:10 +0900)] 
[NEMO-482] Keep library versions updated (#316)

JIRA: [NEMO-482: Keep library versions updated](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-482)

**Major changes:**
- Updates library versions, and performs migrations for the version updates

**Minor changes to note:**
- NEXMark pom was updated to keep up with the changes

**Tests for the changes:**
- Existing tests comply with the changes
- NEXMark now also has an ITCase test

**Other comments:**
- None

Closes #316

14 months ago[MINOR] Installation Script (#308)
Won Wook SONG [Thu, 1 Jul 2021 07:10:54 +0000 (16:10 +0900)] 
[MINOR] Installation Script (#308)

**Major changes:**
- Adds an installation script to the Nemo repository

**Minor changes to note:**
- None

**Tests for the changes:**
- Tested on my Mac and ubuntu machine

**Other comments:**
- None

17 months ago[MINOR] Avoid yarn conflict + Save Metrics after waiting to get full data (#307)
Lemarais [Fri, 2 Apr 2021 10:31:50 +0000 (19:31 +0900)] 
[MINOR] Avoid yarn conflict + Save Metrics after waiting to get full data  (#307)

**Major changes:**
-

**Minor changes to note:**
- Change yarn command in run_beam.sh file to avoid conflict with javascript package manager
- Split metric saving part from flushMetric function and call metric saving part after waiting metric flush

**Tests for the changes:**
-

**Other comments:**
-

Closes #307

21 months ago[NEMO] Release v0.3 (#305)
Won Wook SONG [Fri, 11 Dec 2020 14:13:50 +0000 (23:13 +0900)] 
[NEMO] Release v0.3 (#305)

**Major changes:**
- None

**Minor changes to note:**
- Added Javadocs for the release process
- Bump up the version to 0.4-SNAPSHOT

**Tests for the changes:**
- Existing tests work.

**Other comments:**
- None

Closes #305

22 months ago[NEMO-436] Dynamic re-configuration based on the Sampled Metric Data (#293)
Hwarim [Tue, 24 Nov 2020 16:39:02 +0000 (01:39 +0900)] 
[NEMO-436] Dynamic re-configuration based on the Sampled Metric Data (#293)

JIRA: [NEMO-436: Dynamic re-configuration based on the Sampled Metric Data](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-436)

**Major changes:**
- New Policy and Runtime Pass for DTS
- New Prophet interface to gather data needed in RunTime pass
   - ParallelismProphet launches simulation scheduler and gathers stage duration per task size
- Added RunTimePassType in Control Message and changed information sending logic to include this.

**Minor changes to note:**
- updated PlanRewriter for DTS

**Tests for the changes:**
- New test cases for DTS (WordCount, ALS)

**Other comments:**
- This PR is based on #292 , therefore, it needs to be merged after #292 .
- This is a work-in-process PR. Some issues will have to be solved before being merged.

Closes #293

22 months ago[NEMO-394] Exchange data via shared memory (#304)
jaehwan0214 [Thu, 19 Nov 2020 11:49:42 +0000 (20:49 +0900)] 
[NEMO-394] Exchange data via shared memory (#304)

JIRA: [NEMO-394: Exchange data via shared memory](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-394)

**Major changes:**
- Implemented the LocalInputContext and the LocalOutputContext to enable a pipe output writer and an input reader to exchange data via shared memory when both a parent task and a child task are in the same executor. In such a case, the data transfer doesn't involve data serialization/deserialization.

**Minor changes to note:**
- Renamed the "bytetransfer" package to "transfer," which now also includes LocalTransfer contexts.

**Tests for the changes:**
- Added the LocalTransferContext test.
- Windowed Word Count IT case
- Measured the latency of processing 160MB data before and after using shared memory. Below are the results.
      - With a single executor, the latency reduced by 87 sec (≈70%) after using shared memory.
      - With two executors, the latency reduced by 30 sec (≈20%) after using shared memory.

Closes #304

23 months ago[NEMO-460] Setting coders in CombinePerKey transformation (#303) v0.1 v0.2-rc5
jaehwan0214 [Sun, 18 Oct 2020 05:05:34 +0000 (14:05 +0900)] 
[NEMO-460] Setting coders in CombinePerKey transformation (#303)

JIRA: [NEMO-460: Setting coders in CombinePerKey transformation](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-460)

**Major changes:**
- Added the additional parameter "inputCoder" for GBKTransform constructor.
- Fixed the input coder and the output coder for the partial combine transform and the final combine transform.

**Minor changes to note:**
- Fixed the main output TupleTags for the partial combine transform and the final combine transform.

**Tests for the changes:**
- Current tests suffice.

**Other comments:**
- This needs to be merged after merging #302

Closes #303

23 months ago[NEMO-456] Remove unnecessary shuffling processes in Combine transformation (#302)
jaehwan0214 [Sat, 17 Oct 2020 23:33:44 +0000 (08:33 +0900)] 
[NEMO-456] Remove unnecessary shuffling processes in Combine transformation (#302)

JIRA: [NEMO-456: Remove unnecessary shuffling processes in Combine transformation](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-456)

**Major changes:**
- Removed the unnecessary shuffling process before the partial combine transform

**Tests for the changes:**
- Modified GBKTransformTest.

Closes #302

23 months ago[NEMO-459] Enable Automatic Analysis for sonar cloud (#301)
Won Wook SONG [Fri, 16 Oct 2020 23:16:41 +0000 (08:16 +0900)] 
[NEMO-459] Enable Automatic Analysis for sonar cloud (#301)

JIRA: [NEMO-459: Enable Automatic Analysis for sonar cloud](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-459)

**Major changes:**
- Change the travis configuration file accordingly, to use automatic analysis instead of the CI-enabled analysis.

**Minor changes to note:**
- None

**Tests for the changes:**
- Existing tests confirm for the changes

**Other comments:**
- None

Closes #301

23 months ago[NEMO-457] Improve test coverage score on sonar cloud test (#300)
jaehwan0214 [Fri, 16 Oct 2020 01:50:58 +0000 (10:50 +0900)] 
[NEMO-457] Improve test coverage score on sonar cloud test (#300)

JIRA: [NEMO-457: Improve test coverage score on sonar cloud test](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-457)

**Major changes:**
- Added unit tests for partial and final CombineFn.

**Minor changes to note:**
-

**Tests for the changes:**
-

**Other comments:**
-

Closes #300

23 months ago[NEMO-392] Support combine in streaming (#299)
jaehwan0214 [Wed, 14 Oct 2020 07:33:37 +0000 (16:33 +0900)] 
[NEMO-392] Support combine in streaming (#299)

JIRA: [NEMO-392: Support combine in streaming] (https://issues.apache.org/jira/projects/NEMO/issues/NEMO-392)

**Major changes:**
- Added GBKTransform class to support stream processing for GroupByKey and Combine.Perkey operation.
     - Each time an element arrives, GBKTransform invokes runner to process a single element and stores its state,       instead of waiting to process it until the time to emit it to downstream.
- Removed GroupByKeyAndWindowDoFnTransform since GBKTransform can support both Combine.Perkey
   and GroupByKeyAndWindow operation.

**Minor changes to note:**
- N/A

**Tests for the changes:**
- Added GBKStreamingTransformTest.

**Other comments:**
- Most of the work is done by TaeGun Um

Closes #299

2 years ago[NEMO-434] Logical DAG modification for Dynamic sampling of task metrics during the...
polarcoke2 [Fri, 3 Jul 2020 07:59:43 +0000 (16:59 +0900)] 
[NEMO-434] Logical DAG modification for Dynamic sampling of task metrics during the execution of a stage (#292)

JIRA: [NEMO-434: Logical DAG modification for Dynamic sampling of task metrics during the execution of a stage](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-434)

**Major changes:**
- Added new mechanism for launching runtime pass
   - For runtime passes which need to gather information by itself, use MessageGeneratorVertex and MessageAggregatorVertex
   - For runtime passes which need Metric information, use SignalVertex
- Added SignalVertex and SignalTransform to launch runtime pass without collecting any runtime information
- New Compile time pass for DTS which changes DAG
- New TaskSizeSplitterVertex to distinguish and manage sampled data and the rest in vertex level(extended from LoopVertex)
- Added inserting / deleting TaskSizeSplitterVertex and SignalVertex method in IRDAG

**Minor changes to note:**
- Changed the name of TriggerVertex to MessageGeneratorVertex. Since the combination of TriggerVertex and MAV triggers runtime pass, not trigger vertex alone, the name 'Trigger' can be misleading.
- erased duplicate assignments in EmptyComponents class
- new methods in LoopVertex

**Tests for the changes:**
- IRDAGTest and SkewCompositePassTest has been modified by the class name refactoring
- Added new test methods in IRDAGTest regarding TaskSizeSplitterVertex and SignalVertex

**Other comments:**

Closes #292

2 years ago[MINOR] Fix Travis CI + Bring back support for Java 8 (#296)
Won Wook SONG [Fri, 26 Jun 2020 08:16:53 +0000 (17:16 +0900)] 
[MINOR] Fix Travis CI + Bring back support for Java 8 (#296)

**Major changes:**
- Fixes Travis CI script that have been failing due to gpg error
- Bring back support for Java 8, as Hadoop and REEF still needs support for Java 8 (reverts #291)

**Minor changes to note:**
- None

**Tests for the changes:**
- N/A

**Other comments:**
- None

Closes #296

2 years ago[Nemo-440] Update README.md about java 11 support (#295)
Wooyeon Lee [Sun, 19 Apr 2020 06:28:48 +0000 (15:28 +0900)] 
[Nemo-440] Update README.md about java 11 support (#295)

JIRA: [NEMO-440: Migrate to Java11 and use Java 11 Features](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-440)

**Major changes:**
- Update readme

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- Java 11 support has been done in PR #291
- Nemo build fails with Java 8 due to the pom configuration that does not work with Java 8

2 years ago[NEMO-447] Fix beam pom.xml to resolve build failure (#294)
Lee Dong Joo [Fri, 17 Apr 2020 07:15:07 +0000 (16:15 +0900)] 
[NEMO-447] Fix beam pom.xml to resolve build failure (#294)

JIRA: [NEMO-447: Fix beam pom.xml to resolve build failure](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-447)

**Major Changes:**

**Minor changes to note:**
* changed examples/beam/pom.xml to resolve build failure

**Tests for the changes:**

**Other comments:**

Closes #294

2 years ago[NEMO-324] Distinguish Beam's run and waitUntilFinish methods (#187)
Won Wook SONG [Tue, 24 Mar 2020 23:18:24 +0000 (08:18 +0900)] 
[NEMO-324] Distinguish Beam's run and waitUntilFinish methods (#187)

JIRA: [NEMO-324: Distinguish Beam's run and waitUntilFinish methods](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-324)

**Major changes:**
- Gets rid of the UnsupportedOperation, makes `waitUntilFinish` method to a meaningful one.
- Runs the application asynchronously on the `run` method.

**Minor changes to note:**
- Add the examples output directory to the gitignore.

**Tests for the changes:**
- I've added a same word count program that has a timeout of 1 second, and a test to confirm that it aborts after 1 second.
- Existing tests cover other changes.

**Other comments:**
- None

Closes #187

2 years ago[NEMO-440] Migrate to Java11 and use Java 11 Features (#291)
Won Wook SONG [Mon, 23 Mar 2020 09:34:42 +0000 (18:34 +0900)] 
[NEMO-440] Migrate to Java11 and use Java 11 Features (#291)

JIRA: [NEMO-440: Migrate to Java11 and use Java 11 Features](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-440)

**Major changes:**
- Upgrade POM and modify travis to check Java 11

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #291

2 years ago[NEMO-438] Create a Simulator for Simulating an Execution of an Execution Plan (...
Won Wook SONG [Mon, 9 Mar 2020 06:14:13 +0000 (15:14 +0900)] 
[NEMO-438] Create a Simulator for Simulating an Execution of an Execution Plan (#288)

JIRA: [NEMO-438: Create a Simulator for Simulating an Execution of an Execution Plan](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-438)

**Major changes:**
- Implements a SimulationScheduler that schedules a physical plan
- Implements a SimulatedExecutor that acts like an executor
- Refactor components used by both `BatchScheduler` and `SimulationScheduler` to `SchedulerUtils`

**Minor changes to note:**
- Refactors executor's ResourceSpecification to be able to be accessed in the driver as well
- Enables IRDAG to get the input size of the workload
- Added Job Duration to the Job Metric.

**Tests for the changes:**
- SimulationSchedulerTest.java tests for the implementations

**Other comments:**
- None

Closes #288

2 years ago[NEMO-433] Improvement of Task Metrics and Collecting Them For Sampling (#289)
polarcoke2 [Sat, 7 Mar 2020 09:17:15 +0000 (18:17 +0900)] 
[NEMO-433] Improvement of Task Metrics and Collecting Them For Sampling (#289)

JIRA: [NEMO-433: Improvement of Task Metrics and Collecting Them For Sampling](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-433)

**Major changes:**
- Added some new metrics (task, job) and usages of them
- Added new execution property for DTS (vertex, edge)

**Minor changes to note:**
- changed checkstyle grammer for windows
- BlockInputReader now contains metricMessageSender for sending some task metrics, and it has whole task id instead of just task index.

**Tests for the changes:**
- Tests related to BlockInputReader has changed: TaskExecutorTest, DataTransferTest

**Other comments:**
- N/A

Closes #289

2 years ago[Nemo-439] Upgrade current working version to 0.3-SNAPSHOT (#290)
Wooyeon Lee [Fri, 6 Mar 2020 18:25:40 +0000 (03:25 +0900)] 
[Nemo-439] Upgrade current working version to 0.3-SNAPSHOT (#290)

JIRA: [NEMO-439: Upgrade current working version to 0.3-SNAPSHOT](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-439)

**Major changes:**
- Upgrades the current working version to 0.3-SNAPSHOT as v0.2 has been released.

**Minor changes to note:**
- N/A

**Tests for the changes:**
- N/A

**Other comments:**
- v0.2 has been released.

2 years ago[NEMO-320] Make WebUI scale to big workloads (#183)
Jangho Seo [Sat, 15 Feb 2020 16:27:41 +0000 (01:27 +0900)] 
[NEMO-320] Make WebUI scale to big workloads (#183)

JIRA: [NEMO-320: Make WebUI scale to big workloads](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-320)

**Major changes:**
- WebUI: TaskStatistics: Made WebUI not to recompute rows on every task addition/update
- WebUI: TaskStatistics: Added pagination
- WebUI: JobView: Not it shows stage status
- WebUI: JobsView: Add progress bar to each job

Co-authored-by: Won Wook SONG <wonook@apache.org>
2 years agoRevert "[NEMO-421] Release v0.2 (#286)" (#287)
WooYeon Lee [Sat, 15 Feb 2020 16:05:34 +0000 (01:05 +0900)] 
Revert "[NEMO-421] Release v0.2 (#286)" (#287)

This reverts commit 0cab1dc36f5193a223b0ea448d7a41f9732ef06e.

JIRA: [NEMO-421: Release v0.2](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-421)

**Major changes:**
- Revert the 0.2 release commit, which has been merged by mistake.

**Minor changes to note:**
-

**Tests for the changes:**
-

**Other comments:**
- I'll make a new PR for the release

2 years ago[NEMO-421] Release v0.2 (#286)
Sanha Lee [Thu, 13 Feb 2020 04:40:38 +0000 (13:40 +0900)] 
[NEMO-421] Release v0.2 (#286)

JIRA: [NEMO-421: Release v0.2](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-421)

**Major changes:**
- Update the versions of components according to the release v0.2.

**Minor changes to note:**
- N/A

**Tests for the changes:**
- N/A

**Other comments:**
- I'll open a release vote current version.  If it passed, I'll update the versions to `0.3-SNAPSHOT` and merge this PR. Please do not merge any other PR before the vote. Thanks :)

Closes #421

2 years ago[NEMO-429] SWPP TEAM20 Code Smell Fix (#283)
Jeongyoon Eo [Tue, 11 Feb 2020 05:50:17 +0000 (14:50 +0900)] 
[NEMO-429] SWPP TEAM20 Code Smell Fix (#283)

JIRA: NEMO-429: SWPP TEAM20 Code Smell Fix

Major changes:
- Fixed code smells (SWPP Code Smell session)

Co-authored-by: jihoonrf <kzc123@snu.ac.kr>
Co-authored-by: Spiraline <44594966+Spiraline@users.noreply.github.com>
2 years ago[NEMO-429] SWPP TEAM 14 code smell fix (#284)
Jeongyoon Eo [Mon, 10 Feb 2020 07:41:08 +0000 (16:41 +0900)] 
[NEMO-429] SWPP TEAM 14 code smell fix (#284)

JIRA: [NEMO-429: SWPP TEAM 14 code smell fix](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells (SWPP Code Smell session)

Co-authored-by: Peace-Song <35649318+Peace-Song@users.noreply.github.com>
Co-authored-by: Hank Choi <48513130+zlzlqlzl@users.noreply.github.com>
Co-authored-by: BlackMustard <38564738+mustardlove@users.noreply.github.com>
2 years ago[NEMO-304] Fail-fast for mis-configuration in user application (#167)
Yunseong Lee [Thu, 6 Feb 2020 12:19:48 +0000 (21:19 +0900)] 
[NEMO-304] Fail-fast for mis-configuration in user application (#167)

JIRA: [NEMO-304: Fail-fast for mis-configuration in user application](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-304)

**Major changes:**
- This PR makes JobLauncher check the validity of user main method early, so that we can avoid initializing the whole system if there is a mis-configuration.

**Minor changes to note:**
-

**Tests for the changes:**
- Added `testNotExistingUserMain` in SparkScala.java to test whether the `InvalidUserMainException` is thrown if the argument of `addUserMain()` is a non-existing class.

**Other comments:**
-

Closes #167

Co-authored-by: Taegeon Um <taegeonum@gmail.com>
Co-authored-by: Won Wook SONG <wonook@apache.org>
2 years ago[NEMO-429] SWPP TEAM17 Code Smell Fix (#268)
ske-kr [Thu, 6 Feb 2020 04:35:32 +0000 (13:35 +0900)] 
[NEMO-429] SWPP TEAM17 Code Smell Fix (#268)

JIRA: [NEMO-429: SWPP-TEAM17](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells (SWPP Code Smell session)

**Minor changes to note:**
- remove declaration of thrown exceptions
- serializable issues
-  assertion errors

2 years ago[NEMO-429] SWPP TEAM2 Code Smell Fix (#277)
xixili3 [Thu, 6 Feb 2020 03:15:30 +0000 (12:15 +0900)] 
[NEMO-429] SWPP TEAM2 Code Smell Fix (#277)

JIRA: [NEMO-429: SWPP TEAM2 Code Smell Fix](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- exception elimination

**Minor changes to note:**
- replace lambda with method reference
- swap arguments

2 years ago[NEMO-437] Support Java version 11 (#281)
Won Wook SONG [Wed, 5 Feb 2020 00:37:13 +0000 (09:37 +0900)] 
[NEMO-437] Support Java version 11 (#281)

JIRA: [NEMO-437: Support Java version 11](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-437)

**Major changes:**
- Bump java libraries to recent versions
  - `beam` 2.11.0 --> 2.18.0
  - `grpc` 1.7.0 --> 1.26.0
  - `mockito` 2.13.0 --> 3.2.4
  - `powermock` 2.0.0-beta --> 2.0.4
  - `spark` 2.2.0 --> 2.4.4 (with `scala` 2.11.0 --> 2.12.8)
  - as well as version upgrades for `maven-surefire`, `maven-shade`, `maven-deploy`, `maven-javadoc` plugins and `guava`, `jackson`, `sqlite-jdbc`, `PostgreSQL`, `slf4j`, and `junit`.

**Minor changes to note:**
- Syntax changes with the java library updates

**Tests for the changes:**
- I've included new JDK requirements to the travis script
- Existing tests pass

**Other comments:**
- I've referred to the following PRs for upgrading Beam: [link1](https://github.com/apache/beam/pull/7635/files), [link2](https://github.com/apache/beam/pull/9275/files)

Closes #281

2 years ago[NEMO-411] Bug in ScheduleGroupPass, OutputTag, DuplicateEdgeGroup (#232)
Won Wook SONG [Tue, 28 Jan 2020 01:49:24 +0000 (10:49 +0900)] 
[NEMO-411] Bug in ScheduleGroupPass, OutputTag, DuplicateEdgeGroup (#232)

JIRA: [NEMO-411: Bug in ScheduleGroupPass, OutputTag, DuplicateEdgeGroup](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-411)

**Major changes:**
- Fixes the bugs described in [NEMO-411](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-411)

> When trying to run ALS with TransientResourcePass, I've faced a bug regarding 1. OutputTag, which groups edges based on the output tag, but currently groups all of the edges without an output tag into a single group, which is undesirable, 2. DuplicateEdgeGroup, which does not consider the first edge that points to the first iteration of the loop, but obviously is a part of the duplicate edge group, 3. ScheduleGroupPass, which does not consider vertices with multiple outgoing edges pointing outside when looking for a cycle in the graph.

**Minor changes to note:**
- None

**Tests for the changes:**
- Added a integration test for ALS, the case where it was failing initially.

**Other comments:**
- None

Closes #232

2 years ago[NEMO-429] SWPP TEAM4 Code Smell Fix (#270) 272/head
1207koo [Tue, 21 Jan 2020 06:48:38 +0000 (15:48 +0900)] 
[NEMO-429] SWPP TEAM4 Code Smell Fix (#270)

JIRA: [NEMO-429: SWPP TEAM4 Code Smell Fix](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells

2 years ago[NEMO-429] SWPP TEAM 21 code smell fix (#258)
sang_G [Tue, 21 Jan 2020 04:48:11 +0000 (13:48 +0900)] 
[NEMO-429] SWPP TEAM 21 code smell fix (#258)

JIRA: [NEMO-429: SWPP TEAM 21 code smell fix](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fix sonarcloud code smells

2 years ago[NEMO-429] SWPP TEAM8 Code Smell Fix (#262)
dkwanm1 [Tue, 21 Jan 2020 03:17:18 +0000 (12:17 +0900)] 
[NEMO-429] SWPP TEAM8 Code Smell Fix (#262)

* Fix sonar cloud code smell

Co-authored-by: Donghyun Lee <dominic2009@snu.ac.kr>
Co-authored-by: ty8900 <48665246+ty8900@users.noreply.github.com>
Co-authored-by: Jeong Rok Seo (Rocky) <m.seo1999@gmail.com>
Co-authored-by: Gyewon Lee <strayyyyyy@gmail.com>
2 years ago[NEMO-429] SWPP TEAM7 Code Smell Fix (#251)
Jaehun Jung [Tue, 21 Jan 2020 01:45:53 +0000 (10:45 +0900)] 
[NEMO-429] SWPP TEAM7 Code Smell Fix (#251)

* excluded needless exceptions

* lambda fixed

* Fix Transient_Serializable smell

* Revert no sonar

Co-authored-by: Celine <40828573+lookdeceline@users.noreply.github.com>
Co-authored-by: Peter Hyun <22678965+peterhyun@users.noreply.github.com>
Co-authored-by: jinsun1117 <38969914+jinsun1117@users.noreply.github.com>
Co-authored-by: Gyewon Lee <strayyyyyy@gmail.com>
2 years ago[NEMO-429] SWPP TEAM18 Code Smell Fix (#271)
JungYunsoo [Sun, 19 Jan 2020 05:09:26 +0000 (14:09 +0900)] 
[NEMO-429] SWPP TEAM18 Code Smell Fix (#271)

* change private to transient

* fix lambda

Co-authored-by: lim8540 <51064261+lim8540@users.noreply.github.com>
Co-authored-by: gywls00 <49047636+gywls00@users.noreply.github.com>
Co-authored-by: Namkoong Hugh <47518167+mmsori@users.noreply.github.com>
2 years ago[NEMO-429] SWPP TEAM16 Code Smell Fix (#278)
JooHyung Kim [Fri, 17 Jan 2020 05:00:27 +0000 (14:00 +0900)] 
[NEMO-429] SWPP TEAM16 Code Smell Fix (#278)

JIRA: [NEMO-429: Team16](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells (SWPP Code Smell session)

Co-authored-by: mss8087 <46183183+mss8087@users.noreply.github.com>
Co-authored-by: Daewoong Kim <43679861+dkim94@users.noreply.github.com>
Co-authored-by: Jangho Seo <jangho@jangho.io>
Co-authored-by: WooYeon Lee <wylee@apache.org>
2 years ago[NEMO-429] SWPP TEAM 9 code smell fix (#259)
Yoo-Youngjae [Thu, 16 Jan 2020 13:35:19 +0000 (22:35 +0900)] 
[NEMO-429] SWPP TEAM 9 code smell fix (#259)

* switch statement -> if statement

* transient added

Co-authored-by: manhair <43872798+manhair@users.noreply.github.com>
Co-authored-by: WooYeon Lee <wylee@apache.org>
2 years ago[Nemo-429] SWPP TEAM15 Code Smell Fix (#273)
thoum [Thu, 16 Jan 2020 12:40:48 +0000 (21:40 +0900)] 
[Nemo-429] SWPP TEAM15 Code Smell Fix (#273)

* replace lambda with method reference

* Remove unnecessary exceptions

Co-authored-by: Minji1234 <34624667+Minji1234@users.noreply.github.com>
Co-authored-by: rachelg98 <37978535+rachelg98@users.noreply.github.com>
Co-authored-by: Jangho Seo <jangho@jangho.io>
2 years ago[NEMO-420] OffHeapMemory configuration only supports a single type of executor (...
polarcoke2 [Thu, 16 Jan 2020 08:02:36 +0000 (17:02 +0900)] 
[NEMO-420] OffHeapMemory configuration only supports a single type of executor (#279)

JIRA: [NEMO-420: OffHeapMemory configuration only supports a single type of executor
](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-420)

**Major changes:**
- enabled OffHeapMemory configuration for multiple types of executors
- refactored initialization of MaxOffheapRatio and ExecutorPoisonSec (bind with default value when not existing)
- erased MaxOffheapMb in job configuration and added ExecutorMemoryMb instead to compute size of max off-heap manually if needed

**Minor changes to note:**
- modified typos (poison from posion)
- when changing test.java files which contain initialization of MemoryPoolAssigner, computed ExecutorMemoryMb using default MaxOffheapRatio
- changed type of MaxOffheapRatio and ExecutorPoisonSec from double, int to Optional Double and Optioinal Integer, respectively

**Tests for the changes:**
- N/A

**Other comments:**
- N/A

Closes #279

2 years ago[Nemo-429] SWPP TEAM10 Code Smell Fix (#275)
ByungHeeCha [Mon, 13 Jan 2020 01:39:03 +0000 (10:39 +0900)] 
[Nemo-429] SWPP TEAM10 Code Smell Fix (#275)

* [Team 10 jyeok] changed assertion parameter in correct order

* add transient option to var [jaehonam]

* removed not needed exceptions
ByungHeeCha

* removed unused import

* Update LoopOptimizations.java

Co-authored-by: jyeokchoi <46083459+jyeok@users.noreply.github.com>
Co-authored-by: JaehoNam <32262002+JaehoNam@users.noreply.github.com>
Co-authored-by: th-snu <54742258+th-snu@users.noreply.github.com>
Co-authored-by: WooYeon Lee <wylee.xyzi@gmail.com>
2 years ago[NEMO-429] SWPP TEAM11 Code Smell Fix (#276)
Donghyun Kim [Tue, 7 Jan 2020 08:15:54 +0000 (17:15 +0900)] 
[NEMO-429] SWPP TEAM11 Code Smell Fix (#276)

JIRA: [NEMO-429: TEAM11](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells (SWPP Code Smell session)

Co-authored-by: wonook <wonook@apache.org>
2 years ago[NEMO-429] SWPP TEAM12 Code Smell Fix (#274)
Taeyoung Jang [Tue, 7 Jan 2020 06:37:55 +0000 (15:37 +0900)] 
[NEMO-429] SWPP TEAM12 Code Smell Fix (#274)

JIRA: [NEMO-429: TEAM12](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

Major changes:
- Fixed code smells (SWPP Code Smell session)

2 years ago[NEMO-429] SWPP TEAM 5 code smell fix (#261)
Haesoo Kim [Wed, 1 Jan 2020 13:54:09 +0000 (22:54 +0900)] 
[NEMO-429] SWPP TEAM 5 code smell fix (#261)

JIRA: [NEMO-429: SWPP TEAM 5 code smell fix](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixing code smells

Co-authored-by: Kwak taewon <tykwak94@gmail.com>
Co-authored-by: ttoru96 <44546247+ttoru96@users.noreply.github.com>
Co-authored-by: Jangho Seo <jangho@jangho.io>
2 years ago[NEMO-429] SWPP TEAM6 Code Smell Fix (#260)
deploy-soon [Tue, 24 Dec 2019 07:05:18 +0000 (16:05 +0900)] 
[NEMO-429] SWPP TEAM6 Code Smell Fix (#260)

JIRA: [NEMO-429: SWPP-TEAM6](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells (SWPP Code Smell session)

**Minor changes to note:**
- Assertion errors
- Change lambda expressions
- Serializable issues

Co-authored-by: dreamsh19 <47855638+dreamsh19@users.noreply.github.com>
Co-authored-by: Suh Jangwon <37146511+RXWE@users.noreply.github.com>
Co-authored-by: jangdonghae <48465250+jangdonghae@users.noreply.github.com>
Co-authored-by: Jangho Seo <jangho@jangho.io>
2 years agoBump netty-all from 4.1.16.Final to 4.1.42.Final (#248)
dependabot[bot] [Tue, 17 Dec 2019 06:43:30 +0000 (15:43 +0900)] 
Bump netty-all from 4.1.16.Final to 4.1.42.Final (#248)

Bumps [netty-all](https://github.com/netty/netty) from 4.1.16.Final to 4.1.42.Final.
<details>
<summary>Commits</summary>

- [`bd907c3`](https://github.com/netty/netty/commit/bd907c3b3acf3d746ad1143270224aa0f784cd1b) [maven-release-plugin] prepare release netty-4.1.42.Final
- [`2791f0f`](https://github.com/netty/netty/commit/2791f0fefac82663ca09dc8aa5dda024152c92f8) Avoid use of global AtomicLong for ScheduledFutureTask ids ([#9599](https://github-redirect.dependabot.com/netty/netty/issues/9599))
- [`86ff76a`](https://github.com/netty/netty/commit/86ff76a4f77662fe1bf61deeec5f3e39e16d4020) Fix incorrect comment  ([#9598](https://github-redirect.dependabot.com/netty/netty/issues/9598))
- [`5e69a13`](https://github.com/netty/netty/commit/5e69a13c21bcf89bf26dbf1421c5ec8188561a17) Cleanup JNI code to always correctly free memory when loading fails and also ...
- [`eb3c4bd`](https://github.com/netty/netty/commit/eb3c4bd926e697da3ea16079a5f4211adb375ea5) ChunkedNioFile can use absolute FileChannel::read to read chunks ([#9592](https://github-redirect.dependabot.com/netty/netty/issues/9592))
- [`76592db`](https://github.com/netty/netty/commit/76592db0bde9d4908191c2d7e54af3466ffd4673) Close eventfd shutdown/wakeup race by closely tracking epoll edges ([#9586](https://github-redirect.dependabot.com/netty/netty/issues/9586))
- [`0a2d85f`](https://github.com/netty/netty/commit/0a2d85f1d32749eb278b27502ba7de20de958485) Fix GraalVM native image build error ([#9593](https://github-redirect.dependabot.com/netty/netty/issues/9593))
- [`dc4de7f`](https://github.com/netty/netty/commit/dc4de7fbb4e17c606c92c542d5015042ada48e38) We need to use NewGloblRef when caching jclass instances ([#9595](https://github-redirect.dependabot.com/netty/netty/issues/9595))
- [`4499384`](https://github.com/netty/netty/commit/4499384135cdaca5ccdda85b0c9610d4ecd6c3cc) Update to netty-tcnative 2.0.26.Final ([#9589](https://github-redirect.dependabot.com/netty/netty/issues/9589))
- [`8648171`](https://github.com/netty/netty/commit/8648171abc27497d87a8b47e509e8adabd622f5b) Fix *SslEngineTest to not throw ClassCastException and pass in all cases ([#9588](https://github-redirect.dependabot.com/netty/netty/issues/9588))
- Additional commits viewable in [compare view](https://github.com/netty/netty/compare/netty-4.1.16.Final...netty-4.1.42.Final)
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=io.netty:netty-all&package-manager=maven&previous-version=4.1.16.Final&new-version=4.1.42.Final)](https://help.github.com/articles/configuring-automated-security-fixes)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-nemo/network/alerts).

</details>

2 years ago[NEMO-429] SWPP TEAM19 Code Smell Fix (#269)
Seongmin Park [Tue, 17 Dec 2019 05:48:08 +0000 (14:48 +0900)] 
[NEMO-429] SWPP TEAM19 Code Smell Fix (#269)

JIRA: [NEMO-429: TEAM19](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells (SWPP Code Smell session)

**Minor changes to note:**
- As TA suggested, we add ```TODO #430: Pair elements should be serializable``` at Pair.java

2 years ago[NEMO-429] SWPP TEAM1 Code Smell Fix (#267)
Haeseong Je [Sun, 15 Dec 2019 22:32:45 +0000 (07:32 +0900)] 
[NEMO-429] SWPP TEAM1 Code Smell Fix (#267)

JIRA: [NEMO-429: Dec 2nd, 2019 Code session](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
Fixed code smells

2 years ago[NEMO-429] SWPP TEAM3 Code Smell Fix (#265)
Davin Byeon [Fri, 6 Dec 2019 04:45:23 +0000 (13:45 +0900)] 
[NEMO-429] SWPP TEAM3 Code Smell Fix (#265)

JIRA: [NEMO-429: SWPP TEAM 3 Code Smell Fix](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-429)

**Major changes:**
- Fixed code smells(SWPP Code Smell Session)

2 years ago[NEMO-422] Sonarcloud issue (#246)
Won Wook SONG [Tue, 5 Nov 2019 07:37:34 +0000 (16:37 +0900)] 
[NEMO-422] Sonarcloud issue (#246)

JIRA: [NEMO-422: Sonarcloud issue](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-422)

**Major changes:**
- None

**Minor changes to note:**
- Gets rid of the hard-coded fake password, and handles it with Base64 decoding method.

**Tests for the changes:**
- Existing tests confirm the changes

**Other comments:**
- None

Closes #246

2 years agoFix SonarCloud bugs related to Optional (#247)
Sanha Lee [Mon, 4 Nov 2019 06:45:38 +0000 (15:45 +0900)] 
Fix SonarCloud bugs related to Optional (#247)

JIRA: [NEMO-422: SonarCloud issues](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-422)

**Major changes:**
- Fix some SonarCloud bugs related to `Optional`

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #247

2 years ago[NEMO-399] Include the official WordCount example from the Beam website (#225)
Won Wook SONG [Wed, 23 Oct 2019 12:44:48 +0000 (21:44 +0900)] 
[NEMO-399] Include the official WordCount example from the Beam website (#225)

JIRA: [NEMO-399:  Include the official WordCount example from the Beam website](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-399)

**Major changes:**
- Adds the WordCount example from the Beam website
- Fixes the README accordingly

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #225

2 years ago[NEMO-422] SonarCloud issues (#243)
Haeyoon Cho [Wed, 23 Oct 2019 12:16:57 +0000 (21:16 +0900)] 
[NEMO-422] SonarCloud issues (#243)

JIRA: [NEMO-422: SonarCloud issues](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-422)

**Major changes:**
- None

**Minor changes to note:**
- Minor fixes for SonarCloud

**Tests for the changes:**
- N/A

**Other comments:**
- None

Closes #243

2 years ago[NEMO-422] Address code quality issues for ResourceSitePass, NetworkTraceAnalysis...
Jangho Seo [Tue, 22 Oct 2019 12:19:45 +0000 (21:19 +0900)] 
[NEMO-422] Address code quality issues for ResourceSitePass, NetworkTraceAnalysis, PhysicalPlanGenerator, and bytetransfer (#244)

JIRA: [NEMO-422: SonarCloud issues](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-422)

**Major changes:**
- None

**Minor changes to note:**
- ResourceSitePass: Fixed an operand in arithmetic expression to express its type explicitly.
- example/NetworkTraceAnalysis: Replaced at-most-one-iteration loop with if-else.
- PhysicalPlanGenerator: Use isPresent instead of orElseThrow for sanity checks.
- bytetransfer: Removed 'volatile' for object types.

**Tests for the changes:**
- Existing tests should be sufficient.

**Other comments:**
- None

2 years agoignore factorypath (#245)
Jangho Seo [Sun, 20 Oct 2019 06:42:30 +0000 (15:42 +0900)] 
ignore factorypath (#245)

JIRA: [NEMO-428: Ignore .factorypath for rat check and version control](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-428)

**Major changes:**
- None

**Minor changes to note:**
- Ignore .factorypath for rat check and version control

**Tests for the changes:**
- None

**Other comments:**
- None

2 years ago[NEMO-424] Fix Sonarcloud bugs regarding Optional (#241)
Sanha Lee [Tue, 15 Oct 2019 13:54:08 +0000 (22:54 +0900)] 
[NEMO-424] Fix Sonarcloud bugs regarding Optional (#241)

JIRA: [NEMO-424: Fix Sonarcloud bugs regarding Optional](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-424)

**Major changes:**
- Fixes Sonarcloud bugs regarding Optional

**Minor changes to note:**
- Fixes other Sonarcloud bugs assigned to johnyangk
- Fixes other Sonarcloud bugs assigned to sanha

closes #241

2 years ago[NEMO-422] Sonarcloud bugs (#242)
Won Wook SONG [Mon, 14 Oct 2019 13:08:25 +0000 (22:08 +0900)] 
[NEMO-422] Sonarcloud bugs (#242)

JIRA: [NEMO-422: Sonarcloud issues](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-422)

**Major changes:**
- None

**Minor changes to note:**
- Minor fixes for SonarCloud

**Tests for the changes:**
- N/A

**Other comments:**
- None

Closes #242

3 years ago[NEMO-375] Clean up of remnants of #211 (#224)
Won Wook SONG [Tue, 24 Sep 2019 12:29:04 +0000 (21:29 +0900)] 
[NEMO-375] Clean up of remnants of #211 (#224)

JIRA: [NEMO-375: Add option to turn off metric collection to DB](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-375)

**Major changes:**
- None

**Minor changes to note:**
- Removes the static actions that happen nevertheless of the configuration and misleads users

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #224

3 years ago[NEMO-418] BlockFetchFailureProperty (#238)
John Yang [Tue, 24 Sep 2019 08:27:33 +0000 (17:27 +0900)] 
[NEMO-418] BlockFetchFailureProperty (#238)

JIRA: NEMO-418: BlockFetchFailureProperty

Major changes:

Introduces BlockFetchFailureProperty that is useful for IREdges such as those from transient to reserved resources
In BlockManagerWorker, provides an option to wait until fetching "all elements" of each block
In ParentTaskDataFetcher, provides an option to retry fetching blocks
Minor changes to note:

Implements BlockInputReader#retry
Uses a guava cache with a 2-second eviction policy for block location responses
TransientResourcePolicy uses BlockFetchFailureProperty
Tests for the changes:

ParentTaskDataFetcherTest#testErrorWhenFutureWithRetry

3 years ago[NEMO-422] Sonarcloud issues (#239)
Won Wook SONG [Mon, 9 Sep 2019 09:50:39 +0000 (18:50 +0900)] 
[NEMO-422] Sonarcloud issues (#239)

JIRA: [NEMO-422: Sonarcloud issues](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-422)

**Major changes:**
- Resolves a number of sonarcloud issues

**Minor changes to note:**
- Resolves a number of sonarcloud issues

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #239

3 years ago[NEMO-400] Javadoc compile error (#226)
Won Wook SONG [Sun, 8 Sep 2019 08:39:54 +0000 (17:39 +0900)] 
[NEMO-400] Javadoc compile error (#226)

JIRA: [NEMO-400: Javadoc compile error](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-400)

**Major changes:**
- None

**Minor changes to note:**
- Fixes the line that causes an error on the javadoc generation.
- The `./bin/generate_javadocs.sh` command succeeds now

**Tests for the changes:**
- The checkstyle now checks for the new rules that have been left out, instead of creating a javadoc for every execution of tests.

**Other comments:**
- None

Closes #226

3 years ago[NEMO-397] Separation of JVM heap region and off-heap memory region (#236)
Haeyoon Cho [Mon, 26 Aug 2019 03:01:19 +0000 (12:01 +0900)] 
[NEMO-397] Separation of JVM heap region and off-heap memory region (#236)

JIRA: [NEMO-397: Separation of JVM heap region and off-heap memory region](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-397)

**Major changes:**
- Off-heap memory ratio option added
- slack region reserved for off-heap memory region by adding the two

Closes #236

3 years ago[NEMO-414] Command-line specified runtime data plane configurations not applied ...
Haeyoon Cho [Fri, 23 Aug 2019 07:09:23 +0000 (16:09 +0900)] 
[NEMO-414] Command-line specified runtime data plane configurations not applied (#235)

JIRA: [NEMO-414: Command-line specified runtime data plane configurations not applied](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-414)

**Major changes:**
- User-specified data plane configuration now applies to parameters.

**Minor changes to note:**
- The number of serialization threads for scheduling(schedule_ser_thread) now can be set through the command-line argument.

3 years ago[NEMO-404] Provide user argument to use lambda executor representer (#230)
Gao Zhiyuan [Tue, 20 Aug 2019 00:46:00 +0000 (09:46 +0900)] 
[NEMO-404] Provide user argument to use lambda executor representer (#230)

JIRA: [NEMO-404: Provide user argument to use lambda executor representer] https://issues.apache.org/jira/projects/NEMO/issues/NEMO-404)

**Major changes:**
- Prodivde user argument to use the new representer

**Other comments:**

Try it out by
```
./bin/run_beam.sh \
        -job_id wordCount_testexecutor \
        -executor_json `pwd`/examples/resources/executors/beam_test_executor_resources.json \
        -optimization_policy org.apache.nemo.compiler.optimizer.policy.DefaultPolicy \
        -user_main org.apache.nemo.examples.beam.WordCount \
        -executor_type lambda \
        -user_args "`pwd`/examples/resources/inputs/test_input_wordcount `pwd`/outputs/wordcount --numEventGenerators=1"
```

Blocked by [NEMO-398: ExecutorRepresenter interface and DefaultExecutorRepresenter ](https://github.com/apache/incubator-nemo/pull/227)

Closes #229

3 years ago[NEMO-413] Fix index checking for byte access of MemoryChunk using UNSAFE #234
Haeyoon Cho [Mon, 19 Aug 2019 05:03:05 +0000 (14:03 +0900)] 
[NEMO-413] Fix index checking for byte access of MemoryChunk using UNSAFE #234

JIRA: [NEMO-413: Fix index checking for byte access of MemoryChunk using UNSAFE](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-413)

**Major changes:**
- Index checking for byte operation using UNSAFE fixed: type size for byte changed from 0 to 1.

Closes #234

3 years ago[NEMO-412] Address Sonar Cloud issue for MemoryChunk (#233)
Haeyoon Cho [Mon, 19 Aug 2019 04:36:58 +0000 (13:36 +0900)] 
[NEMO-412] Address Sonar Cloud issue for MemoryChunk (#233)

JIRA: [NEMO-412: Address Sonar Cloud issue for MemoryChunk](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-412)

**Major changes:**
- `checkIndex()` in `MemoryChunk` is fixed to return either true or false.

**Minor changes to note:**
- None

Closes #233

3 years ago[NEMO-398] ExecutorRepresenter interface and DefaultExecutorRepresenter (#227)
Gao Zhiyuan [Sun, 11 Aug 2019 07:08:45 +0000 (16:08 +0900)] 
[NEMO-398] ExecutorRepresenter interface and DefaultExecutorRepresenter (#227)

JIRA: [NEMO-398: ExecutorRepresenter interface and LambdaExecutorRepresenter ](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-398)

**Major Changes:**
`DefaultExecutorRepresenter` has the same function as previous `ExecutorRepresenter` does.
Now `DefaultExecutorRepresenter` implements `ExecutorRepresenter` interface.

**Tests for the changes:**
- Not any

3 years ago[NEMO-388] Off-heap memory management (reuse ByteBuffer) (#223)
Haeyoon Cho [Wed, 7 Aug 2019 06:27:20 +0000 (15:27 +0900)] 
[NEMO-388] Off-heap memory management (reuse ByteBuffer) (#223)

JIRA: [NEMO-388: Off-heap memory management (reuse ByteBuffer)](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-388)

**Major changes:**
- New executor option: `max_offheap_mb` and `chunk_size_kb`
- `MemoryPoolAssigner` class is involved in block creation to use off-heap memory
- `ByteBuffer` is now reused, wrapped by `MemoryChunk`

**Tests for the changes:**
- `MemoryChunk` put/get tests included
- `MemoryPoolAssigner` tests included

**Other comments:**
- Design doc attached in JIRA will be edited soon

3 years ago[Nemo-402] Fix guava version conflict (#228)
Gao Zhiyuan [Sat, 3 Aug 2019 01:33:52 +0000 (10:33 +0900)] 
[Nemo-402] Fix guava version conflict (#228)

JIRA: [NEMO-402: Broken guava version conflicts cause ERROR: Trying to remove a RunningJob that is unknown ](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-###)

**Major changes:**
- guava version 19.0 -> 20.0

3 years ago[NEMO-350] Implement Off-heap SerializedMemoryStore & [NEMO-384] Implement DirectByte...
Haeyoon Cho [Tue, 25 Jun 2019 06:29:49 +0000 (15:29 +0900)] 
[NEMO-350] Implement Off-heap SerializedMemoryStore & [NEMO-384] Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore (#222)

JIRA: [NEMO-350: Implement Off-heap SerializedMemoryStore](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-#350)
[NEMO-384: Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-#384)

**Major changes:**
- When a block is emitted by an executor, we write it directly to off-heap memory using `DirectByteBufferOutputStream` and `DirectByteBufferOutputStream`.

**Minor changes to note:**
- `getData()` and `getBuffer` should be distinguished when acquiring data in `SerializedPartition`

**Other comments:**
- This implementation does not ensure performance gain since the overhead of `allocateDirect` (malloc) surpasses the garbage collection overhead. For this reason, memory management is being implemented.

3 years ago[NEMO-395] Address SonarCloud issues for the scheduler package (#220)
John Yang [Fri, 14 Jun 2019 13:19:21 +0000 (22:19 +0900)] 
[NEMO-395] Address SonarCloud issues for the scheduler package (#220)

JIRA: [NEMO-395: Address SonarCloud issues for the scheduler package](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-395)

**Major changes:**
- Fixes all "Critical" issues
- Fixes most of the "Major" issues

3 years ago[NEMO-377] Fix watermark emission when there are no outputs in GBKWindowTransform...
Taegeon Um [Tue, 11 Jun 2019 03:19:32 +0000 (12:19 +0900)] 
[NEMO-377] Fix watermark emission when there are no outputs in GBKWindowTransform (#210)

JIRA: [NEMO-377: Fix watermark emission when there are no outputs in GBKWindowTransform](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-377)

**Major changes:**
- Set min value when there are no outputs in GBKWindow

3 years ago[NEMO-391] Set GrpcMessageEnvironment as a default implementation (#218)
Taegeon Um [Mon, 10 Jun 2019 01:15:14 +0000 (10:15 +0900)] 
[NEMO-391] Set GrpcMessageEnvironment as a default implementation (#218)

JIRA: [NEMO-391: Set GrpcMessageEnvironment as a default implementation](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-391)

**Major changes:**
- Change the default implementation of MessageEnvironment to GrpcMessageEnvironment

3 years ago[NEMO-385]LambdaPass, LambdaPolicy and ResourceLambdaProperty (#214)
Gao Zhiyuan [Fri, 7 Jun 2019 07:18:54 +0000 (16:18 +0900)] 
[NEMO-385]LambdaPass, LambdaPolicy and ResourceLambdaProperty (#214)

JIRA: [NEMO-385: Support Lambda Pass with lambda policy and lambda resource property](https://issues.apache.org/jira/browse/NEMO-385)

**Major changes:**
- Implement LambdaPass, LambdaPolicy and ResourceLambdaProperty

**Tests for the changes:**
- No test suites implemented

**Other comments:**
- This issue is a subissue under [NEMO-352 Nemo on AWS Lambda](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-352)
- This PR relates to GSoC2019

3 years ago[NEMO-390] Address SonarCloud issues for the IR package (#217)
John Yang [Fri, 7 Jun 2019 03:56:18 +0000 (12:56 +0900)] 
[NEMO-390] Address SonarCloud issues for the IR package  (#217)

JIRA: [NEMO-390: Address SonarCloud issues for the IR package](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-390)

**Major changes:**
- Fixes transient/serializable issues raised by SonarCloud

**Minor changes to note:**
- Renames MessageBarrierVertex to TriggerVertex
- Renames StreamVertex to RelayVertex

Closes #217

3 years ago[NEMO-383] Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore...
Haeyoon Cho [Mon, 15 Apr 2019 08:59:30 +0000 (17:59 +0900)] 
[NEMO-383] Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore (#213)

JIRA: [NEMO-383: Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-383)

**Major changes:**
- Added `DirectByteBufferOuputStream` for writing data off-heap.

**Tests for the changes:**
- Added `DirectByteBufferOutputStreamTest` to test proper write using the `DirectByteBufferOutputStream`.

3 years ago[NEMO-379] Change javadoc goal to a proper one
Won Wook SONG [Mon, 15 Apr 2019 05:06:49 +0000 (14:06 +0900)] 
[NEMO-379] Change javadoc goal to a proper one

JIRA: [NEMO-379: Change javadoc goal to a proper one](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-379)

**Major changes:**
- Change javadoc test command from `javadoc` to `test-javadoc`. This prevents tests from creating unwanted javdoc files, caused by #139 .

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #212

3 years ago[NEMO-375] Add option to turn off metric collection to DB (#211)
Won Wook SONG [Fri, 5 Apr 2019 14:56:46 +0000 (23:56 +0900)] 
[NEMO-375] Add option to turn off metric collection to DB (#211)

JIRA: [NEMO-375: Add option to turn off metric collection to DB](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-375)

**Major changes:**
- Adds configuration for the DB to turn off metric collection to DB by default.
- Adds proper README docs

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #211

3 years ago[NEMO-360] Implementing an 'XGBoostPolicy' (#203)
Won Wook SONG [Sat, 30 Mar 2019 15:17:01 +0000 (00:17 +0900)] 
[NEMO-360] Implementing an 'XGBoostPolicy' (#203)

JIRA: [NEMO-360: Implementing an 'XGBoostPolicy'](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-360)

**Major changes:**
- Adds a python script that runs XGBoost.
  - as well as a bash script for using the python script.
- Uses the Client <-> Driver RPC to run the XGBoost script and return the results to the driver.
- Enables environment tag.
- Refactors the utility classes accordingly.

**Minor changes to note:**
- Miscellaneous methods to help the implementation (e.g., getEdgeById)
- Consistency in the shell scripts
- Appropriately adds files to the gitignore
- Javadocs

**Tests for the changes:**
- MetricUtilsTest confirms that changing an EP to indices and back to EP works correctly (TDD)

**Other comments:**
- None

Closes #203

3 years ago[NEMO-368] include all jar files for global libraries (#209)
Taegeon Um [Sat, 30 Mar 2019 13:10:40 +0000 (22:10 +0900)] 
[NEMO-368] include all jar files for global libraries (#209)

JIRA: [NEMO-368: NEMO-353 breaks the application from running in YARN environments](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-368)

**Major changes:**
- includes all jar files in the class path when starting NemoDriver

3 years ago[NEMO-362] Upgrade of checkstyle version (#208)
Won Wook SONG [Wed, 27 Mar 2019 09:22:17 +0000 (18:22 +0900)] 
[NEMO-362] Upgrade of checkstyle version (#208)

JIRA: [NEMO-362: Upgrade of checkstyle version](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-362)

**Major changes:**
- Upgrade of the checkstyle version.

**Minor changes to note:**
- Removes redundant final modifiers, which creates errors generated by newer check style versions.

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #208

3 years ago[NEMO-361] Consistency on indentations (#207)
Won Wook SONG [Wed, 27 Mar 2019 08:15:43 +0000 (17:15 +0900)] 
[NEMO-361] Consistency on indentations (#207)

JIRA: [NEMO-361: Consistency on indentations](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-361)

**Major changes:**
- Runs the 'Reformat code' command on IntelliJ to auto-indent files.
- Re-organizes imports in an alphabetical order

**Minor changes to note:**
- None

**Tests for the changes:**
- None, it simply fixes the indentations and imports

**Other comments:**
- None

Closes #207

3 years ago[NEMO-369] DirectByteArrayOutputStream usage refactoring (#206)
Haeyoon Cho [Tue, 26 Mar 2019 13:40:01 +0000 (22:40 +0900)] 
[NEMO-369] DirectByteArrayOutputStream usage refactoring (#206)

JIRA: [NEMO-369: DirectByteArrayOutputStream usage refactoring](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-369)

**Major changes:**
- Replaced usage of duplicated method getCount() to size()
- deleted getCount() method

**Minor changes to note:**
- None

**Tests for the changes:**
- None

**Other comments:**
- None

Closes #206

3 years ago[NEMO-364] Upgrade Beam version (#204)
Won Wook SONG [Wed, 20 Mar 2019 11:38:56 +0000 (20:38 +0900)] 
[NEMO-364] Upgrade Beam version (#204)

JIRA: [NEMO-364: Upgrade Beam](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-364)

**Major changes:**
- Upgrades the Beam version to 2.11.0, the newest version, to fix various errors.

**Minor changes to note:**
- Updates to the deprecated packages and syntaxes.

**Tests for the changes:**
- Existing tests confirm the changes

**Other comments:**
- None

Closes #204

3 years ago[NEMO-358] Recycling vertex ids while cloning a vertex (#201)
Won Wook SONG [Tue, 19 Mar 2019 06:36:44 +0000 (15:36 +0900)] 
[NEMO-358] Recycling vertex ids while cloning a vertex  (#201)

JIRA: [NEMO-358: Recycling vertex ids while cloning a vertex](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-358)

**Major changes:**
- Implementation on the IDManager so that it saves the IDs that are no longer used, to be reused later on.
- Fixes the LoopExtractionPass (where cloning occurs) to meet with the new implementations.

**Minor changes to note:**
- None

**Tests for the changes:**
- Existing tests confirm the changes.

**Other comments:**
- None

Closes #201

3 years ago[NEMO-357] Fix broken link on README (#200)
Won Wook SONG [Mon, 18 Mar 2019 07:20:21 +0000 (16:20 +0900)] 
[NEMO-357] Fix broken link on README (#200)

JIRA: [NEMO-357: Fix broken link on README](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-357)

**Major changes:**
- None

**Minor changes to note:**
- Fixes the broken sonar cloud link on the README

**Tests for the changes:**
- N/A

**Other comments:**
- None

Closes #200

3 years ago[NEMO-359] Implementation of getEstimatedSizeBytes in SourceVertex (#202)
Won Wook SONG [Mon, 18 Mar 2019 06:51:11 +0000 (15:51 +0900)] 
[NEMO-359] Implementation of getEstimatedSizeBytes in SourceVertex (#202)

JIRA: [NEMO-359: Implementation of getEstimatedSizeBytes in SourceVertex](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-359)

**Major changes:**
- Addition of getEstimatedSizeBytes on the SourceVertex interface
- Implementation of the method for the bounded sources.

**Minor changes to note:**
- None

**Tests for the changes:**
- N/A

**Other comments:**
- None

Closes #202

3 years ago[NEMO-356] Visualize the name of beam transform in DAG (#199)
Taegeon Um [Sun, 17 Mar 2019 09:04:04 +0000 (18:04 +0900)] 
[NEMO-356] Visualize the name of beam transform in DAG (#199)

JIRA: [NEMO-356: Visualize the name of beam transform in DAG
](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-356)

**Major changes:**
- populate the name of Beam transform.

The visualized DAGs have been uploaded to https://issues.apache.org/jira/projects/NEMO/issues/NEMO-356

3 years ago[NEMO-353] Launch NEXMark applications (#198)
Taegeon Um [Mon, 11 Mar 2019 08:36:53 +0000 (17:36 +0900)] 
[NEMO-353] Launch NEXMark applications (#198)

JIRA: [NEMO-353: Launch NEXMark applications](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-353)

**Major changes:**
- Remove nemo-client dependency from nemo-compiler-frontend-beam and create shaded-client.jar file to include it for the execution of NEXMark jobs.
- Create an example package that contains NEXMark jar file.
- Fix NullPointerException in `BeamUnboundedSourceVertex` when executing NEXMark applications.

**Minor changes to note:**
-Update README