doris-flink-connector.git
6 days ago[improvement] fix uniq delete operation (#63) master
wudi [Mon, 26 Sep 2022 01:40:12 +0000 (09:40 +0800)] 
[improvement] fix uniq delete operation (#63)

fix uniq delete operation

2 weeks ago[Feature] Support flink table lookup join (#61)
wudi [Thu, 15 Sep 2022 06:08:41 +0000 (14:08 +0800)] 
[Feature] Support flink table lookup join (#61)

* add flink table lookup join
Co-authored-by: wudi <>
2 weeks ago[docs] update readme.md flink version (#62)
zhou zhuohan [Wed, 14 Sep 2022 01:37:17 +0000 (09:37 +0800)] 
[docs] update readme.md flink version (#62)

Co-authored-by: ninjazhou <ninjazhou@tencent.com>
3 weeks agoadd tips for Mac OS users in case meeting getopt error (#59)
caoliang-web [Tue, 6 Sep 2022 08:43:51 +0000 (16:43 +0800)] 
add tips for Mac OS users in case meeting getopt error (#59)

add tips for Mac OS users in case meeting getopt error

3 weeks ago[Feature] support doris catalog (#60)
wudi [Tue, 6 Sep 2022 08:35:04 +0000 (16:35 +0800)] 
[Feature] support doris catalog (#60)

* add doris catalog and fix thrift Concurrency bug

4 weeks agofeat(.gitignore): added *.DS_Store (#58)
DingGeGe [Tue, 30 Aug 2022 10:18:18 +0000 (18:18 +0800)] 
feat(.gitignore): added *.DS_Store (#58)

2 months ago[docs] Fix broken link for doris connector docs (#51) release-1.2.0
Paul Lin [Fri, 29 Jul 2022 07:02:40 +0000 (15:02 +0800)] 
[docs] Fix broken link for doris connector docs (#51)

2 months ago Solve the exception caused by MaxRetries = 0 (#50)
yeyudefeng [Mon, 25 Jul 2022 11:31:25 +0000 (19:31 +0800)] 
 Solve the exception caused by MaxRetries = 0 (#50)

 Solve the exception caused by MaxRetries = 0

2 months ago[Feature] Support Flink 1.15 (#49)
wudi [Thu, 21 Jul 2022 11:54:36 +0000 (19:54 +0800)] 
[Feature] Support Flink 1.15 (#49)

* flink 1.15 support

2 months ago[fix]The log does not show the problem (#48)
jiafeng.zhang [Tue, 12 Jul 2022 12:56:54 +0000 (20:56 +0800)] 
[fix]The log does not show the problem (#48)

The log does not show the problem

2 months ago[TLP] remove incubator (#46)
Mingyu Chen [Sun, 10 Jul 2022 00:43:48 +0000 (08:43 +0800)] 
[TLP] remove incubator (#46)

remove incubator

2 months agoremove DISCLAIMER (#45)
jiafeng.zhang [Fri, 8 Jul 2022 12:00:39 +0000 (20:00 +0800)] 
remove DISCLAIMER (#45)

remove DISCLAIMER

3 months ago[optimization] add disable 2pc config (#34) release-1.1.0
madong [Thu, 30 Jun 2022 14:56:33 +0000 (22:56 +0800)] 
[optimization] add disable 2pc config (#34)

3 months agoFIX: fix wrap null (#41)
Stalary [Thu, 30 Jun 2022 14:55:33 +0000 (22:55 +0800)] 
FIX: fix wrap null (#41)

3 months agoLog4j upgrade (#40)
jiafeng.zhang [Thu, 23 Jun 2022 07:12:51 +0000 (15:12 +0800)] 
Log4j upgrade (#40)

upgrade jackson,log4j version

3 months agolink fix (#39)
jiafeng.zhang [Thu, 23 Jun 2022 01:28:48 +0000 (09:28 +0800)] 
link fix (#39)

link fix

3 months ago[Update] Update README and modify pom.xml (#36)
Yankee [Thu, 23 Jun 2022 01:18:43 +0000 (09:18 +0800)] 
[Update] Update README and modify pom.xml (#36)

* Update README

3 months agoremove incubating (#35)
wudi [Mon, 20 Jun 2022 01:06:34 +0000 (09:06 +0800)] 
remove incubating (#35)

Co-authored-by: wudi <>
3 months ago[Fix] fix flink date and timestamp type not mapping
gj-zhang [Sun, 12 Jun 2022 13:56:19 +0000 (21:56 +0800)] 
[Fix] fix flink date and timestamp type not mapping

* [FIX] fix flink date and timestamp type not mapping.
* The type conversion here is recommended to be abstracted and placed in DorisRowConverter.
* overload the constructor for the DorisRowConverter.
add unit test for external convert
* add license header.

4 months agofix broke url and ut (#32)
cxzl25 [Mon, 9 May 2022 06:31:59 +0000 (14:31 +0800)] 
fix broke url and ut (#32)

5 months agofix flink schema and doris schema column mapping (#30)
wudi [Mon, 25 Apr 2022 05:49:30 +0000 (13:49 +0800)] 
fix flink schema and doris schema column mapping (#30)

fix flink schema and doris schema column mapping

5 months ago[Feature] Refactoring DorisSouce based on FLIP-27 (#24)
wudi [Thu, 21 Apr 2022 02:40:47 +0000 (10:40 +0800)] 
[Feature] Refactoring DorisSouce based on FLIP-27 (#24)

* Refactoring DorisSouce based on FLIP-27

5 months ago[Bug] Fix row type decimal convert bug (#26)
aiwenmo [Fri, 15 Apr 2022 03:01:08 +0000 (11:01 +0800)] 
[Bug] Fix row type decimal convert bug (#26)

* Fix row type decimal convert bug

5 months ago[improvement] json load by line (#23)
madong [Fri, 8 Apr 2022 01:29:40 +0000 (09:29 +0800)] 
[improvement] json load by line (#23)

6 months agoUpdate README.md (#19)
BAKEZQ [Fri, 25 Mar 2022 04:12:46 +0000 (12:12 +0800)] 
Update README.md (#19)

* Update README.md

6 months ago[refactor]doris sink enable 2pc (#12)
madong [Tue, 8 Mar 2022 01:45:45 +0000 (09:45 +0800)] 
[refactor]doris sink enable 2pc (#12)

* [refactor]doris sink enable 2pc

6 months ago[chore] Change project name (#16)
Mingyu Chen [Sun, 6 Mar 2022 02:53:40 +0000 (10:53 +0800)] 
[chore] Change project name (#16)

Change project name (#16)

6 months ago[chore] change the GAV (#14)
Mingyu Chen [Sun, 6 Mar 2022 02:33:18 +0000 (10:33 +0800)] 
[chore] change the GAV (#14)

1. modify GAV to "flink-doris-connector-{flink.minor.version}_${scala_version}"

7 months ago[improvement] Supports traversal of Doris FE nodes when searching for Doris BE (#11)
Jiangqiao Xu [Sun, 27 Feb 2022 03:08:53 +0000 (11:08 +0800)] 
[improvement] Supports traversal of Doris FE nodes when searching for Doris BE (#11)

7 months ago[chore] modify build.sh and github action for 1.14 (#10)
Mingyu Chen [Fri, 25 Feb 2022 05:24:46 +0000 (13:24 +0800)] 
[chore] modify build.sh and github action for 1.14 (#10)

7 months agoFix notification settings not taking effect (#8)
Kirs [Fri, 25 Feb 2022 05:24:31 +0000 (13:24 +0800)] 
Fix notification settings not taking effect (#8)

7 months ago[fix] When doing checkpoint, write cache data to doris to prevent loss (#4)
liuyaolin [Tue, 22 Feb 2022 11:19:32 +0000 (19:19 +0800)] 
[fix] When doing checkpoint, write cache data to doris to prevent loss (#4)

7 months ago[feature] Flink 1.14 support (#6)
wudi [Mon, 21 Feb 2022 12:31:29 +0000 (20:31 +0800)] 
[feature] Flink 1.14 support (#6)

7 months ago[community] modify NOTICE and CONTRIBUTING doc (#7)
Mingyu Chen [Mon, 21 Feb 2022 12:31:03 +0000 (20:31 +0800)] 
[community] modify NOTICE and CONTRIBUTING doc (#7)

7 months ago[chore] Modify the build.sh script (#5)
Mingyu Chen [Fri, 18 Feb 2022 03:48:12 +0000 (11:48 +0800)] 
[chore] Modify the build.sh script (#5)

Add "--tag" option so that when we can build from a tag.
Because in tag, the flink and scala version is deterministic and does not need to be specified.

7 months ago[chore] add maven-gpg-plugin (#3)
Mingyu Chen [Mon, 14 Feb 2022 08:14:12 +0000 (16:14 +0800)] 
[chore] add maven-gpg-plugin (#3)

7 months ago[chore] change the repo name (#2)
Mingyu Chen [Mon, 14 Feb 2022 04:46:58 +0000 (12:46 +0800)] 
[chore] change the repo name (#2)

7 months ago[chore] add maven-release-plugin plugin (#1)
Mingyu Chen [Mon, 14 Feb 2022 04:42:13 +0000 (12:42 +0800)] 
[chore] add maven-release-plugin plugin (#1)

Try to solve the error when executing "mvn release:prepare"

7 months ago[init] add .asf.yaml
morningman [Fri, 11 Feb 2022 15:37:33 +0000 (23:37 +0800)] 
[init] add .asf.yaml

7 months ago[init] do some init work
morningman [Fri, 11 Feb 2022 15:07:49 +0000 (23:07 +0800)] 
[init] do some init work

7 months ago[init] init commit
morningman [Fri, 11 Feb 2022 14:59:52 +0000 (22:59 +0800)] 
[init] init commit

Move flink-doris-connector from incubator-doris@df2c756

8 months ago[fix](httpv2) make http v2 and v1 interface compatible (#7848)
jiafeng.zhang [Mon, 31 Jan 2022 14:12:34 +0000 (22:12 +0800)] 
[fix](httpv2) make http v2 and v1 interface compatible (#7848)

http v2 TableSchemaAction adds the return value of aggregation_type,
and modifies the corresponding code of Flink/Spark Connector

8 months ago[chore][fix][doc](fe-plugin)(mysqldump) fix build auditlog plugin error (#7804)
Zhengguo Yang [Wed, 26 Jan 2022 01:11:23 +0000 (09:11 +0800)] 
[chore][fix][doc](fe-plugin)(mysqldump) fix build auditlog plugin error (#7804)

1. fix problems when build fe_plugins
2. format
3. add docs about dump data using mysql dump

8 months ago[Feature][flink-connector] support flink delete option (#7457)
wudi [Sun, 23 Jan 2022 12:24:41 +0000 (20:24 +0800)] 
[Feature][flink-connector] support flink  delete option (#7457)

* Flink Connector supports delete option on Unique models
Co-authored-by: wudi <wud3@shuhaisc.com>
8 months agoFlink / Spark connector compilation problem (#7725)
jiafeng.zhang [Fri, 14 Jan 2022 14:14:48 +0000 (22:14 +0800)] 
Flink / Spark connector compilation problem (#7725)

Flink / Spark connector compilation problem

8 months ago[chore][docs] add deploy spark/flink connectors to maven release repo docs (#7616)
Zhengguo Yang [Thu, 6 Jan 2022 15:23:33 +0000 (23:23 +0800)] 
[chore][docs] add deploy spark/flink connectors to maven release repo docs (#7616)

8 months ago[refactor] update parent pom version and optimize build scripts (#7548)
Zhengguo Yang [Wed, 5 Jan 2022 02:45:11 +0000 (10:45 +0800)] 
[refactor] update parent pom  version and optimize build scripts (#7548)

9 months ago[refactor] Standardize the writing of pom files, prepare for deployment to maven...
Zhengguo Yang [Thu, 30 Dec 2021 02:16:37 +0000 (10:16 +0800)] 
[refactor] Standardize the writing of pom files, prepare for deployment to maven (#7477)

9 months ago[improvement](spark-connector)(flink-connector) Modify the max num of batch written...
jiafeng.zhang [Sun, 26 Dec 2021 03:13:47 +0000 (11:13 +0800)] 
[improvement](spark-connector)(flink-connector) Modify the max num of batch written by Spark/Flink connector each time. (#7485)

Increase the default batch size and flush interval

9 months ago[improvement](flink-connector) flush data without multi httpclients (#7329) (#7450)
Heng Zhao [Fri, 24 Dec 2021 13:28:35 +0000 (21:28 +0800)] 
[improvement](flink-connector) flush data without multi httpclients (#7329) (#7450)

reuse http client to flush data

9 months ago[fix](flink-connector) Connector should visit the surviving BE nodes (#7435)
Heng Zhao [Tue, 21 Dec 2021 03:05:42 +0000 (11:05 +0800)] 
[fix](flink-connector) Connector should visit the surviving BE nodes (#7435)

9 months ago[improvement](flink-connector) DataSourceFunction read doris supports parallel (...
wudi [Wed, 15 Dec 2021 08:21:29 +0000 (16:21 +0800)] 
[improvement](flink-connector) DataSourceFunction read doris supports parallel (#7232)

The previous DataSourceFunction inherited from RichSourceFunction.
As a result, no matter how much the parallelism of flink is set, the parallelism of DataSourceFunction is only 1.
Now modify it to RichParallelSourceFunction.

And when flink has multiple degrees of parallelism, assign the doris data to each parallelism.
For example, read dorisPartitions.size = 10, flink.parallelism = 4
The task is split as follows:
task0: dorisPartitions[0],[4],[8]
task1: dorisPartitions[1],[5],[9]
task2: dorisPartitions[2],[6]
task3: dorisPartitions[3],[7]

9 months ago[chore][community](github) Remove travis and add github action (#7380)
Mingyu Chen [Wed, 15 Dec 2021 05:27:37 +0000 (13:27 +0800)] 
[chore][community](github) Remove travis and add github action (#7380)

1. Remove travis
2. Add github action to build extension:
    1. docs
    2. fs_broker
    3. flink/spark/connector

10 months ago[License] Add License header for missing files (#7130)
Mingyu Chen [Tue, 16 Nov 2021 10:37:54 +0000 (18:37 +0800)] 
[License] Add License header for missing files (#7130)

1. Add License header for missing files
2. Modify the spark pom.xml to correct the location of `thrift`

10 months ago[Feature] Support Flink and Spark connector support String type (#7075)
wudi [Sat, 13 Nov 2021 09:10:22 +0000 (17:10 +0800)] 
[Feature] Support Flink and Spark connector support String type (#7075)

Support String type for Flink and Spark connector

10 months ago[Build]Compile and output the jar file, add Spark, Flink version and Scala version...
jiafeng.zhang [Tue, 9 Nov 2021 02:02:08 +0000 (10:02 +0800)] 
[Build]Compile and output the jar file, add Spark, Flink version and Scala version (#7051)

The jar file compiled by Flink and Spark Connector, with the corresponding Flink, Spark version
and Scala version at compile time, so that users can know whether the version number matches when using it.

Example of output file name:doris-spark-1.0.0-spark-3.2.0_2.12.jar

10 months ago[HTTP][API] Add backends info API for spark/flink connector (#6984)
Mingyu Chen [Fri, 5 Nov 2021 01:43:06 +0000 (09:43 +0800)] 
[HTTP][API] Add backends info API for spark/flink connector (#6984)

Doris should provide a http api to return backends list for connectors to submit stream load,
and without privilege checking, which can let common user to use it

11 months agosupport use char like \x01 in flink-doris-sink column & line delimiter (#6937)
wunan1210 [Fri, 29 Oct 2021 05:56:52 +0000 (13:56 +0800)] 
support use char like \x01 in flink-doris-sink column & line delimiter (#6937)

* support use char like \x01 in flink-doris-sink column & line delimiter

* extend imports

* add docs

11 months ago[Flink]Simplify the use of flink connector (#6892)
xiaokangguo [Sat, 23 Oct 2021 10:10:47 +0000 (18:10 +0800)] 
[Flink]Simplify the use of flink connector  (#6892)

1. Simplify the use of flink connector like other stream sink by GenericDorisSinkFunction.
2. Add the use cases of flink connector.

## Use case
```
env.fromElements("{\"longitude\": \"116.405419\", \"city\": \"北京\", \"latitude\": \"39.916927\"}")
     .addSink(
          DorisSink.sink(
             DorisOptions.builder()
                   .setFenodes("FE_IP:8030")
                   .setTableIdentifier("db.table")
                   .setUsername("root")
                   .setPassword("").build()
                ));
```

11 months ago[Flink][Bug] Fix potential NPE when cancel DorisSourceFunction (#6838)
Yun Tang [Sat, 23 Oct 2021 08:45:24 +0000 (16:45 +0800)] 
[Flink][Bug] Fix potential NPE when cancel DorisSourceFunction (#6838)

Fix potential NPE of `scalaValueReader` when cancelling DorisSourceFunction.

11 months ago[Dependency] Upgrade thirdparty libs (#6766)
Zhengguo Yang [Fri, 15 Oct 2021 05:03:04 +0000 (13:03 +0800)] 
[Dependency] Upgrade thirdparty libs (#6766)

Upgrade the following dependecies:

libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0

after this pr compile doris should use build-env:1.4.0

12 months ago[Fix] Flink connector support json import and use httpclient to streamlaod (#6740)
wudi [Tue, 28 Sep 2021 09:37:03 +0000 (04:37 -0500)] 
[Fix] Flink connector support json import and use httpclient to streamlaod (#6740)

* [Bug]:fix when data null , throw NullPointerException

* [Bug]:Distinguish between null and empty string

* [Feature]:flink-connector supports streamload parameters

* [Fix]:code style

* [Fix]: support json format import and use httpclient to streamload

* [Fix]:remove System out

* [Fix]:upgrade httpclient  version

* [Doc]: add json format import doc

Co-authored-by: wudi <wud3@shuhaisc.com>
12 months ago[Flink] Fix bug of flink doris connector (#6655)
xhmz [Fri, 24 Sep 2021 13:38:35 +0000 (21:38 +0800)] 
[Flink] Fix bug of flink doris connector (#6655)

Flink-Doris-Connector do not support flink 1.13, refactor doris sink forma
to not use GenericRowData. But to use RowData::FieldGetter.

13 months ago[FlinkConnector] Make flink datastream source parameterized (#6473)
wunan1210 [Sun, 22 Aug 2021 14:03:32 +0000 (22:03 +0800)] 
[FlinkConnector] Make flink datastream source parameterized (#6473)

make flink datastream source parameterized as List<?> instead of Object.

13 months ago[Doc] flink/spark connector: add sources/javadoc plugins (#6435)
wunan1210 [Mon, 16 Aug 2021 14:41:24 +0000 (22:41 +0800)] 
[Doc] flink/spark connector: add sources/javadoc plugins (#6435)

spark-doris-connector/flink-doris-connect add plugins to generate javadoc and sources jar,
so can be easy to distribute and debug.

13 months ago[Feature]:Flink-connector supports streamload parameters (#6243)
wudi [Mon, 9 Aug 2021 14:12:46 +0000 (22:12 +0800)] 
[Feature]:Flink-connector supports streamload parameters (#6243)

Flink-connector supports streamload parameters
#6199

14 months ago[Bug][Flink] Fix when data null , flink-connector throw NullPointerException (#6165)
wudi [Thu, 8 Jul 2021 01:55:50 +0000 (09:55 +0800)] 
[Bug][Flink] Fix when data null , flink-connector throw NullPointerException (#6165)

15 months ago[FlinkConnector] Support time interval for flink connector (#5934)
wudi [Wed, 30 Jun 2021 01:27:12 +0000 (09:27 +0800)] 
[FlinkConnector] Support time interval for flink connector (#5934)

15 months ago[Log] Fix a mistake in DorisDynamicOutputFormat.java (#5963)
zhangboya1 [Sun, 6 Jun 2021 14:06:57 +0000 (22:06 +0800)] 
[Log] Fix a mistake in DorisDynamicOutputFormat.java (#5963)

Fix a mistake DorisDynamicOutputFormat.java

16 months ago[Bug] Modify spark, flink doris connector to send request to FE, fix the problem...
jiafeng.zhang [Wed, 19 May 2021 01:28:21 +0000 (09:28 +0800)] 
[Bug] Modify spark, flink doris connector to send request to FE, fix the problem of POST method, it should be the same as the method when sending the request (#5788)

Modify spark, flink doris connector to send request to FE, fix the problem of POST method,
it should be the same as the method when sending the request

17 months ago[Feature] Flink Doris Connector (#5372) (#5375)
wudi [Fri, 23 Apr 2021 01:43:48 +0000 (09:43 +0800)] 
[Feature] Flink Doris Connector (#5372) (#5375)