pinot.git
13 hours agoadd tests for LIMIT/OFFSET in QueryRunnerTest (#9496) master
Almog Gavra [Sat, 1 Oct 2022 01:28:41 +0000 (18:28 -0700)] 
add tests for LIMIT/OFFSET in QueryRunnerTest (#9496)

15 hours agodistinct from scalar functions (#9486)
Yao Liu [Fri, 30 Sep 2022 22:40:29 +0000 (15:40 -0700)] 
distinct from scalar functions (#9486)

18 hours agoFix AddTable for realtime tables (#9506)
Neha Pawar [Fri, 30 Sep 2022 20:31:19 +0000 (13:31 -0700)] 
Fix AddTable for realtime tables (#9506)

* Fix AddTable for realtime tables

* Type

18 hours agoadd debug string to query plan [MINOR] (#9497)
Almog Gavra [Fri, 30 Sep 2022 19:44:36 +0000 (12:44 -0700)] 
add debug string to query plan [MINOR] (#9497)

* add debug string to query plan

* fix checkstyle

* add server instances

* rename to explain and fix some bugs

* fix more bugs

* oops, remove sout line

21 hours agoadding optional client payload (#9465)
Rong Rong [Fri, 30 Sep 2022 16:44:38 +0000 (09:44 -0700)] 
adding optional client payload (#9465)

* adding optional client payload

* adding stable sort to make sure the result sets are not compared against random unstable sort value

Co-authored-by: Rong Rong <rongr@startree.ai>
29 hours agoAdd query interruption flag check to broker groupby reduction (#9499)
Jia Guo [Fri, 30 Sep 2022 09:25:40 +0000 (02:25 -0700)] 
Add query interruption flag check to broker groupby reduction (#9499)

* add query interruption flag check to broker groupby reduction

* add query interruption flag check to broker groupby reduction

* add query interruption flag check to broker groupby reduction

* add benchmark

* tiled loop

* add benchmark

* Trigger Test

34 hours agoIntroduce segment assignment strategy interface #9047 (#9309)
Sharayu Deepak Gandhi [Fri, 30 Sep 2022 03:41:26 +0000 (20:41 -0700)] 
Introduce segment assignment strategy interface #9047 (#9309)

{
  "segmentAssignmentConfigMap": {
    "OFFLINE": {
      "segmentAssignmentStrategy": "balanced/replicaGroup/allServers"
    }
  }
  ...
}

36 hours ago[multistage] restructure runner test (#9489)
Rong Rong [Fri, 30 Sep 2022 02:16:44 +0000 (19:16 -0700)] 
[multistage] restructure runner test (#9489)

* fix dispatcher/server shutdown

* fix lint

Co-authored-by: Rong Rong <rongr@startree.ai>
36 hours agoFix JsonExtractScalar when no value is extracted (#9500)
Xiaotian (Jackie) Jiang [Fri, 30 Sep 2022 02:14:37 +0000 (19:14 -0700)] 
Fix JsonExtractScalar when no value is extracted (#9500)

39 hours ago[Cleanup] Simplify the operator name for group-by (#9483)
Xiaotian (Jackie) Jiang [Thu, 29 Sep 2022 23:08:24 +0000 (16:08 -0700)] 
[Cleanup] Simplify the operator name for group-by (#9483)

40 hours agofix null transform bound check (#9495)
Rong Rong [Thu, 29 Sep 2022 22:33:36 +0000 (15:33 -0700)] 
fix null transform bound check (#9495)

* fix bound check
* add test

Co-authored-by: Rong Rong <rongr@startree.ai>
41 hours agoForwardIndexHandler: Change compressionType during segmentReload (#9454)
Vivek Iyer Vaidyanathan [Thu, 29 Sep 2022 21:06:41 +0000 (14:06 -0700)] 
ForwardIndexHandler: Change compressionType during segmentReload (#9454)

* FrwardIndexHandler: Allow changing compressionType for SV columns on reload

* Address review comments.

* Add more tests and address review comments

41 hours agoEliminate explosion of metrics due to gapfill queries (#9490)
Elon Azoulay [Thu, 29 Sep 2022 20:52:21 +0000 (13:52 -0700)] 
Eliminate explosion of metrics due to gapfill queries (#9490)

41 hours ago[multistage] add LEFT JOIN (#9446)
Rong Rong [Thu, 29 Sep 2022 20:41:04 +0000 (13:41 -0700)] 
[multistage] add LEFT JOIN (#9446)

* adding LEFT JOIN
* also add test for null after and fix case when

Co-authored-by: Rong Rong <rongr@startree.ai>
44 hours ago[multistage] support inequality JOIN (#9448)
Rong Rong [Thu, 29 Sep 2022 18:35:53 +0000 (11:35 -0700)] 
[multistage] support inequality JOIN (#9448)

* support inequality JOIN

* also support pure inequality join

* address diff comment and add a SEMI join test case for this as well

Co-authored-by: Rong Rong <rongr@startree.ai>
44 hours agoadd parsing for AT TIME ZONE (#9477)
Almog Gavra [Thu, 29 Sep 2022 17:50:04 +0000 (10:50 -0700)] 
add parsing for AT TIME ZONE (#9477)

* add parsing for AT TIME ZONE

This commit adds the custom Calcite extension for
parsing AT TIME ZONE expressions, but does not yet
support executing these queries.

45 hours agoAdd UI to show fine-grained minion task progress (#9488)
Jayesh Choudhary [Thu, 29 Sep 2022 17:34:50 +0000 (23:04 +0530)] 
Add UI to show fine-grained minion task progress (#9488)

2 days agoAdd a new API to fix segment date time in metadata (#9413)
Kartik Khare [Thu, 29 Sep 2022 11:55:53 +0000 (17:25 +0530)] 
Add a new API to fix segment date time in metadata (#9413)

* Add API to handle change in timestamp format

* Fix linting

* Add support for storing raw segment start and end time in metadata

* Do not store start end time in segment metadata seperately

* remove refresh flag

* Store new start/end time with proper time unit

* Move the API to segment resource

* Fix test failure

* Add test

* Cleanup: Remove duplicate methods

* Store start/end time in milliseconds in zookeeper

* Refactor: change method names and reduce scope of exceptions

* Remove redundant timeunit conversion

* Throw user errors and check for time column

* Fix Segment tests

Co-authored-by: Kartik Khare <kharekartik@Kartiks-MacBook-Pro.local>
2 days agofix regression due to refactoring (#9481)
Navina Ramesh [Thu, 29 Sep 2022 09:35:20 +0000 (15:05 +0530)] 
fix regression due to refactoring (#9481)

* fix regression due to refactoring

* lint fix

2 days agorefine the minion task progress api a bit (#9482)
Xiaobing [Thu, 29 Sep 2022 00:25:34 +0000 (17:25 -0700)] 
refine the minion task progress api a bit (#9482)

* refine the minion task progress api a bit

* fix ut

2 days agoBump Spark Connector max inbound message size (#9475)
Caner Balci [Wed, 28 Sep 2022 18:21:17 +0000 (11:21 -0700)] 
Bump Spark Connector max inbound message size (#9475)

2 days agoextend PinotFS impls with listFilesWithMetadata and some bugfix (#9478)
Xiaobing [Wed, 28 Sep 2022 17:01:17 +0000 (10:01 -0700)] 
extend PinotFS impls with listFilesWithMetadata and some bugfix (#9478)

3 days agoExtract record keys, headers and metadata from Stream sources (#9224)
Navina Ramesh [Wed, 28 Sep 2022 05:34:56 +0000 (11:04 +0530)] 
Extract record keys, headers and metadata from Stream sources (#9224)

* initial commit from kishore

adding StreamMessage concept and using it with kafka

move StreamMessage into pinot-spi

Use StreamDataDecoder interface for deocding a StreamMessage in LLRealtimeSegmentDataManager

verified key and header with realtime quick start

checkstyle

updating realtime quickstart to include headers and key parsing

fix failing tests

checkstyle

revert to continue using hlc in integration tests

added TODO for failing test; revert adding header in the integ test

nit

Deleting unused class: MessageAndOffset and MessageAndOffsetAndMetadata

* Adding metadata to RowMetadata to distinguish it from headers
Addressing feeback

* clear reuse genericrow

* Making offset as a part of KafkaStreamMessageMetadata; changed StreamMessage into a concrete class

* rename RowMetadataExtractor.java to KafkaMetadataExtractor.java

* adding unit test
addressing PR feedback

* Always populate offset and ingestion time in record time

* update the comment

* rename to recordIngestionTimeMs; Fixed unit test assertion in KafkaPartition consumer -  Kafka will always have metadata

* include kafka recordTimestamp in streamMessageMetadata object

* do not use Map.of

3 days agoLoad inverted index in BaseSingleValueQueriesTest (#9474)
Jia Guo [Tue, 27 Sep 2022 23:41:49 +0000 (16:41 -0700)] 
Load inverted index in BaseSingleValueQueriesTest (#9474)

* Load inverted index in test

* Fix test cases

* jdk 1.8

* check style

* fix BaseMultiValueQueriesTest and BaseMultiValueRawQueriesTest

3 days agocollect file info like mtime, length while listing files for free (#9466)
Xiaobing [Tue, 27 Sep 2022 21:41:57 +0000 (14:41 -0700)] 
collect file info like mtime, length while listing files for free (#9466)

3 days ago[hotfix] upgrade cache action version (#9469)
Rong Rong [Tue, 27 Sep 2022 21:40:30 +0000 (14:40 -0700)] 
[hotfix] upgrade cache action version (#9469)

* upgrade cache action version

3 days agoDecouple ser/de from DataTable (#9468)
Xiaotian (Jackie) Jiang [Tue, 27 Sep 2022 20:49:44 +0000 (13:49 -0700)] 
Decouple ser/de from DataTable (#9468)

3 days ago[enhancement] [release-note] Update get bytes to return raw bytes of string and suppo...
Yao Liu [Tue, 27 Sep 2022 16:14:41 +0000 (09:14 -0700)] 
[enhancement] [release-note] Update get bytes to return raw bytes of string and support getBytesMV (#9441)

* get raw bytes for string value column

* get raw bytes for string value column

* support getBytesMV

* fix fail test

* fix style

* Address sytle commets

* address comments

4 days agoFix Data-Correctness Bug in GTE Comparison in BinaryOperatorTransformFunction (#9461)
Ankit Sultana [Tue, 27 Sep 2022 02:52:32 +0000 (08:22 +0530)] 
Fix Data-Correctness Bug in GTE Comparison in BinaryOperatorTransformFunction (#9461)

* Fix Bug in Handling GTE Comparison in BinaryOperatorTransformFunction

* Add UT

* Fix bug and add another test

4 days agotrack progress from within segment processor framework (#9457)
Xiaobing [Tue, 27 Sep 2022 01:14:55 +0000 (18:14 -0700)] 
track progress from within segment processor framework (#9457)

* track progress from within segment processor framework

and keep full exception stack trace instead of the top line in task info field

* cr

4 days ago[multistage] generated test for v2 engine (#9452)
Rong Rong [Tue, 27 Sep 2022 01:13:14 +0000 (18:13 -0700)] 
[multistage] generated test for v2 engine (#9452)

* add generated test for v2 engine

* enable integration test

* fix condition checker

Co-authored-by: Rong Rong <rongr@startree.ai>
4 days ago[multistage][hotfix] fix type cast handling and value literal gen (#9456)
Rong Rong [Mon, 26 Sep 2022 21:53:34 +0000 (14:53 -0700)] 
[multistage][hotfix] fix type cast handling and value literal gen (#9456)

* also support implicit type casting
* also add value literal bug fix when casting
* address comments

Co-authored-by: Rong Rong <rongr@startree.ai>
4 days agorefactor LLRealtimeSegmentDataManager (#9424)
Navina Ramesh [Mon, 26 Sep 2022 21:15:57 +0000 (02:45 +0530)] 
refactor LLRealtimeSegmentDataManager (#9424)

* use only createPartitionMetadataProvider in LLRealtimeSegmentDataManager

* Moving columns into a container class

* remove from HLRealtimeSegmentDataManager

* Addressing PR feedback

* Remove instance variables

* rename private method

* renaming stream with partition

* lint

* kinsis consumer factory should return streamMetadataProvider for both partition and stream level

4 days agoAdaptive Server Selection: address review comments (#9462)
Vivek Iyer Vaidyanathan [Mon, 26 Sep 2022 20:28:54 +0000 (13:28 -0700)] 
Adaptive Server Selection: address review comments (#9462)

4 days agoBump snakeyaml from 1.30 to 1.33 (#9464)
dependabot[bot] [Mon, 26 Sep 2022 18:21:21 +0000 (11:21 -0700)] 
Bump snakeyaml from 1.30 to 1.33 (#9464)

Bumps [snakeyaml](https://bitbucket.org/snakeyaml/snakeyaml) from 1.30 to 1.33.
- [Commits](https://bitbucket.org/snakeyaml/snakeyaml/branches/compare/snakeyaml-1.33..snakeyaml-1.30)

---
updated-dependencies:
- dependency-name: org.yaml:snakeyaml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7 days agoRemove netty server query support from presto-pinot-driver to remove pinot-core and...
Xiang Fu [Sat, 24 Sep 2022 09:41:11 +0000 (02:41 -0700)] 
Remove netty server query support from presto-pinot-driver to remove pinot-core and pinot-segment-local dependencies (#9455)

7 days agoUse codecov action (#9459)
Xiang Fu [Sat, 24 Sep 2022 05:26:46 +0000 (22:26 -0700)] 
Use codecov action (#9459)

7 days agospark-connector should use v2/brokers endpoint (#9451)
Christopher Peck [Sat, 24 Sep 2022 01:11:58 +0000 (18:11 -0700)] 
spark-connector should use v2/brokers endpoint (#9451)

7 days agofix typo (#9458)
Xiaobing [Fri, 23 Sep 2022 21:22:18 +0000 (14:22 -0700)] 
fix typo (#9458)

7 days agoFix the range check for range index on raw column (#9453)
Xiaotian (Jackie) Jiang [Fri, 23 Sep 2022 16:57:39 +0000 (09:57 -0700)] 
Fix the range check for range index on raw column (#9453)

7 days agorefine minion worker event observer to track finer grained progress for tasks (#9449)
Xiaobing [Fri, 23 Sep 2022 16:15:48 +0000 (09:15 -0700)] 
refine minion worker event observer to track finer grained progress for tasks (#9449)

* refine minion worker event observer to track finer grained progress for tasks

* cr

* cr

8 days agoHandle Invalid timestamps (#9355)
Kartik Khare [Fri, 23 Sep 2022 06:37:39 +0000 (12:07 +0530)] 
Handle Invalid timestamps (#9355)

* Add sample value in datetime fieldspec to check correct format

* Bug fix and add test cases for sample value

* Fix linting

* throw exception in case of error

Co-authored-by: Kartik Khare <kharekartik@Kartiks-MacBook-Pro.local>
8 days ago[multistage][hotfix] fix filter operator type convert (#9450)
Rong Rong [Fri, 23 Sep 2022 05:45:36 +0000 (22:45 -0700)] 
[multistage][hotfix] fix filter operator type convert (#9450)

FilterOperand should use the non-cast result type,
- fix the current behavior to always use the left-hand side for type casting
- fixing type resulting in null issue

Co-authored-by: Rong Rong <rongr@startree.ai>
8 days ago[multistage] adding support for range predicate (#9445)
Rong Rong [Fri, 23 Sep 2022 04:50:14 +0000 (21:50 -0700)] 
[multistage] adding support for range predicate (#9445)

* adding rules to decompose range predicate into OR joined simple comparison.
* if the look up is point or complemented points, we should still use IN / NOT IN
* if the search result in a constant value (true of false) it might trigger logical value literal node, which is also supported here.

Co-authored-by: Rong Rong <rongr@startree.ai>
8 days agoCache Deleted Segment Names in Server to Avoid SegmentMissingError (#9423)
Ankit Sultana [Thu, 22 Sep 2022 23:27:07 +0000 (04:57 +0530)] 
Cache Deleted Segment Names in Server to Avoid SegmentMissingError (#9423)

* Cache Deleted Segment Names in Server to Avoid SegmentMissingError

* self-review: remove unneeded call + change type for numMissingSegments

* Address feedback

* Add UT + Address Feedback

8 days ago[Feature] Add BigDecimal Metric In QuickStart (#9390)
Antwi Gambrah [Thu, 22 Sep 2022 20:48:17 +0000 (20:48 +0000)] 
[Feature] Add BigDecimal Metric In QuickStart (#9390)

* enable  big decimal in fieldspec

* add big_decimal metric example table

* fix failing quickstart

* add empty lines

* add empty line to csv

* revert change

* Update FieldSpecTest.java

Co-authored-by: Xiang Fu <xiangfu.1024@gmail.com>
8 days agofix typo in log msg (#9447)
Xiaobing [Thu, 22 Sep 2022 18:05:40 +0000 (11:05 -0700)] 
fix typo in log msg (#9447)

9 days agoAdd support for gracefully handling the errors while transformations (#9377)
Kartik Khare [Thu, 22 Sep 2022 11:43:58 +0000 (17:13 +0530)] 
Add support for gracefully handling the errors while transformations (#9377)

* Add support for gracefully handling the errors while transformations

* Maintain consistent checks across transformers

* Add incompleteRowCount metric to realtime servers

Co-authored-by: Kartik Khare <kharekartik@Kartiks-MacBook-Pro.local>
9 days agoAdd support for spaces in file paths (#9426)
Kartik Khare [Thu, 22 Sep 2022 07:13:37 +0000 (12:43 +0530)] 
Add support for spaces in file paths (#9426)

Co-authored-by: Kartik Khare <kharekartik@Kartiks-MacBook-Pro.local>
9 days agouse MinionEventObserver to track finer grained task progress status on worker (#9432)
Xiaobing [Wed, 21 Sep 2022 23:44:35 +0000 (16:44 -0700)] 
use MinionEventObserver to track finer grained task progress status on worker (#9432)

* use MinionEventObserver to track finer grained task progress status in memory

* UTs and refine

* cr and more UTs

9 days agofix(ui): table only displaying first 10 results when not explicitly paginated (#9404)
Jayesh Choudhary [Wed, 21 Sep 2022 22:28:15 +0000 (03:58 +0530)] 
fix(ui): table only displaying first 10 results when not explicitly paginated (#9404)

* fix(ui): table only displaying first 10 results when not explicitly paginated

* show pagination by default when the table rows are greater than 10

9 days agomake IndexHandler methods reentrant (#9440)
Xiaobing [Wed, 21 Sep 2022 21:05:40 +0000 (14:05 -0700)] 
make IndexHandler methods reentrant (#9440)

9 days agoAdaptive Server Selection (#9311)
Vivek Iyer Vaidyanathan [Wed, 21 Sep 2022 18:48:49 +0000 (11:48 -0700)] 
Adaptive Server Selection (#9311)

* Implementation for AdaptiveServerSelection feature

* Add tests for Adaptive Server Selection feature

* Fix flaky unit test

* Address review comments

* Fix test and indentation.

9 days agoAlways pass clientId to kafka's consumer properties (#9444)
Navina Ramesh [Wed, 21 Sep 2022 18:04:33 +0000 (23:34 +0530)] 
Always pass clientId to kafka's consumer properties (#9444)

10 days agoFix the bug of hybrid table request using the same request id (#9443)
Xiaotian (Jackie) Jiang [Wed, 21 Sep 2022 08:00:26 +0000 (01:00 -0700)] 
Fix the bug of hybrid table request using the same request id (#9443)

10 days ago[multistage] make leaf stage run query optimizer (#9439)
Rong Rong [Wed, 21 Sep 2022 02:04:41 +0000 (19:04 -0700)] 
[multistage] make leaf stage run query optimizer (#9439)

* add table info to multistage leave engine
* fix helix manager not started issue by not acquiring property store until start.

Co-authored-by: Rong Rong <rongr@startree.ai>
10 days agoMake minion task metadata manager methods more generic (#9436)
Saurabh Dubey [Tue, 20 Sep 2022 20:54:37 +0000 (02:24 +0530)] 
Make minion task metadata manager methods more generic (#9436)

Co-authored-by: Saurabh Dubey <saurabh.dubey@Saurabhs-MacBook-Pro.local>
10 days ago[multistage] support NULL in data blocks (#9427)
Rong Rong [Tue, 20 Sep 2022 19:52:06 +0000 (12:52 -0700)] 
[multistage] support NULL in data blocks (#9427)

* initial commit for supporting null in datablock

* add test according to comment

Co-authored-by: Rong Rong <rongr@startree.ai>
10 days agoExtract the common logic for upsert metadata manager (#9435)
Xiaotian (Jackie) Jiang [Tue, 20 Sep 2022 19:44:27 +0000 (12:44 -0700)] 
Extract the common logic for upsert metadata manager (#9435)

10 days agoFix skip segment logic in MinMaxValueBasedSelectionOrderByCombineOperator (#9434)
Xiaotian (Jackie) Jiang [Tue, 20 Sep 2022 17:24:28 +0000 (10:24 -0700)] 
Fix skip segment logic in MinMaxValueBasedSelectionOrderByCombineOperator (#9434)

11 days agoDo not implicit cast for BOOLEAN and TIMESTAMP (#9385)
Xiaotian (Jackie) Jiang [Mon, 19 Sep 2022 23:14:41 +0000 (16:14 -0700)] 
Do not implicit cast for BOOLEAN and TIMESTAMP (#9385)

11 days agoCleanup null value handling in data table (#9429)
Xiaotian (Jackie) Jiang [Mon, 19 Sep 2022 23:12:21 +0000 (16:12 -0700)] 
Cleanup null value handling in data table (#9429)

11 days agoFix the flaky NullHandlingIntegrationTest (#9430)
Xiaotian (Jackie) Jiang [Mon, 19 Sep 2022 21:46:19 +0000 (14:46 -0700)] 
Fix the flaky NullHandlingIntegrationTest (#9430)

11 days agoBug fix: failure status in ingestion jobs doesn't get reflect in exitCode (#9410)
Kartik Khare [Mon, 19 Sep 2022 20:17:29 +0000 (01:47 +0530)] 
Bug fix: failure status in ingestion jobs doesn't get reflect in exitCode (#9410)

11 days agoFix thread safety issues with SDF (#9425)
Saurabh Dubey [Mon, 19 Sep 2022 20:15:07 +0000 (01:45 +0530)] 
Fix thread safety issues with SDF (#9425)

11 days agofix histogram null pointer exception (#9428)
Jia Guo [Mon, 19 Sep 2022 20:13:00 +0000 (13:13 -0700)] 
fix histogram null pointer exception (#9428)

2 weeks ago[multistage] test using BaseClusterIntegrationTestSet (#9412)
Rong Rong [Fri, 16 Sep 2022 22:47:02 +0000 (15:47 -0700)] 
[multistage] test using BaseClusterIntegrationTestSet (#9412)

by running the hard-coded queries list first

- make relRoot info available in query plan
- uses fields / collation / hints properly before returning results
- fix SQL conformance
- some other bug fixes

this PR also listed out TODOs for features we don't support right now.

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks agoAdd missing properties in CSV plugin (#9399) 9422/head
Kartik Khare [Fri, 16 Sep 2022 09:51:01 +0000 (15:21 +0530)] 
Add missing properties in CSV plugin (#9399)

* Add missing properties in CSV plugin

* Fix linting

Co-authored-by: Kartik Khare <kharekartik@Kartiks-MacBook-Pro.local>
2 weeks agoUse ISODateTimeFormat as default for SIMPLE_DATE_FORMAT (#9378)
Kartik Khare [Fri, 16 Sep 2022 09:45:52 +0000 (15:15 +0530)] 
Use ISODateTimeFormat as default for SIMPLE_DATE_FORMAT  (#9378)

* Use ISODateTimeFormat as default for SIMPLE_DATE_FORMAT

* Fix linting

* Fix tests

* Add default MILLISECONDS for EPOCH in pipe format

* Reduce log level

Co-authored-by: Kartik Khare <kharekartik@Kartiks-MacBook-Pro.local>
2 weeks ago[multistage] Initial commit to support h2 testing (#9406)
Rong Rong [Fri, 16 Sep 2022 02:10:14 +0000 (19:10 -0700)] 
[multistage] Initial commit to support h2 testing (#9406)

* Initial commit to support h2 testing

- adding H2 connection to load the same dataset
- verify the final result against entire query resultset

* - fix typo in test case
- make it stable by sending ts into order by as well

* also fix operator utils not working issue

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks agoBenchmark thread cpu time (#9408)
Jia Guo [Fri, 16 Sep 2022 01:33:06 +0000 (18:33 -0700)] 
Benchmark thread cpu time (#9408)

* Benchmark ThreadMXBean CPU Time

* Benchmark ThreadMXBean CPU Time

* Benchmark ThreadMXBean CPU Time

2 weeks ago[Bugfix] schema update bug fix (#9382)
MeihanLi [Fri, 16 Sep 2022 01:25:21 +0000 (18:25 -0700)] 
[Bugfix] schema update bug fix (#9382)

2 weeks agoset MDC so that one can route minion task logs to separate files cleanly (#9400)
Xiaobing [Thu, 15 Sep 2022 22:19:59 +0000 (15:19 -0700)] 
set MDC so that one can route minion task logs to separate files cleanly (#9400)

2 weeks agoDisable logging for interrupted exceptions in kinesis (#9405)
Kartik Khare [Thu, 15 Sep 2022 20:43:50 +0000 (02:13 +0530)] 
Disable logging for interrupted exceptions in kinesis (#9405)

2 weeks ago[multistage] Support IN and NOT-IN Clauses (#9374)
Ankit Sultana [Thu, 15 Sep 2022 01:56:28 +0000 (07:26 +0530)] 
[multistage] Support IN and NOT-IN Clauses (#9374)

* [multistage] Support IN Clause With 1 Argument

* Working in/not-in
* Refactor RexExpressionUtils after rebasing with master

2 weeks ago[Feature] Support Coalesce for Column Names (#9327)
Yao Liu [Wed, 14 Sep 2022 23:50:13 +0000 (16:50 -0700)] 
[Feature] Support Coalesce for Column Names (#9327)

2 weeks ago[Feature] Support IsDistinctFrom and IsNotDistinctFrom (#9312)
Yao Liu [Wed, 14 Sep 2022 22:53:49 +0000 (15:53 -0700)] 
[Feature]  Support IsDistinctFrom and IsNotDistinctFrom (#9312)

The operators IsDistinctFrom and IsNotDistinctFrom only supports column names as argument for now.
When null option is disabled, the row is considered as not null by default.
Expected value:
`Null is IsDistinctFrom ValueA`: True
`Null is IsDistinctFrom Null`: False
`ValueA is IsDistinctFrom ValueB`: `NotEquals(ValueA, ValueB)`
`Null is IsNotDistinctFrom ValueA`: False
`Null is IsNotDistinctFrom Null`: True
`ValueA is IsNotDistinctFrom ValueB`: Equals(ValueA, ValueB)`

Example Usage:
`ColumnA IsDistinctFrom ColumnB`
`ColumnA IsNotDistinctFrom ColumnB`

2 weeks agoReduce the log in tests (#9398)
Xiaotian (Jackie) Jiang [Wed, 14 Sep 2022 21:33:46 +0000 (14:33 -0700)] 
Reduce the log in tests (#9398)

2 weeks agoadding routing entry checker for null tables (#9396)
Rong Rong [Wed, 14 Sep 2022 05:21:35 +0000 (22:21 -0700)] 
adding routing entry checker for null tables (#9396)

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks ago[multistage] add cast function (#9384)
Rong Rong [Tue, 13 Sep 2022 18:42:17 +0000 (11:42 -0700)] 
[multistage] add cast function (#9384)

* add CAST by using the DataConversion func

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks ago[multistage][bug-fix] Fix the join condition order different from join table order...
Yao Liu [Tue, 13 Sep 2022 17:42:49 +0000 (10:42 -0700)] 
[multistage][bug-fix] Fix the join condition order different from join table order (#9350)

* bug fix for join order
* use join info to get join keys
* remove util func

2 weeks agosupport SEMI join (#9367)
Rong Rong [Tue, 13 Sep 2022 08:21:51 +0000 (01:21 -0700)] 
support SEMI join (#9367)

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks ago[multistage][bugfix] fix group-by without agg call plan failure (#9383)
Rong Rong [Tue, 13 Sep 2022 08:09:55 +0000 (01:09 -0700)] 
[multistage][bugfix] fix group-by without agg call plan failure (#9383)

* add test

* add rel rule to allow distinct agg

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks agoTerminate the query after plan generation if timeout (#9386)
Jia Guo [Tue, 13 Sep 2022 02:29:21 +0000 (19:29 -0700)] 
Terminate the query after plan generation if timeout (#9386)

* Terminate the query after plan generation if timeout

* Use TimeoutException

* Use TimeoutException

* Update error message

2 weeks agoOptimize combine operator to fully utilize threads (#9387)
Xiaotian (Jackie) Jiang [Tue, 13 Sep 2022 01:43:29 +0000 (18:43 -0700)] 
Optimize combine operator to fully utilize threads (#9387)

2 weeks agoDeprecate instanceId Config For Broker/Minion Specific Configs (#9308)
Ankit Sultana [Tue, 13 Sep 2022 00:54:31 +0000 (06:24 +0530)] 
Deprecate instanceId Config For Broker/Minion Specific Configs (#9308)

2 weeks agoEnhance `and` filter predicate evaluation efficiency (#9336)
Jia Guo [Mon, 12 Sep 2022 21:51:59 +0000 (14:51 -0700)] 
Enhance `and` filter predicate evaluation efficiency   (#9336)

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

* Improve the efficiency of and evaluation

2 weeks agoskip late cron job with max allowed delay (#9372)
Xiaobing [Mon, 12 Sep 2022 17:57:27 +0000 (10:57 -0700)] 
skip late cron job with max allowed delay (#9372)

2 weeks ago[multistage] add calcite function catalog (#9375)
Rong Rong [Mon, 12 Sep 2022 17:04:19 +0000 (10:04 -0700)] 
[multistage] add calcite function catalog (#9375)

* planner can parse custom function
* use chained operator table
also
* fix typo in partition carrying
* fix rules in singleton exchange optimization.

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks agoFix log to reflect job type (#9381)
Kartik Khare [Mon, 12 Sep 2022 10:56:25 +0000 (16:26 +0530)] 
Fix log to reflect job type (#9381)

2 weeks ago[multistage] hybrid routing support (#9379)
Rong Rong [Sun, 11 Sep 2022 22:28:41 +0000 (15:28 -0700)] 
[multistage] hybrid routing support (#9379)

Preliminary support for hybrid routing on the multistage engine.

It copies some of the logic in the broker request side but not all. and only has basic testing.
- copied some time boundary logic (manager and associated logic) into either the core module or replicated in the multi-stage planner
- created a hybrid routing dispatchable StageMetadata
  - indexed each segment with table types
  - added time boundary info if dispatching multiple table types, otherwise ignore
  - reconstructed server requests multiple times.

Co-authored-by: Rong Rong <rongr@startree.ai>
2 weeks agoAllow expression transformer cotinue on error (#9376)
Xiang Fu [Sun, 11 Sep 2022 03:58:57 +0000 (20:58 -0700)] 
Allow expression transformer cotinue on error (#9376)

2 weeks agoadd a test for a join b using(col) (#9373)
Yao Liu [Sat, 10 Sep 2022 14:47:35 +0000 (07:47 -0700)] 
add a test for a join b using(col) (#9373)

3 weeks agoImprove segment name check in metadata push (#9359)
Haitao Zhang [Sat, 10 Sep 2022 10:29:45 +0000 (03:29 -0700)] 
Improve segment name check in metadata push (#9359)

* Improve segment name check in metadata push

* address comments

* fix broken tests

3 weeks ago[Feature][multistage] Thread-safe query planning (#9344)
Yao Liu [Sat, 10 Sep 2022 04:04:55 +0000 (21:04 -0700)] 
[Feature][multistage] Thread-safe query planning (#9344)

* multi-thread query planning

* Use auto-close planner context and fix test

* address styple comments

* address javadoc

* multi-thread query planning

* Use auto-close planner context and fix test

* address styple comments

* address javadoc

3 weeks agoadd test (#9357)
Rong Rong [Sat, 10 Sep 2022 00:05:40 +0000 (17:05 -0700)] 
add test (#9357)

Co-authored-by: Rong Rong <rongr@startree.ai>
3 weeks agoAdding more test cases for parquet files (#9352)
Xiang Fu [Fri, 9 Sep 2022 23:07:39 +0000 (16:07 -0700)] 
Adding more test cases for parquet files (#9352)

3 weeks ago[Minor] Add the missing formatter on in RealtimeNonReplicaGroupSegmentAssignmentTest...
Xiaotian (Jackie) Jiang [Fri, 9 Sep 2022 21:23:12 +0000 (14:23 -0700)] 
[Minor] Add the missing formatter on in RealtimeNonReplicaGroupSegmentAssignmentTest (#9363)

3 weeks agoBloomFilterRule should only recommend for supported column type (#9364)
Benson Yuan [Fri, 9 Sep 2022 21:21:14 +0000 (14:21 -0700)] 
BloomFilterRule should only recommend for supported column type (#9364)

3 weeks agofixing the typo in pinot-java-client-jdk8 pom (#9361)
Xiang Fu [Fri, 9 Sep 2022 20:37:45 +0000 (13:37 -0700)] 
fixing the typo in pinot-java-client-jdk8 pom (#9361)