arrow-ballista.git
23 hours agoGenerate into source folder to make IDEs happy (#247) master
Brent Gardner [Fri, 23 Sep 2022 18:56:32 +0000 (11:56 -0700)] 
Generate into source folder to make IDEs happy (#247)

26 hours agoAdd list of jobs to scheduler UI (#241)
Andy Grove [Fri, 23 Sep 2022 16:22:35 +0000 (10:22 -0600)] 
Add list of jobs to scheduler UI (#241)

26 hours agoUpdate release instructions (#237)
Andy Grove [Fri, 23 Sep 2022 16:22:02 +0000 (10:22 -0600)] 
Update release instructions (#237)

* update release instructions

* clippy

* clippy

41 hours agoAdd user guide for Flight SQL (#264)
Brent Gardner [Fri, 23 Sep 2022 00:43:17 +0000 (17:43 -0700)] 
Add user guide for Flight SQL (#264)

2 days agoAdd LaunchMultiTask rpc interface for executor (#255)
yahoNanJing [Thu, 22 Sep 2022 01:21:16 +0000 (09:21 +0800)] 
Add LaunchMultiTask rpc interface for executor (#255)

Co-authored-by: yangzhong <yangzhong@ebay.com>
2 days agoAdd a docker-compose file and dockerfiles (#249)
Brent Gardner [Wed, 21 Sep 2022 19:42:53 +0000 (12:42 -0700)] 
Add a docker-compose file and dockerfiles (#249)

4 days agoFix Scheduler UI in Docker image (#251)
Andy Grove [Tue, 20 Sep 2022 17:15:32 +0000 (11:15 -0600)] 
Fix Scheduler UI in Docker image (#251)

5 days agoAdd scheduler UI to ballista-scheduler Docker image (#238)
Andy Grove [Mon, 19 Sep 2022 14:52:49 +0000 (08:52 -0600)] 
Add scheduler UI to ballista-scheduler Docker image (#238)

5 days agoUse bind host rather than the external host for starting a local executor service...
yahoNanJing [Mon, 19 Sep 2022 13:32:49 +0000 (21:32 +0800)] 
Use bind host rather than the external host for starting a local executor service (#245)

Co-authored-by: yangzhong <yangzhong@ebay.com>
7 days agoFix some issues found when testing rc1 (#233) 0.8.0 0.8.0-rc2
Andy Grove [Fri, 16 Sep 2022 19:54:27 +0000 (13:54 -0600)] 
Fix some issues found when testing rc1 (#233)

7 days agoFix integration tests (#232) 0.8.0-rc1
Andy Grove [Fri, 16 Sep 2022 19:12:35 +0000 (13:12 -0600)] 
Fix integration tests (#232)

* Fix integration tests

* add variable for protoc version, fix script

7 days agoPrepare for 0.8.0 release (#183)
Andy Grove [Fri, 16 Sep 2022 18:47:14 +0000 (12:47 -0600)] 
Prepare for 0.8.0 release (#183)

8 days agoMINOR: Increase default shuffle.partitions to 16 (#222)
Andy Grove [Fri, 16 Sep 2022 17:10:36 +0000 (11:10 -0600)] 
MINOR: Increase default shuffle.partitions to 16 (#222)

* Increase default shuffle.partitions to 16

* prettier

8 days agoMake executor logging less verbose (#212)
Andy Grove [Fri, 16 Sep 2022 17:10:13 +0000 (11:10 -0600)] 
Make executor logging less verbose (#212)

* make shuffle reader exec less verbose

* change some executor logging from info to debug

* fix imports

8 days agouse datafusion 12.0.0 (#223)
Andy Grove [Fri, 16 Sep 2022 13:24:28 +0000 (07:24 -0600)] 
use datafusion 12.0.0 (#223)

8 days agoPropagate Ballista configs to DataFusion context in scheduler (#220)
Andy Grove [Fri, 16 Sep 2022 13:23:51 +0000 (07:23 -0600)] 
Propagate Ballista configs to DataFusion context in scheduler (#220)

9 days agoExecutor uses all available cores by default + add tuning guide (#202)
Andy Grove [Thu, 15 Sep 2022 16:03:32 +0000 (10:03 -0600)] 
Executor uses all available cores by default + add tuning guide (#202)

9 days agoFix scheduler_name format for executor report task status in push-staged mode (#217)
r.4ntix [Thu, 15 Sep 2022 13:53:07 +0000 (21:53 +0800)] 
Fix scheduler_name format for executor report task status in push-staged mode (#217)

9 days agoUpdate config spec, make the values of optional parameters more explicit (#216)
r.4ntix [Thu, 15 Sep 2022 13:49:12 +0000 (21:49 +0800)] 
Update config spec, make the values of optional parameters more explicit (#216)

9 days agoAdd all binary ops in serde (#210)
Andy Grove [Thu, 15 Sep 2022 12:28:48 +0000 (06:28 -0600)] 
Add all binary ops in serde (#210)

10 days agoAllow some config options to be set when creating context (#204)
Andy Grove [Tue, 13 Sep 2022 23:49:28 +0000 (17:49 -0600)] 
Allow some config options to be set when creating context (#204)

11 days agoUse DataFusion 12.0.0-rc1 (and add support for DateTimeIntervalExpr and more binary...
Andy Grove [Mon, 12 Sep 2022 21:36:01 +0000 (15:36 -0600)] 
Use DataFusion 12.0.0-rc1 (and add support for DateTimeIntervalExpr and more binary operators) (#200)

* Add serde support for DateTimeInterval

* Use try_into for schema and types

* bump versions again

* use 12.0.0-rc1

* Update Python module

13 days agobump datafusion version (#199)
Andy Grove [Sun, 11 Sep 2022 05:22:17 +0000 (23:22 -0600)] 
bump datafusion version (#199)

2 weeks agoExecutor lost handling (#184)
mingmwang [Fri, 9 Sep 2022 00:14:15 +0000 (08:14 +0800)] 
Executor lost handling (#184)

* Executor Lost Handling

* add dead executor sets to ExecutorManager

* add UT for reset_stages in ExecutionGraph

* add ExecutorLost to QueryStageSchedulerEvent

* Fix rollback ResolvedStage

* Tiny fix

* Resolve review comments, add more UT

2 weeks ago[MINOR] Add log info in stdout (#187)
Yang Jiang [Thu, 8 Sep 2022 03:00:23 +0000 (11:00 +0800)] 
[MINOR] Add log info in stdout (#187)

* add log info in stdout

* make log print to stdout deafult

2 weeks agoFix CI to prevent future breaking of FlightSql (#191)
Brent Gardner [Tue, 6 Sep 2022 16:45:41 +0000 (09:45 -0700)] 
Fix CI to prevent future breaking of FlightSql (#191)

* Fix CI to prevent future breaking of FlightSql

* Update API to match breaking arrow changes

3 weeks agoUpdate task status to the its job curator scheduler (#181)
yahoNanJing [Thu, 1 Sep 2022 16:37:59 +0000 (00:37 +0800)] 
Update task status to the its job curator scheduler (#181)

Co-authored-by: yangzhong <yangzhong@ebay.com>
3 weeks agoRemove revive offer event loop (#156)
yahoNanJing [Thu, 1 Sep 2022 12:59:39 +0000 (20:59 +0800)] 
Remove revive offer event loop (#156)

* Add TaskUpdating event for QueryStageSchedulerEvent

* Remove revive offer event loop

Co-authored-by: yangzhong <yangzhong@ebay.com>
3 weeks agoUse latest DataFusion, and arrow 21.0.0 (#182)
Andy Grove [Wed, 31 Aug 2022 15:03:04 +0000 (09:03 -0600)] 
Use latest DataFusion, and arrow 21.0.0 (#182)

3 weeks agoMINOR: Remove legacy code from CLI (#168)
Andy Grove [Tue, 30 Aug 2022 16:20:51 +0000 (10:20 -0600)] 
MINOR: Remove legacy code from CLI (#168)

3 weeks agoupdate latest datafusion. (#175)
Yang Jiang [Tue, 30 Aug 2022 16:20:31 +0000 (00:20 +0800)] 
update latest datafusion. (#175)

3 weeks agoAdd basic Python docs and enable information_schema in Python context (#170)
Andy Grove [Tue, 30 Aug 2022 02:38:02 +0000 (20:38 -0600)] 
Add basic Python docs and enable information_schema in Python context (#170)

3 weeks agoAdd tokio::signal to executor Cargo.toml (#172)
Trent Feda [Mon, 29 Aug 2022 19:15:06 +0000 (15:15 -0400)] 
Add tokio::signal to executor Cargo.toml (#172)

4 weeks agoFix docker build by updating Rust version to 1.63 (#164)
Andy Grove [Sat, 27 Aug 2022 17:59:29 +0000 (11:59 -0600)] 
Fix docker build by updating Rust version to 1.63 (#164)

4 weeks agoMake the user guide about Ballista, not DataFusion (#161)
Andy Grove [Sat, 27 Aug 2022 17:58:33 +0000 (11:58 -0600)] 
Make the user guide about Ballista, not DataFusion (#161)

* Remove Raspberry Pi docs

* Delete DataFusion logos

* User guide is now specific to Ballista

* prettier

4 weeks agoIntroduce CuratorTaskManager for make an active job be curated by only one scheduler...
yahoNanJing [Sat, 27 Aug 2022 17:00:01 +0000 (01:00 +0800)] 
Introduce CuratorTaskManager for make an active job be curated by only one scheduler (#153)

* Add scheduler name for scheduler identification

* Introduce scheduler id for execution graph as its curator

* Introduce state machine for the execution stage

* Introduce stage change event to execution graph

* Introduce cache for active execution graphs in the curator scheduler

* Fix PR review

Co-authored-by: yangzhong <yangzhong@ebay.com>
4 weeks ago[Python] Fix Ballista Python bindings (#157)
Andy Grove [Fri, 26 Aug 2022 16:52:25 +0000 (10:52 -0600)] 
[Python] Fix Ballista Python bindings  (#157)

4 weeks agoAdd job cancellation (#146)
Dan Harris [Thu, 25 Aug 2022 17:52:27 +0000 (13:52 -0400)] 
Add job cancellation (#146)

* Add job cancellation

* Use util method to create grpc connection

4 weeks agoUpdate ahash requirement from 0.7 to 0.8 (#125)
dependabot[bot] [Wed, 24 Aug 2022 12:31:52 +0000 (06:31 -0600)] 
Update ahash requirement from 0.7 to 0.8 (#125)

4 weeks agouse info instead print (#154)
Yang Jiang [Wed, 24 Aug 2022 12:30:08 +0000 (20:30 +0800)] 
use info instead print (#154)

4 weeks agoMove ExecutionGraph encoding and decoding logic into execution_graph for better encap...
yahoNanJing [Mon, 22 Aug 2022 14:02:34 +0000 (22:02 +0800)] 
Move ExecutionGraph encoding and decoding logic into execution_graph for better encapsulation (#150)

Co-authored-by: yangzhong <yangzhong@ebay.com>
4 weeks agoStop Executor Impl, Executor Graceful Shutdown (#151)
mingmwang [Mon, 22 Aug 2022 04:33:08 +0000 (12:33 +0800)] 
Stop Executor Impl, Executor Graceful Shutdown (#151)

5 weeks agoIgnore the previous job_id inside fill_reservations() (#141)
yahoNanJing [Wed, 17 Aug 2022 14:38:20 +0000 (22:38 +0800)] 
Ignore the previous job_id inside fill_reservations() (#141)

* Normalize the serialization and deserialization places of protobuf structs

* Use a vector to indicate the output links for ExecutionStage

* Ignore the previous job_id inside fill_reservations()

Co-authored-by: yangzhong <yangzhong@ebay.com>
5 weeks agoSpawn a thread for execution plan generation (#135)
yahoNanJing [Wed, 17 Aug 2022 01:10:48 +0000 (09:10 +0800)] 
Spawn a thread for execution plan generation (#135)

* Remove session id in QueryStageSchedulerEvent::JobQueued

* Refine on_receive interface of EventAction

* Spawn a thread for execution plan generation

Co-authored-by: yangzhong <yangzhong@ebay.com>
5 weeks agoBallista Executor report plan/operators metrics to Ballista Scheduler (#124)
mingmwang [Tue, 16 Aug 2022 21:54:05 +0000 (05:54 +0800)] 
Ballista Executor report plan/operators metrics to Ballista Scheduler  (#124)

* Add timeout and keep-alive settings for Grpc Client

* Add timeout and keep-alive settings for Grpc Server

* move server settings to utils

* fix fmt

* set tcp_nodelay to true explicitly

* Ballista Executor report plan/operators metrics to Ballista Scheduler

* Resolve review comments

* Fix plan display with metrics

* fix shuffle writer metrics collection

* Remove Keyspace::QueuedJobs (#134)

* Remove Keyspace::QueuedJobs

* Fix UT

* Fix cargo clippy for rust 1.63

Co-authored-by: yangzhong <yangzhong@ebay.com>
Co-authored-by: yahoNanJing <90197956+yahoNanJing@users.noreply.github.com>
Co-authored-by: yangzhong <yangzhong@ebay.com>
5 weeks agoRemove Keyspace::QueuedJobs (#134)
yahoNanJing [Mon, 15 Aug 2022 06:16:07 +0000 (14:16 +0800)] 
Remove Keyspace::QueuedJobs (#134)

* Remove Keyspace::QueuedJobs

* Fix UT

* Fix cargo clippy for rust 1.63

Co-authored-by: yangzhong <yangzhong@ebay.com>
6 weeks agoUsing tokio tracing for log file (#123)
Yang Jiang [Thu, 11 Aug 2022 04:40:28 +0000 (12:40 +0800)] 
Using tokio tracing for log file (#123)

6 weeks agoAdd timeout and keep-alive settings for Grpc Client and Server (#115)
mingmwang [Mon, 8 Aug 2022 13:06:48 +0000 (21:06 +0800)] 
Add timeout and keep-alive settings for Grpc Client and Server (#115)

* Add timeout and keep-alive settings for Grpc Client

* Add timeout and keep-alive settings for Grpc Server

* move server settings to utils

* fix fmt

* set tcp_nodelay to true explicitly

8 weeks agoUse another channel to update the status of a task set for executor (#104)
Yang Jiang [Thu, 28 Jul 2022 22:08:21 +0000 (06:08 +0800)] 
Use another channel to update the status of a task set for executor  (#104)

* Use another channel to update the status of a task set for executor

* fix clippy

8 weeks agoAdd config for log level in ballista (#103)
Yang Jiang [Thu, 28 Jul 2022 21:48:19 +0000 (05:48 +0800)] 
Add config for log level in ballista (#103)

8 weeks agoAdd config for concurrent_task in executo (#95)
Yang Jiang [Wed, 27 Jul 2022 08:02:15 +0000 (16:02 +0800)] 
Add config for concurrent_task in executo (#95)

8 weeks agoAdd FlightSQL support (#93)
Brent Gardner [Wed, 27 Jul 2022 08:01:50 +0000 (02:01 -0600)] 
Add FlightSQL support (#93)

8 weeks agoadd the `ballista-cli` to member of the workspace (#89)
Kun Liu [Wed, 27 Jul 2022 03:55:11 +0000 (11:55 +0800)] 
add the `ballista-cli` to member of the workspace (#89)

2 months agoSupport for multi-scheduler deployments (#59)
Dan Harris [Sat, 16 Jul 2022 15:47:40 +0000 (11:47 -0400)] 
Support for multi-scheduler deployments (#59)

* Initial design and implementation

* ExecutorManager tests

* Only consider alive executors

* Use correct session ID provided in request

* Fix bug in etcd key scan

* Debugging

* Drop for EtcdLock

* Better locking

* Debug for ExecutionGraph

* Fix partition accounting in ExecutionGraph

* Fix input partition accounting

* Handle stages with multiple inputs better

* Simplify output buffer

* Cleanup

* Cleanup

* Linting

* Linting and docs

* Job queueing and general cleanup

* Handle job queueing and failure

* Tests

* Fix doc comments

* Tests

* Add license header

* Fix graph complete logic

* Fix bug in partition mapping

* Eagerly offer pending tasks

* Tests for event loop

* Merge upstream

* Fix compiler error after rebase

* Clippy fix

* Merge pull request #4 from coralogix/scheduler-fix

Scheduler fixes

* Use correct bind address for executor registration

* Use correct keyspace when initing heartbeats

* Fix after cherry-pick bugfixes

* Fix conflicts after merge

Co-authored-by: Martins Purins <martins.purins@coralogix.com>
2 months agoUse latest DataFusion (#86)
yahoNanJing [Wed, 13 Jul 2022 19:30:52 +0000 (03:30 +0800)] 
Use latest DataFusion (#86)

* Update datafusion dependency to commit d0d5564b8f689a01e542b8c1df829d74d0fab2b0

* Fix inconsistency

* Use latest DataFusion

* Fix tomlfmt

* Fix PR review

Co-authored-by: yangzhong <yangzhong@ebay.com>
2 months agoSupport sled path in config file (#80)
Yang Jiang [Wed, 6 Jul 2022 06:56:42 +0000 (14:56 +0800)] 
Support sled path in config file (#80)

3 months agoUse latest DataFusion (#68)
Andy Grove [Sun, 19 Jun 2022 13:55:05 +0000 (07:55 -0600)] 
Use latest DataFusion (#68)

3 months agoreturn task status on panic (#67)
Remco Verhoef [Tue, 14 Jun 2022 14:43:44 +0000 (16:43 +0200)] 
return task status on panic (#67)

3 months agoUpdate the K8s deployment docs (#65)
Xin Hao [Sun, 12 Jun 2022 18:25:03 +0000 (02:25 +0800)] 
Update the K8s deployment docs (#65)

3 months agoRevert "remove python (#1518)" (#61)
Andy Grove [Wed, 8 Jun 2022 16:20:30 +0000 (10:20 -0600)] 
Revert "remove python (#1518)" (#61)

* Revert "remove python (#1518)"

This reverts commit bac97fa04ad8e9def814507b845d64038687f4a2.

* fix cargo.toml lint issues and exclude from workspace

* use original datafusion dependency

* fix build

* lint

3 months agoDelegate to `datafusion-proto` for logical plan serde (#57)
Andy Grove [Sat, 4 Jun 2022 15:18:12 +0000 (09:18 -0600)] 
Delegate to `datafusion-proto` for logical plan serde (#57)

3 months agoUse latest DataFusion version (#56)
Andy Grove [Sat, 4 Jun 2022 00:03:36 +0000 (18:03 -0600)] 
Use latest DataFusion version (#56)

3 months agoUpdate snmalloc-rs requirement from 0.2 to 0.3 (#55)
dependabot[bot] [Fri, 3 Jun 2022 13:06:15 +0000 (07:06 -0600)] 
Update snmalloc-rs requirement from 0.2 to 0.3 (#55)

Updates the requirements on [snmalloc-rs](https://github.com/SchrodingerZhu/snmalloc-rs) to permit the latest version.
- [Release notes](https://github.com/SchrodingerZhu/snmalloc-rs/releases)
- [Changelog](https://github.com/SchrodingerZhu/snmalloc-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/SchrodingerZhu/snmalloc-rs/commits)

---
updated-dependencies:
- dependency-name: snmalloc-rs
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
3 months agoRemove ObjectStore from FileScanConfig and ListingTableConfig (#53)
Raphael Taylor-Davies [Thu, 2 Jun 2022 13:11:05 +0000 (14:11 +0100)] 
Remove ObjectStore from FileScanConfig and ListingTableConfig (#53)

* Remove ObjectStore from FileScanConfig and ListingTableConfig

* Update datafusion pin

3 months agoBump eventsource from 1.0.7 to 1.1.1 in /ballista/ui/scheduler (#54)
dependabot[bot] [Thu, 2 Jun 2022 13:08:39 +0000 (07:08 -0600)] 
Bump eventsource from 1.0.7 to 1.1.1 in /ballista/ui/scheduler (#54)

3 months agoUpdate to arrow 15 (#49)
Raphael Taylor-Davies [Tue, 31 May 2022 13:00:33 +0000 (14:00 +0100)] 
Update to arrow 15 (#49)

* Update to arrow 15

* Update datafusion pin

* Update datafusion pin

3 months agoUpdate to apache/arrow-datafusion#2578 (#48)
Raphael Taylor-Davies [Tue, 31 May 2022 11:35:20 +0000 (12:35 +0100)] 
Update to apache/arrow-datafusion#2578 (#48)

* Update to https://github.com/apache/arrow-datafusion/pull/2578

* Fix standalone build

* Update datafusion pin

3 months agoMINOR: Improve developer docs (#41)
Andy Grove [Mon, 30 May 2022 15:35:22 +0000 (09:35 -0600)] 
MINOR: Improve developer docs (#41)

3 months agoFilter field for `JoinNode` and `HashJoinExecNode` (#36)
Eduard Karacharov [Sat, 28 May 2022 20:51:44 +0000 (23:51 +0300)] 
Filter field for `JoinNode` and  `HashJoinExecNode` (#36)

* bump datafuion rev (#35)

* support join filter in ballista serde

* dafafusion rev

* another datafusion update

* bump datafusion rev

Co-authored-by: Andy Grove <andygrove73@gmail.com>
3 months agoremove duplicate readme (#45)
Yang Jiang [Sat, 28 May 2022 20:50:44 +0000 (04:50 +0800)] 
remove duplicate readme (#45)

3 months agofix integration tests (#43)
Andy Grove [Sat, 28 May 2022 12:30:08 +0000 (06:30 -0600)] 
fix integration tests (#43)

3 months agoUse datafusion arrow re-export (#44)
Raphael Taylor-Davies [Fri, 27 May 2022 12:36:04 +0000 (13:36 +0100)] 
Use datafusion arrow re-export (#44)

4 months agoUpdate with file format breaking change (#40)
Raphael Taylor-Davies [Tue, 24 May 2022 18:44:28 +0000 (19:44 +0100)] 
Update with file format breaking change (#40)

4 months agoMINOR: update homepage and repository urls (#37)
Andy Grove [Tue, 24 May 2022 15:46:57 +0000 (09:46 -0600)] 
MINOR: update homepage and repository urls (#37)

* update homepage and repository urls

* rename benchmark crate

4 months agoMINOR: Improve the examples (#34)
Andy Grove [Tue, 24 May 2022 04:27:30 +0000 (22:27 -0600)] 
MINOR: Improve the examples (#34)

4 months agobump datafuion rev (#35)
Andy Grove [Sun, 22 May 2022 17:33:10 +0000 (11:33 -0600)] 
bump datafuion rev (#35)

4 months agoImprove top-level README (#33)
Andy Grove [Sat, 21 May 2022 21:03:29 +0000 (15:03 -0600)] 
Improve top-level README (#33)

4 months agouse datafusion rev cb84504fed4e613c9ed18c4e2a2022c701add2d9 (#31)
Andy Grove [Sat, 21 May 2022 15:18:58 +0000 (09:18 -0600)] 
use datafusion rev cb84504fed4e613c9ed18c4e2a2022c701add2d9 (#31)

4 months agoupdate project description (#29)
Andy Grove [Fri, 20 May 2022 14:51:02 +0000 (08:51 -0600)] 
update project description (#29)

4 months agoRemove submodules (#28)
Andy Grove [Fri, 20 May 2022 03:37:31 +0000 (21:37 -0600)] 
Remove submodules (#28)

4 months agoFix integration tests (#26)
Andy Grove [Fri, 20 May 2022 00:50:16 +0000 (18:50 -0600)] 
Fix integration tests (#26)

4 months agoReplace README with Ballista version (#4)
Andy Grove [Thu, 19 May 2022 18:35:26 +0000 (12:35 -0600)] 
Replace README with Ballista version (#4)

4 months agoRemove DataFusion crates (#1)
Andy Grove [Thu, 19 May 2022 17:25:51 +0000 (11:25 -0600)] 
Remove DataFusion crates (#1)

* remove DataFusion crates and depend on arrow-datafusion git repo

* add customer.csv file from datafusion repo to fix a test

* delete DataFusion integration tests and CI checks

* fmt

* remove datafusion CI checks

* fix ci

* fix ci

4 months agoFix avro tests (#2570) (#2571)
Raphael Taylor-Davies [Wed, 18 May 2022 18:34:24 +0000 (19:34 +0100)] 
Fix avro tests (#2570) (#2571)

4 months agoRemove `scan_empty` method from `LogicalPlanBuilder` (#2568)
Andy Grove [Wed, 18 May 2022 18:32:39 +0000 (12:32 -0600)] 
Remove `scan_empty` method from `LogicalPlanBuilder` (#2568)

4 months agomake datafusion-proto match exhaustive (#2567)
Andy Grove [Wed, 18 May 2022 14:03:39 +0000 (08:03 -0600)] 
make datafusion-proto match exhaustive (#2567)

4 months agoIntroduce Expr.variant_name() function (#2564)
Jeremy Dyer [Wed, 18 May 2022 13:14:44 +0000 (09:14 -0400)] 
Introduce Expr.variant_name() function (#2564)

* Introduce Expr.variant_name() function

* Change return type from Result<String> -> String

* change return type from String -> &str

4 months agofix `NULL <op> column` evaluation, tests for same (#2510)
Andrew Lamb [Wed, 18 May 2022 01:29:20 +0000 (21:29 -0400)] 
fix `NULL <op> column` evaluation, tests for same (#2510)

4 months agoMove expression utils from sql module to expr crate (#2553)
Andy Grove [Wed, 18 May 2022 01:16:48 +0000 (19:16 -0600)] 
Move expression utils from sql module to expr crate (#2553)

4 months agoFix some 404 links in the contribution guide (#2561)
二手掉包工程师 [Tue, 17 May 2022 19:58:43 +0000 (03:58 +0800)] 
Fix some 404 links in the contribution guide (#2561)

Signed-off-by: hi-rustin <rustin.liu@gmail.com>
4 months agoSupport for OFFSET in LogicalPlan (#2521)
Jeremy Dyer [Tue, 17 May 2022 18:36:54 +0000 (14:36 -0400)] 
Support for OFFSET in LogicalPlan (#2521)

* Introduce support for OFFSET

* lint fixes

* Slightly modify existing test to include LIMIT and OFFSET

* Uncomment accidental comment out for pre-commit script

* OFFSET should come before LIMIT

* Check for OFFSET <= 0 and add more tests

4 months agoFix Redundant ScalarValue Boxed Collection (#2523)
comphead [Tue, 17 May 2022 17:54:30 +0000 (10:54 -0700)] 
Fix Redundant ScalarValue Boxed Collection (#2523)

* unbox scalars

* fix conflicts

* fix clippy

* Update datafusion/common/src/scalar.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Update datafusion/common/src/scalar.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* boxing datatype

* fixing test

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
4 months agoUpdate datafusion-cli readme cli version (#2559)
二手掉包工程师 [Tue, 17 May 2022 16:05:04 +0000 (00:05 +0800)] 
Update datafusion-cli readme cli version (#2559)

Signed-off-by: hi-rustin <rustin.liu@gmail.com>
4 months agoMINOR: Move `expr_rewriter.rs` to `datafusion-expr` crate (#2552)
Andy Grove [Tue, 17 May 2022 11:07:08 +0000 (05:07 -0600)] 
MINOR: Move `expr_rewriter.rs` to `datafusion-expr` crate (#2552)

* move expr_rewrite to expr crate

* Move expr_rewriter to expr crate

4 months agoUpdate to arrow-rs 14.0.0 (#2528)
Andrew Lamb [Tue, 17 May 2022 11:03:43 +0000 (07:03 -0400)] 
Update to arrow-rs 14.0.0  (#2528)

* TEMP: Patch to use apache repo

* Update to arrow 14.0.0

* Consolidate to single OffsetSizeTrait

* Update for new API

* clippy

* moar clippy

* TEMP: patch datafusion cli

* fixup

* Update datafusion-cli deps

4 months agoRemove `scan_csv` methods from `LogicalPlanBuilder` (#2537)
Andy Grove [Mon, 16 May 2022 22:12:17 +0000 (16:12 -0600)] 
Remove `scan_csv` methods from `LogicalPlanBuilder` (#2537)

4 months agoMINOR: Fix release packaging issues (#2545)
Andy Grove [Mon, 16 May 2022 20:38:54 +0000 (14:38 -0600)] 
MINOR: Fix release packaging issues (#2545)

* fix release issues

* add newline at end of Cargo.toml

4 months agoFix size_of_scalar test (#2531)
Andrew Lamb [Mon, 16 May 2022 17:32:56 +0000 (13:32 -0400)] 
Fix size_of_scalar test (#2531)

* Fix size_of_scalar test

* add comments

* Update tests

4 months agoRemove scan_avro methods from LogicalPlanBuilder (#2540)
Andy Grove [Mon, 16 May 2022 15:00:33 +0000 (09:00 -0600)] 
Remove scan_avro methods from LogicalPlanBuilder (#2540)

4 months agosplit ON expressions only by AND operator (#2534)
Eduard Karacharov [Mon, 16 May 2022 12:23:30 +0000 (15:23 +0300)] 
split ON expressions only by AND operator (#2534)