arrow-rs.git
3 hours agoRemove sample.parquet (#2776) master
Raphael Taylor-Davies [Sat, 24 Sep 2022 18:46:10 +0000 (19:46 +0100)] 
Remove sample.parquet (#2776)

3 hours agoFix Backwards Compatible Parquet List Encodings (#1915) (#2774)
Raphael Taylor-Davies [Sat, 24 Sep 2022 18:39:13 +0000 (19:39 +0100)] 
Fix Backwards Compatible Parquet List Encodings (#1915) (#2774)

* Fix schema for non-list repeated fields (#1915)

* Clippy

6 hours agoRemove ArrowNativeType: FromStr (#2775)
Raphael Taylor-Davies [Sat, 24 Sep 2022 16:26:24 +0000 (17:26 +0100)] 
Remove ArrowNativeType: FromStr (#2775)

* Remove ArrowNativeType: FromStr

* Format

6 hours agoTrim parquet row selection (#2705)
Raphael Taylor-Davies [Sat, 24 Sep 2022 16:20:49 +0000 (17:20 +0100)] 
Trim parquet row selection (#2705)

32 hours agoRe-enable golang integration tests (#2688) (#2773)
Raphael Taylor-Davies [Fri, 23 Sep 2022 14:12:02 +0000 (15:12 +0100)] 
Re-enable golang integration tests (#2688) (#2773)

2 days agoMINOR: Fix clippy for rust 1.64.0 (#2772)
Liang-Chi Hsieh [Thu, 22 Sep 2022 20:43:37 +0000 (13:43 -0700)] 
MINOR: Fix clippy for rust 1.64.0 (#2772)

* Fix clippy

* More

2 days agoFix clippy (#2771)
Liang-Chi Hsieh [Thu, 22 Sep 2022 18:47:28 +0000 (11:47 -0700)] 
Fix clippy (#2771)

2 days agoAdd divide_scalar_opt_dyn (#2768)
Liang-Chi Hsieh [Thu, 22 Sep 2022 16:51:06 +0000 (09:51 -0700)] 
Add divide_scalar_opt_dyn (#2768)

2 days agoSplit out `arrow-data` into a separate crate (#2746)
Raphael Taylor-Davies [Thu, 22 Sep 2022 09:35:57 +0000 (10:35 +0100)] 
Split out `arrow-data` into a separate crate (#2746)

* Split out arrow-data

* RAT

* Fix integration test

2 days agoAdd divide dyn kernel which produces null for division by zero (#2764)
Liang-Chi Hsieh [Thu, 22 Sep 2022 02:12:24 +0000 (19:12 -0700)] 
Add divide dyn kernel which produces null for division by zero (#2764)

* Add divide_dyn_opt kernel

* Add test

* Fix clippy

* Rename function

3 days agoSplit out arrow-schema (#2594) (#2711)
Raphael Taylor-Davies [Wed, 21 Sep 2022 12:04:52 +0000 (13:04 +0100)] 
Split out arrow-schema (#2594) (#2711)

* Split out arrow-schema (#2594)

* Flatten schema

* Move decimal logic

* Fix doc

* Fix tests

* Fix integration-test

* Remove pyarrow orphan

* PyArrow fixes

* Move ArrowError to arrow-schema

* Fix pyarrow

* Fix test

* Fix conflicts

* Fix pyarrow

* Tweak feature flags

* Test juggling

* Derive PyArrowConvert for Vec

4 days agoAdd dyn_arith_dict feature flag (#2760)
Raphael Taylor-Davies [Tue, 20 Sep 2022 21:30:27 +0000 (22:30 +0100)] 
Add dyn_arith_dict feature flag (#2760)

* Add dyn_arith_dict feature flag

* Document feature flag

4 days agoMINOR: tweak arrow release instructions (#2758)
Andrew Lamb [Tue, 20 Sep 2022 10:30:55 +0000 (06:30 -0400)] 
MINOR: tweak arrow release instructions (#2758)

4 days agoAdd overflow-checking variants of arithmetic dyn kernels (#2740)
Liang-Chi Hsieh [Tue, 20 Sep 2022 10:30:37 +0000 (03:30 -0700)] 
Add overflow-checking variants of arithmetic dyn kernels (#2740)

* Init

* More

* More

* Add tests

* Fix clippy

* Remove macro

* Update doc

* Fix clippy

* Remove length check

* Tweak try_binary to coordinate latest optimization

* Fix clippy

* Use for loop

* Split non-null variant into never inline function

* Add value type check

* Multiply by get_byte_width of output type.

5 days agoFix: Issue 2721 : binary function should not panic but return error when array length...
aksharau [Mon, 19 Sep 2022 05:48:23 +0000 (11:18 +0530)] 
Fix: Issue 2721 : binary function should not panic but return error when array lengths are unequal (#2750)

7 days agoAdd value type check in try_unary_dict (#2755)
Liang-Chi Hsieh [Sat, 17 Sep 2022 08:04:02 +0000 (01:04 -0700)] 
Add value type check in try_unary_dict (#2755)

7 days agoSpeed up checked kernels for non-null data (~1.4-5x faster) (#2749)
Daniël Heres [Sat, 17 Sep 2022 03:58:09 +0000 (05:58 +0200)] 
Speed up checked kernels for non-null data (~1.4-5x faster) (#2749)

* Speed up checked kernels

* Fast path for non-null

* Move some code

7 days agoFix `verify_release_candidate.sh` for new arrow subcrates (#2752)
Andrew Lamb [Fri, 16 Sep 2022 23:42:20 +0000 (19:42 -0400)] 
Fix `verify_release_candidate.sh` for new arrow subcrates (#2752)

8 days agoupdate new `arrow-buffer` crate to 23.0.0 (#2748) 23.0.0
Andrew Lamb [Fri, 16 Sep 2022 16:40:37 +0000 (12:40 -0400)] 
update new `arrow-buffer` crate to 23.0.0 (#2748)

* update new `arrow-buffer` crate to 23.0.0

* Update dependency

8 days agoUpdate version to `23.0.0` and update `CHANGELOG`, add `label_issue.py` script (...
Ian Alexander Joiner [Fri, 16 Sep 2022 16:23:29 +0000 (12:23 -0400)] 
Update version to `23.0.0` and update `CHANGELOG`, add `label_issue.py` script (#2734)

* feature complete

* fix footer issue

* fix duplicate changelog issue

* use tac instead of head for head -n -<num> is not universal

* adjust blank lines

* fix footer dropping

* line adj

* add .bak2 to gitignore

* Create changelog

* Update version

* Add initial relabeling script

* more script

* tweaks

* Runnable as a script

* Update changelog

* updates

* remove overzealous api change labeling

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
8 days agoSpeedup string equal/not equal to empty string, cleanup like/ilike kernels, fix escap...
Daniël Heres [Fri, 16 Sep 2022 12:52:48 +0000 (14:52 +0200)] 
Speedup string equal/not equal to empty string, cleanup like/ilike kernels, fix escape bug (#2743)

* Speedup string == ""

* neq too

* Simplify kernels

* Simplify kernels

* Fix test

* Escape contains

* Fmt

* Fix

8 days agoUpdate `try_binary` and `checked_ops`, and remove `math_checked_op` (#2717)
Remzi Yang [Fri, 16 Sep 2022 10:47:20 +0000 (18:47 +0800)] 
Update `try_binary` and `checked_ops`, and remove `math_checked_op` (#2717)

* update try_binary
delete math_checked_op
update the return type of checked ops

Signed-off-by: remzi <13716567376yh@gmail.com>
* float div not panic on zero

Signed-off-by: remzi <13716567376yh@gmail.com>
* fix nan test

Signed-off-by: remzi <13716567376yh@gmail.com>
* add float divide by zero

Signed-off-by: remzi <13716567376yh@gmail.com>
* add float tests

Signed-off-by: remzi <13716567376yh@gmail.com>
* fix compile error

Signed-off-by: remzi <13716567376yh@gmail.com>
Signed-off-by: remzi <13716567376yh@gmail.com>
8 days agoBetter construction of RecordBatchOptions (#2729)
askoa [Fri, 16 Sep 2022 09:59:39 +0000 (05:59 -0400)] 
Better construction of RecordBatchOptions (#2729)

* include builder for RecordBatchOptions

* fix clippy warnings

* fix clippy warnings

* remove builder struct

* removed a wrong comment

* Update comment  in arrow/src/record_batch.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Update comment in arrow/src/record_batch.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: askoa <askoa@local>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
9 days agoPartially flatten arrow-buffer (#2737)
Raphael Taylor-Davies [Thu, 15 Sep 2022 21:44:22 +0000 (22:44 +0100)] 
Partially flatten arrow-buffer (#2737)

* Partially flatten arrow-buffer

* Format

9 days agoAutomate updates to `CHANGELOG-old.md` (#2732)
Ian Alexander Joiner [Thu, 15 Sep 2022 19:21:11 +0000 (15:21 -0400)] 
Automate updates to `CHANGELOG-old.md` (#2732)

* feature complete

* fix footer issue

* fix duplicate changelog issue

* use tac instead of head for head -n -<num> is not universal

* adjust blank lines

* fix footer dropping

* line adj

* add .bak2 to gitignore

9 days agobenchmark: bitwise operation (#2718)
Kun Liu [Thu, 15 Sep 2022 19:12:06 +0000 (03:12 +0800)] 
benchmark: bitwise operation (#2718)

* add benchmark for bitwise operation

* add bench for bitwise or xor not

9 days agoUpdate read parquet example in parquet/arrow home (#2730)
Marc Garcia [Thu, 15 Sep 2022 16:23:22 +0000 (17:23 +0100)] 
Update read parquet example in parquet/arrow home (#2730)

* Update example to read parquet

* Remove outdated comment

9 days agoFix multipart uploads on Minio (#2731)
Artjoms Iskovs [Thu, 15 Sep 2022 16:22:53 +0000 (17:22 +0100)] 
Fix multipart uploads on Minio (#2731)

The official Minio SDK uses "uploads=" as the URL when it initiates a
multipart upload instead of "uploads". This affects the AWSV4 signature
and causes object_store to fail a signature check when initiating the
upload to Minio.

It's possible that this contradicts the AWS S3 API docs:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html#API_CreateMultipartUpload_RequestSyntax
and we need to instead keep the URL as `?uploads` and
change the URL that goes into the signature instead.

9 days agoSplit out arrow-buffer crate (#2594) (#2693)
Raphael Taylor-Davies [Thu, 15 Sep 2022 14:30:54 +0000 (15:30 +0100)] 
Split out arrow-buffer crate (#2594) (#2693)

* Split out arrow-buffer crate (#2594)

* Fix doc

* Review feedback

* Review feedback

* Use 64-bit wide collect_bool

9 days agoAdd overflow-checking variants of arithmetic scalar dyn kernels (#2713)
Liang-Chi Hsieh [Wed, 14 Sep 2022 23:54:23 +0000 (16:54 -0700)] 
Add overflow-checking variants of arithmetic scalar dyn kernels (#2713)

* Add overflow-checking variants of arithmetic scalar dyn kernels

* Update doc

* For review

10 days agoAdd support of sorting dictionary of other primitive arrays (#2701)
Liang-Chi Hsieh [Wed, 14 Sep 2022 21:05:58 +0000 (14:05 -0700)] 
Add support of sorting dictionary of other primitive arrays (#2701)

* Add support of sorting dictionary of other primitive arrays

* Collapse match statements

* Add one helper to match primitive types

10 days agoMove JSON Test Format To integration-testing (#2724)
Raphael Taylor-Davies [Wed, 14 Sep 2022 13:42:11 +0000 (14:42 +0100)] 
Move JSON Test Format To integration-testing (#2724)

* Move JSON Test Format To integration-testing

* Fix RAT

11 days agoAdd divide_opt kernel which produce null values on division by zero error (#2710)
Liang-Chi Hsieh [Tue, 13 Sep 2022 21:23:52 +0000 (14:23 -0700)] 
Add divide_opt kernel which produce null values on division by zero error (#2710)

* Add divide_opt kernel

* Add fast-path for non-null arrays

* Add doc

11 days agosupport bitwise op: or,xor,not (#2716)
Kun Liu [Tue, 13 Sep 2022 19:08:47 +0000 (03:08 +0800)] 
support bitwise op: or,xor,not (#2716)

11 days agooptimize the `numeric_cast_with_error` (#2661)
Kun Liu [Tue, 13 Sep 2022 06:06:27 +0000 (14:06 +0800)] 
optimize the `numeric_cast_with_error` (#2661)

* optimize the numeric_cast_with_error

* fix error message and change the function name

11 days agoSupport bitwise and operation in the kernel (#2703)
Kun Liu [Tue, 13 Sep 2022 03:12:32 +0000 (11:12 +0800)] 
Support bitwise and operation in the kernel (#2703)

* add bitwise add op

* address comments: change the test

12 days agoAdd DataType::is_nested() (#2707)
Konstantin Fastov [Mon, 12 Sep 2022 19:12:03 +0000 (22:12 +0300)] 
Add DataType::is_nested() (#2707)

* Add DataType::is_nested()

* Fix documentation for is_nested

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
12 days agoUpdate criterion requirement from 0.3 to 0.4 (#2706)
dependabot[bot] [Mon, 12 Sep 2022 15:32:46 +0000 (16:32 +0100)] 
Update criterion requirement from 0.3 to 0.4 (#2706)

Updates the requirements on [criterion](https://github.com/bheisler/criterion.rs) to permit the latest version.
- [Release notes](https://github.com/bheisler/criterion.rs/releases)
- [Changelog](https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bheisler/criterion.rs/compare/0.3.0...0.4.0)

---
updated-dependencies:
- dependency-name: criterion
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
12 days agoOverflow-checking variant of arithmetic scalar kernels (#2650)
Liang-Chi Hsieh [Mon, 12 Sep 2022 07:43:35 +0000 (00:43 -0700)] 
Overflow-checking variant of arithmetic scalar kernels (#2650)

* Overflow-checking variant of arithmetic scalar kernels

* Remove division scalar change for now.

13 days agoClarify docs of binary and string builders (#2699)
Marc Garcia [Sun, 11 Sep 2022 12:15:21 +0000 (14:15 +0200)] 
Clarify docs of binary and string builders (#2699)

* Clarify docs of binary and string builders

* Improve doc of with capacity based on review feedback

13 days agoVerify valid UTF-8 when converting byte array (#2205) (#2686)
Raphael Taylor-Davies [Sun, 11 Sep 2022 06:34:51 +0000 (07:34 +0100)] 
Verify valid UTF-8 when converting byte array (#2205) (#2686)

* Verify valid UTF-8 when converting byte array (#2205)

* Add doc comment

13 days agoAdd try_unary, binary, try_binary kernels (#2666)
Raphael Taylor-Davies [Sun, 11 Sep 2022 06:34:12 +0000 (07:34 +0100)] 
Add try_unary, binary, try_binary kernels (#2666)

2 weeks agoUpdate thrift v0.16 and vendor parquet-format (#2502) (#2626)
Raphael Taylor-Davies [Sat, 10 Sep 2022 21:26:27 +0000 (22:26 +0100)] 
Update thrift v0.16 and vendor parquet-format (#2502) (#2626)

* Update thrift v0.16 vendor parquet-format (#2502)

* Lint

* Add linguist-generated

* Use archlinux docker image

* Review feedback

* Fix doc

* Format

2 weeks agoSort indices of dictionary string values (#2698)
Liang-Chi Hsieh [Sat, 10 Sep 2022 21:24:47 +0000 (14:24 -0700)] 
Sort indices of dictionary string values (#2698)

* Refactor dictionary string sorting

* Fix clippy

2 weeks agoComparable Row Format (#2593)
Raphael Taylor-Davies [Sat, 10 Sep 2022 20:27:37 +0000 (21:27 +0100)] 
Comparable Row Format (#2593)

* Add row format

* Skip miri on heavier tests

* Handle nulls in dictionary values

* Don't fuzz test dictionaries with null values

* Add docs

* Add error plumbing

* Review feedback

* Fix docs

2 weeks agoUpdate quick-xml to 0.25 (#2695)
Raphael Taylor-Davies [Sat, 10 Sep 2022 17:55:46 +0000 (18:55 +0100)] 
Update quick-xml to 0.25 (#2695)

2 weeks agoSupport sorting dictionary encoded primitive integer arrays (#2680)
Liang-Chi Hsieh [Sat, 10 Sep 2022 07:22:38 +0000 (00:22 -0700)] 
Support sorting dictionary encoded primitive integer arrays (#2680)

* Support sorting dictionary encoded primitive arrays

* Reduce combinatorial fanout

* Change from &SortOptions to SortOptions

* Fix value order and add a test

* Fix null ordering and add test

* Add comment and increase test coverage.

2 weeks agoAdd support for empty projection in RecordBatch::project (#2691)
Daniël Heres [Fri, 9 Sep 2022 02:50:22 +0000 (04:50 +0200)] 
Add support for empty projection in RecordBatch::project (#2691)

* Add support for empty projection in RecordBatch::project

* Simplify

2 weeks agoSimplify DictionaryBuilder constructors (#2684) (#2054) (#2685)
Raphael Taylor-Davies [Thu, 8 Sep 2022 21:16:26 +0000 (22:16 +0100)] 
Simplify DictionaryBuilder constructors (#2684) (#2054) (#2685)

* Simplify DictionaryBuilder constructors (#2684) (#2054)

* Apply suggestions from code review

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
2 weeks agoFix master (#2692)
Raphael Taylor-Davies [Thu, 8 Sep 2022 18:01:34 +0000 (19:01 +0100)] 
Fix master (#2692)

2 weeks agoDeprecate RecordBatch::concat (#2594) (#2683)
Raphael Taylor-Davies [Thu, 8 Sep 2022 17:27:23 +0000 (18:27 +0100)] 
Deprecate RecordBatch::concat (#2594) (#2683)

2 weeks agoTemporarily disable Golang integration tests re-enabled JS (#2689)
Raphael Taylor-Davies [Thu, 8 Sep 2022 15:55:36 +0000 (16:55 +0100)] 
Temporarily disable Golang integration tests re-enabled JS (#2689)

2 weeks agoPrepare object_store 0.5 release (#2682) object_store_0.5.0
Raphael Taylor-Davies [Thu, 8 Sep 2022 11:20:03 +0000 (12:20 +0100)] 
Prepare object_store 0.5 release (#2682)

* Prepare object_store 0.5 release

* Review feedback

2 weeks agoFix multiple part uploads at once making vector size inconsistent (#2681)
Marko Grujic [Thu, 8 Sep 2022 09:05:50 +0000 (11:05 +0200)] 
Fix multiple part uploads at once making vector size inconsistent (#2681)

* Fix multiple part uploads at once making vector size inconsistent

* Calculate total_parts prior to resizing the vector

2 weeks agoSkip RowSelectors with zero rows (#2678)
askoa [Thu, 8 Sep 2022 08:02:43 +0000 (04:02 -0400)] 
Skip RowSelectors with zero rows (#2678)

* Skip RowSelectors with zero rows

* include test for zero RowSelector fix

Co-authored-by: srib <srib@MacDesktop.local>
2 weeks agoSupport building comparator for dictionaries of primitive integer values (#2673)
Liang-Chi Hsieh [Thu, 8 Sep 2022 00:10:38 +0000 (17:10 -0700)] 
Support building comparator for dictionaries of primitive integer values (#2673)

* Support comparing dictionary of primitive value.

* Change to generic function

* Trigger Build

2 weeks agoFaster Null Path Selection in ArrayData Equality (#2676)
Dhruv Vats [Wed, 7 Sep 2022 14:07:33 +0000 (19:37 +0530)] 
Faster Null Path Selection in ArrayData Equality (#2676)

* Use contains_nulls if we don't need null count

* Inline function call

2 weeks agoUse BitIndexIterator (#2675)
Liang-Chi Hsieh [Wed, 7 Sep 2022 13:42:37 +0000 (06:42 -0700)] 
Use BitIndexIterator (#2675)

2 weeks agoSupport DictionaryArray in temporal kernels (#2623)
Liang-Chi Hsieh [Wed, 7 Sep 2022 07:11:01 +0000 (00:11 -0700)] 
Support DictionaryArray in temporal kernels (#2623)

* Support dictionary array in temporal kernels

* Support dictionary array in temporal kernels

* Prepare for merging conflicts

* Keep same kernel signature for primitive array

* For review

* Add doc

2 weeks agoAdd ffi, json compilation check (#2671)
Liang-Chi Hsieh [Tue, 6 Sep 2022 22:17:28 +0000 (15:17 -0700)] 
Add ffi, json compilation check (#2671)

* Add ffi compilation check

* Add compilation check for json

2 weeks agoSupport comparison between dictionary array and binary array (#2645)
Liang-Chi Hsieh [Tue, 6 Sep 2022 20:31:41 +0000 (13:31 -0700)] 
Support comparison between dictionary array and binary array (#2645)

* Support comparison between dictionary array and binary array

* Use downcast_dictionary_array

2 weeks agoUse downcast_dictionary_array in unary_dyn (#2663)
Raphael Taylor-Davies [Tue, 6 Sep 2022 20:02:53 +0000 (21:02 +0100)] 
Use downcast_dictionary_array in unary_dyn (#2663)

* Use downcast_dictionary_array in unary_dyn

* Further cleanups

* Clippy

2 weeks agoChange macro to generic helper function (#2658)
Liang-Chi Hsieh [Tue, 6 Sep 2022 16:06:58 +0000 (09:06 -0700)] 
Change macro to generic helper function (#2658)

2 weeks agojson feature always requires base64 feature (#2668)
Kohei Suzuki [Tue, 6 Sep 2022 15:56:41 +0000 (00:56 +0900)] 
json feature always requires base64 feature (#2668)

It should fix build failure when only json feature is enabled.

```
% cargo build --no-default-features --features json
error[E0433]: failed to resolve: use of undeclared crate or module `base64`
   --> parquet/src/record/api.rs:691:46
    |
691 |             Field::Bytes(b) => Value::String(base64::encode(b.data())),
    |                                              ^^^^^^ use of undeclared crate or module `base64`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `parquet` due to previous error
```

2 weeks agoFix generate random selection (#2656)
Dan Harris [Tue, 6 Sep 2022 13:39:36 +0000 (09:39 -0400)] 
Fix generate random selection (#2656)

2 weeks agoFully qualifying parquet items (#2638)
X [Tue, 6 Sep 2022 10:07:13 +0000 (18:07 +0800)] 
Fully qualifying parquet items (#2638)

* fully qualifying parquet items

* rustfmt

* remove glob import

* remove unnecessary borrow

* import repeated items

* rustfmt

2 weeks agoffi feature also requires layout (#2660)
Liang-Chi Hsieh [Tue, 6 Sep 2022 07:55:58 +0000 (00:55 -0700)] 
ffi feature also requires layout (#2660)

2 weeks agosupport `CastOption` for casting numeric (#2649)
Kun Liu [Tue, 6 Sep 2022 02:44:45 +0000 (10:44 +0800)] 
support `CastOption` for casting numeric (#2649)

* cast numeric to numeric with error

* fix comment

2 weeks agoIgnore flaky test (#2655)
Liang-Chi Hsieh [Mon, 5 Sep 2022 19:44:25 +0000 (12:44 -0700)] 
Ignore flaky test (#2655)

2 weeks agoupdate doc for object_store copy_if_not_exists (#2653)
JanKaul [Mon, 5 Sep 2022 16:52:19 +0000 (18:52 +0200)] 
update doc for object_store copy_if_not_exists (#2653)

2 weeks agoMutableBuffer::typed_data - shared ref access to the typed slice (#2652)
Michael Edwards [Mon, 5 Sep 2022 15:46:56 +0000 (17:46 +0200)] 
MutableBuffer::typed_data - shared ref access to the typed slice (#2652)

2 weeks agoFix bug in page skipping (#2552)
Dan Harris [Mon, 5 Sep 2022 11:17:47 +0000 (07:17 -0400)] 
Fix bug in page skipping (#2552)

* Fix bug in page skipping

* Revert

* When skipping records, only read dictionary page

* Fix remaining page skipping bug and add unit tests

2 weeks agoUse `downcast_primitive_array` in arithmetic kernels (#2640)
Liang-Chi Hsieh [Mon, 5 Sep 2022 11:06:15 +0000 (04:06 -0700)] 
Use `downcast_primitive_array` in arithmetic kernels (#2640)

* Use downcast_primitive_array in arithmetic kernels

* Constrain two sides to be same type

2 weeks agoHelp LLVM vectorize comparison kernel ~50-80% faster (#2646)
Raphael Taylor-Davies [Sun, 4 Sep 2022 17:55:49 +0000 (18:55 +0100)] 
Help LLVM vectorize comparison kernel ~50-80% faster (#2646)

* Help LLVM vectorize comparison kernel

* Add MutableBuffer::collect_bool

* Add SAFETY comments

2 weeks agoAdd overflow-checking variant for primitive arithmetic kernels and explicitly define...
Liang-Chi Hsieh [Sun, 4 Sep 2022 09:44:36 +0000 (02:44 -0700)] 
Add overflow-checking variant for primitive arithmetic kernels and explicitly define overflow behavior (#2643)

* Add overflow-checking variant for add kernel and explicitly define overflow behavior for add

* For subtract, multiply, divide

* Fix tests

* Fix different error message

* Fix typo

* Rename APIs and add more comments. Print values in error message.

* Add one more test to distinct divide_by_zero behavior on divide.

* Fix clippy

* Update divide doc with dividing by zero behavior for other numeric types.

* Hide ArrowNativeTypeOp

* Fix a typo

3 weeks agoUpdate flight definitions including backwards-incompatible change to GetSchema (...
Kun Liu [Sat, 3 Sep 2022 11:42:34 +0000 (19:42 +0800)] 
Update flight definitions including backwards-incompatible change to GetSchema (#2586)

* update flight doc and code

* fix cliyyp

* backward compatibility for schema result

* fix lint

* Update arrow/src/ipc/convert.rs

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* Update arrow/src/ipc/convert.rs

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
3 weeks agoUpdate version to `22.0.0` and update `CHANGELOG` (#2612) 22.0.0
Ian Alexander Joiner [Fri, 2 Sep 2022 17:52:17 +0000 (13:52 -0400)] 
Update version to `22.0.0` and update `CHANGELOG` (#2612)

* Update version

* Fix hypercorrection

* Remove gitignore-worthy stuff

* Fix changelog

* Update changelog

* prepend as opposed to overwrite

* Change dev README

* Update with tags

* Doc updates

* final updates

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
3 weeks agoAdd Linux, Mac and Windows .gitignore (#2630)
Ian Alexander Joiner [Fri, 2 Sep 2022 14:36:15 +0000 (10:36 -0400)] 
Add Linux, Mac and Windows .gitignore (#2630)

3 weeks agoAdd downcast macros (#2635) (#2636)
Raphael Taylor-Davies [Fri, 2 Sep 2022 14:32:44 +0000 (15:32 +0100)] 
Add downcast macros (#2635) (#2636)

* Add downcast macros (#2635)

* Add Float16 support and trailing commas

* Review feedback

3 weeks agoUse http with fake-gcs (#2632)
Raphael Taylor-Davies [Fri, 2 Sep 2022 11:34:22 +0000 (12:34 +0100)] 
Use http with fake-gcs (#2632)

3 weeks agoDocument all arrow features in docs.rs (#2633) (#2634)
Raphael Taylor-Davies [Fri, 2 Sep 2022 11:30:36 +0000 (12:30 +0100)] 
Document all arrow features in docs.rs (#2633) (#2634)

* Document all arrow features in docs.rs (#2633)

* Tweak pyarrow doc

* Update arrow/Cargo.toml

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
3 weeks agoDocument dyn_cmp_dict (#2624)
Raphael Taylor-Davies [Thu, 1 Sep 2022 16:07:30 +0000 (17:07 +0100)] 
Document dyn_cmp_dict (#2624)

3 weeks agoUpdate quick-xml 0.24 (#2625)
Raphael Taylor-Davies [Thu, 1 Sep 2022 16:07:21 +0000 (17:07 +0100)] 
Update quick-xml 0.24 (#2625)

3 weeks agoMake JSON support optional via a feature flag (#2300) (#2601)
Raphael Taylor-Davies [Thu, 1 Sep 2022 14:27:49 +0000 (15:27 +0100)] 
Make JSON support optional via a feature flag (#2300) (#2601)

* Add json feature (#2300)

* Fix parquet

* Add json feature to wasm build

* Update README

3 weeks agoUse required-features for CSV benchmarks/examples (#2602)
Raphael Taylor-Davies [Thu, 1 Sep 2022 13:32:41 +0000 (14:32 +0100)] 
Use required-features for CSV benchmarks/examples (#2602)

3 weeks agoCast timestamp array to string array with timezone (#2608)
Liang-Chi Hsieh [Thu, 1 Sep 2022 07:17:04 +0000 (00:17 -0700)] 
Cast timestamp array to string array with timezone (#2608)

* Cast timestamp array to string array with timezone.

* Add comments

3 weeks agoSupport comparison between DictionaryArray to BooleanArray (#2618)
Liang-Chi Hsieh [Wed, 31 Aug 2022 19:19:21 +0000 (12:19 -0700)] 
Support comparison between DictionaryArray to BooleanArray (#2618)

3 weeks agoMove gcp.rs (#2619)
Raphael Taylor-Davies [Wed, 31 Aug 2022 18:58:39 +0000 (19:58 +0100)] 
Move gcp.rs (#2619)

3 weeks agoUse total_cmp for floating value ordering and remove nan_ordering flag (#2614)
Liang-Chi Hsieh [Tue, 30 Aug 2022 19:54:16 +0000 (12:54 -0700)] 
Use total_cmp for floating value ordering and remove nan_ordering flag (#2614)

3 weeks agoAdd IMDSv1 fallback (#2609) (#2610)
Raphael Taylor-Davies [Tue, 30 Aug 2022 16:42:26 +0000 (17:42 +0100)] 
Add IMDSv1 fallback (#2609) (#2610)

* Add IMDSv1 fallback (#2609)

* Add config option

3 weeks agoCompare dictionary array with string array (#2549)
Liang-Chi Hsieh [Mon, 29 Aug 2022 20:42:25 +0000 (13:42 -0700)] 
Compare dictionary array with string array (#2549)

* Compare dictionary array to string array

* Trigger Build

3 weeks agoSupport SQL-compliant NaN ordering for DictionaryArray and non-DictionaryArray (...
Liang-Chi Hsieh [Mon, 29 Aug 2022 20:02:07 +0000 (13:02 -0700)] 
Support SQL-compliant NaN ordering for DictionaryArray and non-DictionaryArray (#2600)

3 weeks agoSupport empty projection in CSV and JSON readers (#2604)
Daniël Heres [Mon, 29 Aug 2022 17:24:17 +0000 (20:24 +0300)] 
Support empty projection in CSV and JSON readers (#2604)

* Add support for CSV and JSON readers

* Fix projection option not set

* Fix doc

3 weeks agoadd bench: decimal with byte array and fixed length byte array (#2529)
Kun Liu [Mon, 29 Aug 2022 11:32:09 +0000 (19:32 +0800)] 
add bench: decimal with byte array and fixed length byte array (#2529)

* add bench: decimal with byte array and fixed length byte array

* change comments

3 weeks agoSplit out integration test plumbing (#2594) (#2300) (#2598)
Raphael Taylor-Davies [Sun, 28 Aug 2022 09:55:23 +0000 (10:55 +0100)] 
Split out integration test plumbing (#2594) (#2300) (#2598)

* Split out integration test plumbing (#2594) (#2300)

* Fix RAT

4 weeks agoGate dyn comparison of dictionary arrays behind `dyn_cmp_dict` (#2597)
Raphael Taylor-Davies [Sat, 27 Aug 2022 17:56:50 +0000 (18:56 +0100)] 
Gate dyn comparison of dictionary arrays behind `dyn_cmp_dict` (#2597)

* Add dyn_cmp_dict feature flag

* Fix tests

* Clippy

4 weeks agoObjectStore cleanup (#2587) (#2590)
Raphael Taylor-Davies [Sat, 27 Aug 2022 16:54:22 +0000 (17:54 +0100)] 
ObjectStore cleanup (#2587) (#2590)

* ObjectStore cleanup (#2587)

* Fix CI

4 weeks agoRefactor Binary Builder and String Builder Constructors (#2592)
Vrishabh [Sat, 27 Aug 2022 12:44:17 +0000 (18:14 +0530)] 
Refactor Binary Builder and String Builder Constructors (#2592)

* Binary and String array builder refactor

* Fix minor typo

* Fix errors

* Fix clippy lints

4 weeks agoMove JsonSerializable to json module (#2300) (#2595)
Raphael Taylor-Davies [Sat, 27 Aug 2022 07:41:12 +0000 (08:41 +0100)] 
Move JsonSerializable to json module (#2300) (#2595)

4 weeks agoDictionary like scalar kernels (#2591)
Vrishabh [Fri, 26 Aug 2022 22:43:10 +0000 (04:13 +0530)] 
Dictionary like scalar kernels (#2591)

* Intial implmentation of like kernels

* Refactor nlike_scalar kernels

* Fix cargo.toml

* Add other dict scalar kernels

* Replace macro with array accessor functions

* Remove commented code

* Fix typo in error message

* Fix doc comments