beam.git
20 hours agoJdbcIO fetchSize can be set to Integer.MIN_VALUE (#23444) master
Mc [Sat, 1 Oct 2022 13:12:03 +0000 (15:12 +0200)] 
JdbcIO fetchSize can be set to Integer.MIN_VALUE (#23444)

* JdbcIO fetchSize can be set to Integer.MIN_VALUE

* Apply suggestions from code review

Add comments about non-standard value.

* Fix formatting and allow users to reset to default value of 0.

* Apply suggestions from code review

Co-authored-by: Lukasz Cwik <lcwik@google.com>
34 hours agoAdd cron job to trigger Java JMH micro-benchmarks weekly (#23388)
Moritz Mack [Fri, 30 Sep 2022 22:47:18 +0000 (00:47 +0200)] 
Add cron job to trigger Java JMH micro-benchmarks weekly  (#23388)

* Add cron job to trigger Java JMH micro-benchmarks weekly (addresses #22238)

* Update jobs inventory

36 hours agoSupport VR test including TestStream for Spark runner in streaming mode (#22620)
Moritz Mack [Fri, 30 Sep 2022 21:12:36 +0000 (23:12 +0200)] 
Support VR test including TestStream for Spark runner in streaming mode (#22620)

38 hours agoFix type inference for set/delete attr. (#23242)
Robert Bradshaw [Fri, 30 Sep 2022 19:01:12 +0000 (12:01 -0700)] 
Fix type inference for set/delete attr. (#23242)

38 hours agoMerge pull request #23194 Batch encoding and decoding of schema data.
Robert Bradshaw [Fri, 30 Sep 2022 19:00:16 +0000 (12:00 -0700)] 
Merge pull request #23194 Batch encoding and decoding of schema data.

38 hours ago[Playground] [Backend] Edited the function for getting executable name (#22957)
Vladislav Chunikhin [Fri, 30 Sep 2022 18:55:01 +0000 (22:55 +0400)] 
[Playground] [Backend] Edited the function for getting executable name (#22957)

* [Playground] [Infrastructure] format python code style

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Backend] added the datastore query to get an example catalog

* [Playground] [Backend] added datastore queries to get precompiled objects

* [Playground] [Backend] added cache component for examples

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Backend] updated example endpoints

* [Playground] [Backend] added sdk as a part of example response

* [Playground] [Backend] removed the storage client from playground backend

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for db utils

* [Playground] [Backend] updated CONTRIBUTE.md file

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Infrastructure] added complexity tag

* [Playground] [Infrastructure] added complexity to example comment

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* squash! [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* [Playground] [Backend] updated playground README

* [Playground] [Backend] added complexity field to playground response

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] added new schema version

* [Playground] [Infrastructure] added tags field to playground examples

* [Playground] [Infrastructure] fixed py unit test

* [Playground] [Infrastructure] removed extra import

* added tags for examples

added tags for examples

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Infrastructure] added the google cloud project id as an env variable

* [Playground] [Backend] updated docker files for runners

* [Playground] [Backend] added SDK validation to save a code snippet

* [Playground] [Backend] removed comments

* [Playground] [Backend] resolved comments after review

* [Playground] [Infrastructure] updated dependency versions

* [Playground] [Backend] fixed the java runner for examples with graphs

* [Playground] [Infrastructure] added loading catalogs to the cloud datastore

* [Playground] [Backend] remove extra imports after merging

* [Playground] [Backend] fixes after merging with other branches

* [Playground] [Backend] added genreics for one method

* [Playground] [Backend] fixes after merging

* [Playground] [Backend] fixes after merging

* [Playground] [Infrastructure] updated python scripts for ci cd

* [Playground] [Infrastructure] added complexity to example comment

* updating example complexity values

updating complexity values

* [Playground] [Backend] added tags as part of playground response

* minor fix related with fmt package

* [Playground] [Backend] edted the function to get an executable name

* [Playground] [Backend] updated the complexity field for code snippets

* [Playground] [Backend] fixes after merging with master

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
39 hours ago[Playground] [Backend] Adding the tags field to the example response (#22882)
Vladislav Chunikhin [Fri, 30 Sep 2022 18:11:25 +0000 (22:11 +0400)] 
[Playground] [Backend] Adding the tags field to the example response (#22882)

* [Playground] [Infrastructure] format python code style

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Backend] added the datastore query to get an example catalog

* [Playground] [Backend] added datastore queries to get precompiled objects

* [Playground] [Backend] added cache component for examples

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Backend] updated example endpoints

* [Playground] [Backend] added sdk as a part of example response

* [Playground] [Backend] removed the storage client from playground backend

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for db utils

* [Playground] [Backend] updated CONTRIBUTE.md file

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Infrastructure] added complexity tag

* [Playground] [Infrastructure] added complexity to example comment

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* squash! [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* [Playground] [Backend] updated playground README

* [Playground] [Backend] added complexity field to playground response

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] added new schema version

* [Playground] [Infrastructure] added tags field to playground examples

* [Playground] [Infrastructure] fixed py unit test

* [Playground] [Infrastructure] removed extra import

* added tags for examples

added tags for examples

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Infrastructure] added the google cloud project id as an env variable

* [Playground] [Backend] updated docker files for runners

* [Playground] [Backend] added SDK validation to save a code snippet

* [Playground] [Backend] removed comments

* [Playground] [Backend] resolved comments after review

* [Playground] [Infrastructure] updated dependency versions

* [Playground] [Backend] fixed the java runner for examples with graphs

* [Playground] [Infrastructure] added loading catalogs to the cloud datastore

* [Playground] [Backend] remove extra imports after merging

* [Playground] [Backend] fixes after merging with other branches

* [Playground] [Backend] added genreics for one method

* [Playground] [Backend] fixes after merging

* [Playground] [Backend] fixes after merging

* [Playground] [Infrastructure] updated python scripts for ci cd

* [Playground] [Infrastructure] added complexity to example comment

* updating example complexity values

updating complexity values

* [Playground] [Backend] added tags as part of playground response

* minor fix related with fmt package

* [Playground] [Backend] updated the complexity field for code snippets

* [Playground] [Backend] fixes after merging with master

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
39 hours agoreplaced tabs with spaces in readme file (#23446)
Ryan Thompson [Fri, 30 Sep 2022 17:19:54 +0000 (13:19 -0400)] 
replaced tabs with spaces in readme file (#23446)

40 hours agoRelax `pip` check in setup.py to allow installation via other package managers (e...
Daniel Smilkov [Fri, 30 Sep 2022 16:32:20 +0000 (12:32 -0400)] 
Relax `pip` check in setup.py to allow installation via other package managers (e.g. `poetry`) (#23326)

42 hours agoRunInference Benchmarks UI (#23426)
Anand Inguva [Fri, 30 Sep 2022 15:17:11 +0000 (11:17 -0400)] 
RunInference Benchmarks UI (#23426)

* Add additional plots for 2 more metrics

* Change the alias to distinguish among graphs

* Change Yaxes for mean_inference_request_batch_size_*

* Change axis measure unit from seconds to appropriate unit

* Add metrics dashboard for Language Modeling Bert

2 days agoUpdate Python katas to latest version of EduTools and Beam 2.41 (#23180)
Israel Herraiz [Fri, 30 Sep 2022 06:19:10 +0000 (08:19 +0200)] 
Update Python katas to latest version of EduTools and Beam 2.41 (#23180)

* Remove automatically generated files

The files *-remote-info.yaml are automatically generated by Edutools when a new
version of the course is generated (e.g. new package created for offline
distribution, or when a new version is uploaded to the courses marketplace).

* Update Python katas to latest version of EduTools

With the latest versions of the EduTools plugin, the previous katas stopped
working, and this made it impossible to release new versions of the
katas (e.g. after a new Beam release).

The main change is in the testing method. EduTools now uses Python's unittest
for checking the exercises.

I have also re-organized some of the katas (e.g. windowing and triggers are now
under the Streaming category), and updated the katas to Beam 2.41.

* Add missing license headers

2 days ago[BEAM-10785] Change RowAsDictJsonCoder to not ensure ASCII while encoding (#22312)
Seunghwan Hong [Fri, 30 Sep 2022 06:17:45 +0000 (15:17 +0900)] 
[BEAM-10785] Change RowAsDictJsonCoder to not ensure ASCII while encoding (#22312)

* Change RowAsDictJsonCoder to not ensure ASCII while encoding

Signed-off-by: Seunghwan Hong <harrydrippin@gmail.com>
* Format code, Refactor test for readability

Signed-off-by: Seunghwan Hong <harrydrippin@gmail.com>
Signed-off-by: Seunghwan Hong <harrydrippin@gmail.com>
Co-authored-by: Pablo <pabloem@users.noreply.github.com>
2 days agoAdd batch testing for standard row coders. 23194/head
Robert Bradshaw [Wed, 14 Sep 2022 23:25:06 +0000 (16:25 -0700)] 
Add batch testing for standard row coders.

2 days agoAdd microbenchmark for batch row encoding.
Robert Bradshaw [Mon, 12 Sep 2022 18:55:41 +0000 (11:55 -0700)] 
Add microbenchmark for batch row encoding.

2 days agoBatch encoding and decoding of schema data.
Robert Bradshaw [Fri, 9 Sep 2022 23:34:34 +0000 (16:34 -0700)] 
Batch encoding and decoding of schema data.

This implements conversion directly from a dict of numpy arrays to
a concatinated-row-encoded form, and vice versa.

This is generally 5-20x more efficient for encoding and 2-10x more
efficient for decoding, depending on the row types and contents.

Benchmarks:

tiny_row, BatchRowCoderFalse, 1000 element(s)    : p. element median time cost: 3.79658e-06 sec, relative std: 5.82%
tiny_row, BatchRowCoderTrue, 1000 element(s)     : p. element median time cost: 3.9053e-07 sec, relative std: 6256.41%
large_row, BatchRowCoderFalse, 1000 element(s)   : p. element median time cost: 2.4623e-05 sec, relative std: 2.46%
large_row, BatchRowCoderTrue, 1000 element(s)    : p. element median time cost: 1.91796e-06 sec, relative std: 8.61%
nullable_row, BatchRowCoderFalse, 1000 element(s): p. element median time cost: 2.3849e-05 sec, relative std: 2.15%
nullable_row, BatchRowCoderTrue, 1000 element(s) : p. element median time cost: 2.27439e-06 sec, relative std: 28.73%
diverse_row, BatchRowCoderFalse, 1000 element(s) : p. element median time cost: 9.17304e-06 sec, relative std: 2.60%
diverse_row, BatchRowCoderTrue, 1000 element(s)  : p. element median time cost: 2.42209e-06 sec, relative std: 10.16%

2 days agoMerge pull request #23047 Expand Python cross language docs.
Robert Bradshaw [Fri, 30 Sep 2022 00:00:35 +0000 (17:00 -0700)] 
Merge pull request #23047 Expand Python cross language docs.

2 days ago[Playground] [Backend] Removing the code related to the Cloud Storage (#22872)
Vladislav Chunikhin [Thu, 29 Sep 2022 20:07:56 +0000 (00:07 +0400)] 
[Playground] [Backend] Removing the code related to the Cloud Storage (#22872)

* [Playground] [Backend] added the datastore query to get an example catalog

* [Playground] [Backend] added datastore queries to get precompiled objects

* [Playground] [Backend] added cache component for examples

* [Playground] [Backend] updated example endpoints

* [Playground] [Backend] added sdk as a part of example response

* [Playground] [Backend] removed the storage client from playground backend

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for db utils

* [Playground] [Backend] updated CONTRIBUTE.md file

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* squash! [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* [Playground] [Backend] updated playground README

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Infrastructure] added the google cloud project id as an env variable

* [Playground] [Backend] updated docker files for runners

* [Playground] [Backend] added SDK validation to save a code snippet

* [Playground] [Backend] removed comments

* [Playground] [Backend] resolved comments after review

* [Playground] [Infrastructure] updated dependency versions

* [Playground] [Backend] fixed the java runner for examples with graphs

* [Playground] [Infrastructure] added loading catalogs to the cloud datastore

* [Playground] [Backend] remove extra imports after merging

* [Playground] [Backend] fixes after merging with other branches

* [Playground] [Backend] added genreics for one method

* [Playground] [Backend] fixes after merging

* [Playground] [Backend] fixes after merging

* fixing dependencies after merge

fixing dependencies after merge with failed tests

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
2 days ago[Playground] [Backend] Adding SDK to the example response (#22871)
Vladislav Chunikhin [Thu, 29 Sep 2022 19:00:17 +0000 (23:00 +0400)] 
[Playground] [Backend] Adding SDK to the example response (#22871)

* [Playground] [Backend] added the datastore query to get an example catalog

* [Playground] [Backend] added datastore queries to get precompiled objects

* [Playground] [Backend] added cache component for examples

* [Playground] [Backend] updated example endpoints

* [Playground] [Backend] added sdk as a part of example response

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* squash! [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Infrastructure] added the google cloud project id as an env variable

* [Playground] [Backend] updated docker files for runners

* [Playground] [Backend] added SDK validation to save a code snippet

* [Playground] [Backend] removed comments

* [Playground] [Backend] resolved comments after review

* [Playground] [Infrastructure] updated dependency versions

* [Playground] [Backend] fixed the java runner for examples with graphs

* [Playground] [Infrastructure] added loading catalogs to the cloud datastore

* [Playground] [Backend] remove extra imports after merging

* [Playground] [Backend] fixes after merging with other branches

* [Playground] [Backend] added genreics for one method

* [Playground] [Backend] fixes after merging

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
2 days agoSend JavaScript messages to Playground iframes when switching the language in docs...
alexeyinkin [Thu, 29 Sep 2022 18:15:52 +0000 (22:15 +0400)] 
Send JavaScript messages to Playground iframes when switching the language in docs (#22361) (#22960)

* Send JavaScript messages to Playground iframes when switching the language in docs (#22361)

* Change to the production URL (#22361)

* Fix after internal review (#22361)

* Trigger a stand build (#22361)

* Trigger build (#22361)

* Remove !important from CSS (#22361)

2 days ago[Playground] [Backend] Updating endpoints for playground examples (#22870)
Vladislav Chunikhin [Thu, 29 Sep 2022 17:52:03 +0000 (21:52 +0400)] 
[Playground] [Backend] Updating endpoints for playground examples (#22870)

* [Playground] [Backend] added the datastore query to get an example catalog

* [Playground] [Backend] added datastore queries to get precompiled objects

* [Playground] [Backend] added cache component for examples

* [Playground] [Backend] updated example endpoints

* [Playground] [Backend] refactoring for unit tests

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* squash! [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* updated complexity field

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* fixed integration tests

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited test execution

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Backend] updated integration tests

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Infrastructure] added the google cloud project id as an env variable

* [Playground] [Backend] updated docker files for runners

* [Playground] [Backend] added SDK validation to save a code snippet

* [Playground] [Backend] removed comments

* [Playground] [Backend] resolved comments after review

* [Playground] [Infrastructure] updated dependency versions

* [Playground] [Backend] fixed the java runner for examples with graphs

* [Playground] [Infrastructure] added loading catalogs to the cloud datastore

* [Playground] [Backend] remove extra imports after merging

* [Playground] [Backend] fixes after merging with other branches

* [Playground] [Backend] added genreics for one method

* [Playground] [Backend] fixes after merging

* [Playground] [Backend] removed extra logs

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
2 days agoIncrease Go Dataflow Postcommit timeout to 5h (#23423)
Robert Burke [Thu, 29 Sep 2022 17:25:15 +0000 (10:25 -0700)] 
Increase Go Dataflow Postcommit timeout to 5h (#23423)

Stopgap for #23422

2 days agoBump google.golang.org/api from 0.97.0 to 0.98.0 in /sdks (#23394)
dependabot[bot] [Thu, 29 Sep 2022 16:57:17 +0000 (09:57 -0700)] 
Bump google.golang.org/api from 0.97.0 to 0.98.0 in /sdks (#23394)

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.97.0 to 0.98.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.97.0...v0.98.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 days agoMerge pull request #23409 from rohdesamuel/interactive_env
Ning Kang [Thu, 29 Sep 2022 16:41:18 +0000 (09:41 -0700)] 
Merge pull request #23409 from rohdesamuel/interactive_env

Add documentation link to the interactive environment

2 days agoExample of Online Clustering (#23289)
Shubham Krishna [Thu, 29 Sep 2022 15:27:10 +0000 (17:27 +0200)] 
Example of Online Clustering  (#23289)

* Add Online Clustering Code and first draft of documentation

* Update Documentation and correct code snippets

* fix formatting, improve documentation

* Fix formatting and linting

* Fix isort errors

* Fix formatting using yapf and add new lines

* Remove new lines

* Remove new line

* Fix import order errors

* Remove trailing newline

Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>
2 days agoPubsubIO - Improve limit validations to consider attributes (#23023)
Bruno Volpato [Thu, 29 Sep 2022 15:16:04 +0000 (11:16 -0400)] 
PubsubIO - Improve limit validations to consider attributes (#23023)

* PubsubIO - Improve limits validation to consider attributes

Consider the overhead that the proto will have for each attribute

* Fix: do not attempt to publish an empty batch to Pub/Sub

2 days agoFix Small pytorch notebook bug fix (#23407)
Danny McCormick [Thu, 29 Sep 2022 14:14:53 +0000 (10:14 -0400)] 
Fix Small pytorch notebook bug fix (#23407)

3 days agoUpdate state cache to not fail when measuring object sizes. (#23391) nightly-refs/heads/master
Lukasz Cwik [Thu, 29 Sep 2022 01:03:35 +0000 (18:03 -0700)] 
Update state cache to not fail when measuring object sizes. (#23391)

* Update state cache to not fail when measuring object sizes.

Internal testing within Google discovered that weakref.proxy causes isinstance to raise an Exception and that certain Pandas types causes sys.getsizeof to fail.

See https://github.com/apache/beam/issues/23389 and https://github.com/liran-funaro/objsize/issues/6 for additional details.

* Fix lint and mypy

* More lint fixes

3 days agoAdd relevant docs to Cloud Profiler exceptions. (#23404)
Sam sam [Thu, 29 Sep 2022 00:02:38 +0000 (17:02 -0700)] 
Add relevant docs to Cloud Profiler exceptions. (#23404)

Co-authored-by: Sam Rohde <srohde@google.com>
3 days agoSupport google-cloud-spanner v3 and fixes broken unit tests (#23365)
Yi Hu [Wed, 28 Sep 2022 23:39:41 +0000 (19:39 -0400)] 
Support google-cloud-spanner v3 and fixes broken unit tests (#23365)

3 days agoRegenerate Go Protos (#23408)
Robert Burke [Wed, 28 Sep 2022 22:54:38 +0000 (15:54 -0700)] 
Regenerate Go Protos (#23408)

3 days agoAdd documentation link to the interactive environment 23409/head
Sam Rohde [Wed, 28 Sep 2022 22:43:19 +0000 (15:43 -0700)] 
Add documentation link to the interactive environment

3 days agoAdds support in Samza Runner to run DoFn.processElement in parallel inside Samza...
Xinyu Liu [Wed, 28 Sep 2022 22:20:36 +0000 (15:20 -0700)] 
Adds support in Samza Runner to run DoFn.processElement in parallel inside Samza tasks (#23313)

3 days ago[Tour Of Beam] API adjustments (#23349)
Evgeny Antyshev [Wed, 28 Sep 2022 16:37:43 +0000 (19:37 +0300)] 
[Tour Of Beam] API adjustments (#23349)

* sdk

* use sample/api

* sdk_list.json

* nits

* title

* fix integration_tests

* unit/module id

* sdks

* unitId->id in param

* id/title fix

* empty

* CORS

* optimize

* Update sdk.go

* fixing format error

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
4 days agoPin objsize version to avoid regression in 0.6.0 (#23396)
Lukasz Cwik [Wed, 28 Sep 2022 05:29:44 +0000 (22:29 -0700)] 
Pin objsize version to avoid regression in 0.6.0 (#23396)

* Pin objsize version to avoid regression in 0.6.0

tox -c tox.ini -e py37 -- apache_beam/transforms/stats_test.py::ApproximateUniqueTest_0

fails with:

objsize.py
    exclude_set.update(id(vars(m)) for m in list(sys.modules.values()))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <list_iterator object at 0x7fbe55e89410>

>   exclude_set.update(id(vars(m)) for m in list(sys.modules.values()))
E   TypeError: vars() argument must have __dict__ attribute [while running 'Globally/CombineGlobally(ApproximateQuantilesCombineFn)/InjectDefault']

Fixes #23395

* Add an upper bound to objsize.

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
4 days agoAdd ISSUE#23071 to CHANGES.md (#23297)
Iñigo San Jose Visiers [Wed, 28 Sep 2022 00:45:10 +0000 (19:45 -0500)] 
Add ISSUE#23071 to CHANGES.md (#23297)

* Add ISSUE#23071 to CHANGES.md

* Update CHANGES.md

Co-authored-by: Brian Hulette <hulettbh@gmail.com>
* Revert "Add ISSUE#23071 to CHANGES.md"

This reverts commit 3671aea254b4ffba45c90801525ffef195db3d63.

* Add ISSUE#23071 to CHANGES.md

* Add ISSUE#23071 to CHANGES.md

Co-authored-by: Brian Hulette <hulettbh@gmail.com>
4 days agoMerge pull request #23370: Make identifier of Date/Time/DateTime to be a public stati...
Andrew Pilloud [Tue, 27 Sep 2022 21:57:14 +0000 (14:57 -0700)] 
Merge pull request #23370: Make identifier of Date/Time/DateTime to be a public static field

4 days agoset upper bound on google-cloud-profiler (#23354)
Anand Inguva [Tue, 27 Sep 2022 17:11:39 +0000 (13:11 -0400)] 
set upper bound on google-cloud-profiler (#23354)

Co-authored-by: Anand Inguva <anandinguva@google.com>
4 days agoRunInference Benchmarks UI (#23371)
Anand Inguva [Tue, 27 Sep 2022 15:38:52 +0000 (11:38 -0400)] 
RunInference Benchmarks UI (#23371)

* Add UI for RunInference Benchmarks

* Add a sample json for ML Beam RunInference Benchmarks

* Change interval to 24 hours

* Change name

* Add Mean to the DistributionResult

* Add mean metrics to the dashboard instead of count

4 days agoAdd a tensorflow example to the run_inference_basic notebook (#23173)
Ryan Thompson [Tue, 27 Sep 2022 11:42:16 +0000 (07:42 -0400)] 
Add a tensorflow example to the run_inference_basic notebook (#23173)

* Add a tensorflow example to the run_inference_basic notebook

* added tensorflow notebook example

* updated first comment

* added license

* added sklearn notebook

* added pytorch example

* added readme

* added notebooks for various frameworks

* updated notebooks and readme

* Update examples/notebooks/beam-ml/README.md

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Added keyed handler, fixed formatting of floats to limit to 2 digits

* Update examples/notebooks/beam-ml/run_inference_sklearn.ipynb

good catch

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Update examples/notebooks/beam-ml/README.md

* Update examples/notebooks/beam-ml/README.md

Co-authored-by: Danny McCormick <dannymccormick@google.com>
5 days agoFix TextSource incorrect handling in channels that return short reads. (#23376)
Lukasz Cwik [Tue, 27 Sep 2022 03:31:30 +0000 (20:31 -0700)] 
Fix TextSource incorrect handling in channels that return short reads. (#23376)

* Fix TextSource incorrect handling in channels that return short reads.

The issue is that readDefaultLine and readCustomLine was incorrectly calculating the appendLength when the buffer returned was 0 length.
This was solved by ensuring that the internal read loop always read at least one byte allowing for the code to ensure that we were making progress. For readDefaultLine we kept track of whether we need to skip an LF in the next buffer if the current buffer ended with a CR and for readCustomLine we had to remember how much of the delimiter we have read so far in this buffer.

The bug was introduced in https://github.com/apache/beam/commit/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c

There was no noticeable change in the TextSourceBenchmark performance results.

Fixes #23375

5 days agoChanges CoGroupByKey typehint from List to Iterable (#22984)
Ryan Thompson [Tue, 27 Sep 2022 02:50:04 +0000 (22:50 -0400)] 
Changes CoGroupByKey typehint from List to Iterable (#22984)

Co-authored-by: Lukasz Cwik <lcwik@google.com>
5 days ago[#22478]: Add read_time support to Google Firestore connector (#22966)
yixiaoshen [Tue, 27 Sep 2022 00:39:32 +0000 (00:39 +0000)] 
[#22478]: Add read_time support to Google Firestore connector (#22966)

* [#22478]: Add read_time support to Google Firestore connector

* address review comments

* address style issues

* fix style issues

* fix style issues

5 days agoStack Trace Decoration for Beam Samza Runner (#23221)
Sanil Jain [Mon, 26 Sep 2022 21:06:26 +0000 (14:06 -0700)] 
Stack Trace Decoration for Beam Samza Runner (#23221)

5 days agoUpgraded Flink on Dataproc support from Interacitve Beam
Ning Kang [Mon, 26 Sep 2022 20:55:25 +0000 (13:55 -0700)] 
Upgraded Flink on Dataproc support from Interacitve Beam

* Added parallelism and max_parallelism Flink pipeline options.
* Supported high parallelism and custom container on cloud container registry.
* Fixed a bug.

5 days agoGA Migration PreCommit and PostCommit Tables in CI.md (#23372)
Fernando Morales [Mon, 26 Sep 2022 19:28:52 +0000 (13:28 -0600)] 
GA Migration PreCommit and PostCommit Tables in CI.md (#23372)

* adding precommit and postcommit tables

* reflecting inclusion of workflow_dispatch for precommits

* Update CI.md

5 days agoUpgraded Flink on Dataproc support from Interacitve Beam 23271/head
KevinGG [Fri, 9 Sep 2022 19:13:07 +0000 (12:13 -0700)] 
Upgraded Flink on Dataproc support from Interacitve Beam

* Added initialization actions to configure docker credsHelper with gcloud.
* Improved taskmanager configurations for higher parallelism.
* Fixed a cluster metadata bug that always creates a new cluster instead
  of reusing the lastly used one if region is not set.
* Added parallelism and max_parallelism Flink pipeline options.
* Changed how Flink-YARN session is started to take effect of
  configuration changes as well as where to parse Flink Master.

5 days agoBump org.nosphere.apache.rat from 0.7.0 to 0.8.0 (#23330)
dependabot[bot] [Mon, 26 Sep 2022 14:01:02 +0000 (10:01 -0400)] 
Bump org.nosphere.apache.rat from 0.7.0 to 0.8.0 (#23330)

Bumps org.nosphere.apache.rat from 0.7.0 to 0.8.0.

---
updated-dependencies:
- dependency-name: org.nosphere.apache.rat
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5 days agomake identifier of Time to be a public static field 23370/head
shaojwu [Mon, 26 Sep 2022 13:11:14 +0000 (21:11 +0800)] 
make identifier of Time to be a public static field

5 days agomake identifier of Date&DateTime to be a public static field
shaojwu [Mon, 26 Sep 2022 13:06:20 +0000 (21:06 +0800)] 
make identifier of Date&DateTime to be a public static field

6 days agoBump Java FnApi Container version to beam-master-20220923 (#23352)
Yi Hu [Sun, 25 Sep 2022 20:25:41 +0000 (16:25 -0400)] 
Bump Java FnApi Container version to beam-master-20220923 (#23352)

7 days agoExtract playground components (#23253)
alexeyinkin [Sat, 24 Sep 2022 19:45:33 +0000 (23:45 +0400)] 
Extract playground components (#23253)

* [Tour of Beam][Frontend][#22600] TourScreen layout

* theme setup

* Replaced ThemeProvider with ThemeSwitchNotifier

* header with theme mode switcher and logo

* page container with header & footer

* theme mode tests

* renamed the directory to tour-of-beam

* compressed beam_logo.png

* added missing license comments

* rudimentary layout of the first screen

* review comments fixes #1

* moved notifyListeners inside then

* responsive todo

* split into 2 simple functions

* deleted redundant constants &
replaced 2018 text theme with 2021

* styling refinement

* home screen layout

* clickable sign in text

* font weights fix

* removed _getBaseFontTheme function

* fixed border and bg color

* color fixes

* difficulty component

* _LastModuleBody

* todo in test

* footer border

* fixed overflows

* replaced Project prefix with Tob

* replaced then with await

* inferred type

* started translation of the home screen

* sorted translations

* Complexity comments

* comment fixes

* home screen translations

* sign in overlay

* import fix

* integration test does not fail

* playground_components package with
dismissible_overlay

* missing license

* removed _dots from build

* widgets refinement

* renamed home screen to welcome screen

* deleted copyWith

* _SdkButton

* trailing comma & pubspec formatting

* license and lints

* license

* removed license from .metadata

* pubspec formatting

* total lints update

* changed from tour_of_beam to
tour-of-beam in build.gradle.kts

* license check

* _SdkButton mimics Radio button

* renamed MyApp to TourOfBeamApp

* onChanged mimics Radio button

Tour of Beam frontend blank project

[Tour of Beam][Frontend][#22600] TourScreen layout

TourScreen layout (#22600)

common theme, constants, split view

missing license

flutter_gen, summary layout details

content layout details

no functional widgets in split view

main screen todos & translation

main screen todos & translation

comment fixes #1

ExpansionTileWrapper

SplitViewController

lists in tour screen widgets

comment fixes #1 (31.08)

split view package in PGC

fixed button overflow

splitter theme color

comment fixes #2 (31.08)

gradlew check

welcome screen overflow test (#22600)

SDK dropdown (#22600)

flexible complete unit OutlinedButton (#22600)

renamed PageContainer to TobScaffold

dropdown style refinement

DropdownButton implicit type

sdk instead of e

licenses #22600

renamed _ShrinkedTour to _NarrowTour #22600

tour screen style refinement #22600

BeamDivider in PGC #22600

removed todo, added license #22600

built with text #22600

_WideWelcome with IntrinsicHeight (#22600)

Co-Authored-By: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>
* addressing review comments #22600

replaced magic numbers #22600

comments (#22600)

comments #22600

comments #22600

comments #22600

comments #22600

comments, flutter 3.3.0 upgrade #22600

renamed ActionPadding to ActionVerticalPadding #22600

actions formatting #22600

* branded sign in buttons #22600

* _BrandedSignInButtons #22600

* _Divider color #22600

* profile #22600

* moved split_view from PGC into ToB #22600

* indentation fix #22600

* split ProfileContent into widgets #22600

* Extract playground components to a separate package (#22600)

* Minor fixes (#22600)

* Address review issues (#22600)

* Upgrade Flutter to v3.3.2 (#22600)

* Add precommit Gradle task for playground_components, add code generation to frontend Gradle task, remove generated mocks, fix linter issues (#22600)

* startTour button (#22600)

* lint fixes (#22600)

* Fix highlighting for Python and SCIO (#22600)

Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>
8 days agoBump cloud.google.com/go/bigquery from 1.41.0 to 1.42.0 in /sdks (#23329)
dependabot[bot] [Fri, 23 Sep 2022 19:15:25 +0000 (15:15 -0400)] 
Bump cloud.google.com/go/bigquery from 1.41.0 to 1.42.0 in /sdks (#23329)

Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.41.0 to 1.42.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.41.0...bigquery/v1.42.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/bigquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8 days agolint fixes to go (#23351)
Ritesh Ghorse [Fri, 23 Sep 2022 17:48:13 +0000 (13:48 -0400)] 
lint fixes to go (#23351)

8 days agoBump cloud.google.com/go/storage from 1.26.0 to 1.27.0 in /sdks (#23336)
dependabot[bot] [Fri, 23 Sep 2022 15:20:06 +0000 (08:20 -0700)] 
Bump cloud.google.com/go/storage from 1.26.0 to 1.27.0 in /sdks (#23336)

Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.26.0...spanner/v1.27.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8 days agoMerge pull request #23337: Fix Nexmark default log level
Alexey Romanenko [Fri, 23 Sep 2022 11:06:08 +0000 (13:06 +0200)] 
Merge pull request #23337: Fix Nexmark default log level

9 days agoFix Nexmark default log level 23337/head
Moritz Mack [Fri, 23 Sep 2022 06:45:26 +0000 (08:45 +0200)] 
Fix Nexmark default log level

9 days agoChange google_cloud_bigdataoss_version to 2.2.8. (#23300)
scwhittle [Thu, 22 Sep 2022 20:14:03 +0000 (22:14 +0200)] 
Change google_cloud_bigdataoss_version to 2.2.8. (#23300)

9 days agoMerge pull request #23214: Use avro DataFileReader to read avro container files
Ismaël Mejía [Thu, 22 Sep 2022 16:58:10 +0000 (18:58 +0200)] 
Merge pull request #23214: Use avro DataFileReader to read avro container files

9 days agoImproved pipeline translation in SparkStructuredStreamingRunner (#22446)
Moritz Mack [Thu, 22 Sep 2022 10:30:03 +0000 (12:30 +0200)] 
Improved pipeline translation in SparkStructuredStreamingRunner (#22446)

* Closes #22445: Improved pipeline translation in SparkStructuredStreamingRunner (also closes #22382)

10 days agoMerge pull request #23294: SpannerIO - Do not use .get() on ValueProvider during...
Alexey Romanenko [Thu, 22 Sep 2022 08:46:30 +0000 (10:46 +0200)] 
Merge pull request #23294: SpannerIO - Do not use .get() on ValueProvider during pipeline creation

10 days agoAllow longer Class-Path entries (#23269)
Shane Hansen [Wed, 21 Sep 2022 20:12:47 +0000 (13:12 -0700)] 
Allow longer Class-Path entries (#23269)

jar Class-Path entries have a max line length which, when
exceeded, will cause java to fail to start. Use newline plus
extra space to support large numbers of jar files.

10 days agoExtend protocol in windmill.proto used by google-cloud-dataflow-java runner. (#23256)
arne-alex [Wed, 21 Sep 2022 19:23:01 +0000 (21:23 +0200)] 
Extend protocol in windmill.proto used by google-cloud-dataflow-java runner. (#23256)

10 days ago[Java SDK core] emit watermark from PeriodicSequence (#23301) (#23302)
Jan Lukavský [Wed, 21 Sep 2022 17:12:27 +0000 (19:12 +0200)] 
[Java SDK core] emit watermark from PeriodicSequence (#23301) (#23302)

11 days agoDo not use .get() on ValueProvider during pipeline creation 23294/head
Bruno Volpato [Mon, 19 Sep 2022 17:56:37 +0000 (13:56 -0400)] 
Do not use .get() on ValueProvider during pipeline creation

12 days ago[BEAM-14378] [CdapIO] SparkReceiverIO Read via SDF (#17828)
Vitaly Terentyev [Tue, 20 Sep 2022 03:50:13 +0000 (07:50 +0400)] 
[BEAM-14378] [CdapIO] SparkReceiverIO Read via SDF (#17828)

* [BEAM-14378] Add SparkReceiverIO

* Fix comments

* Resolve comments

* Fix checkstyle

* Add TestOutputDoFn's result set size assertion

Co-authored-by: akashorabek <aka.shorabek01@gmail.com>
12 days ago[BEAM-13416] Introduce Schema provider for AWS models and deprecate low level coders...
Moritz Mack [Mon, 19 Sep 2022 19:31:08 +0000 (21:31 +0200)] 
[BEAM-13416] Introduce Schema provider for AWS models and deprecate low level coders (#16947)

12 days ago[Playground] [Backend] Cache component for playground examples (#22869)
Vladislav Chunikhin [Mon, 19 Sep 2022 17:09:24 +0000 (21:09 +0400)] 
[Playground] [Backend] Cache component for playground examples (#22869)

* [Playground] [Backend] added the datastore query to get an example catalog

* [Playground] [Backend] added datastore queries to get precompiled objects

* [Playground] [Backend] added cache component for examples

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] refactoring for datastore db

* [Playground] [Backend] minor refactoring after review

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* [Playground] [Backend] updated complexity field

* squash! [Playground] [Backend] updated complexity field

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] fixed integration tests for datastore

* [Playground] [Backend] added testing data cleaner

* [Playground] [Backend] refactoring for integration tests

* [Playground] [Backend] added one more gradle task to run tests without cache

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited getting datastore key

* [Playground] [Backend] updated unit tests for diferent namespaces

* [Playground] [Backend] edited cache component tests

* [Playground] [Infrastructure] added the datastore client

* [Playground] [Infrastructure] added the type property to an example entity

* [Playground] [Infrastructure] removed the cloud storage client

* [Playground] [Infrastructure] changed validation examples for duplicates by name

* [Playground] [Infrastructure] fixed duplicated example names

* [Playground] [Infrastructure] added the google cloud project id as an env variable

* [Playground] [Backend] updated docker files for runners

* [Playground] [Backend] added SDK validation to save a code snippet

* [Playground] [Backend] removed comments

* [Playground] [Backend] resolved comments after review

* [Playground] [Infrastructure] updated dependency versions

* [Playground] [Backend] fixed the java runner for examples with graphs

* [Playground] [Infrastructure] added loading catalogs to the cloud datastore

* [Playground] [Backend] remove extra imports after merging

* [Playground] [Backend] fixes after merging with other branches

* [Playground] [Backend] added genreics for one method

* [Playground] [Backend] fixed tests after merging with a master

Co-authored-by: oborysevych <oleg.borisevich@akvelon.com>
12 days agoMerge pull request #23265 from pabloem/rv-pbt
Pablo [Mon, 19 Sep 2022 17:07:42 +0000 (13:07 -0400)] 
Merge pull request #23265 from pabloem/rv-pbt

Revert "Trying out property-based tests for Beam python coders (#22233)"

12 days agoFix wrong comment (#23272)
coldWater [Mon, 19 Sep 2022 17:04:16 +0000 (01:04 +0800)] 
Fix wrong comment (#23272)

12 days ago[Go SDK] Add timer coder support (#23222)
Ritesh Ghorse [Mon, 19 Sep 2022 16:50:06 +0000 (12:50 -0400)] 
[Go SDK] Add timer coder support (#23222)

12 days agoBump google.golang.org/api from 0.95.0 to 0.96.0 in /sdks (#23246)
dependabot[bot] [Mon, 19 Sep 2022 16:34:47 +0000 (09:34 -0700)] 
Bump google.golang.org/api from 0.95.0 to 0.96.0 in /sdks (#23246)

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.95.0 to 0.96.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.95.0...v0.96.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
13 days agoEnable verbose output for RAT Precommit (#23279)
Danny McCormick [Sun, 18 Sep 2022 23:40:23 +0000 (19:40 -0400)] 
Enable verbose output for RAT Precommit (#23279)

* Enable verbose output

* TEST - Remove license header

* Remove arbitrary header

* Add back licenses

* Add description to RAT on Readme

* Fix spurious diff

2 weeks agoAdd drop_example flag to the RunInference and Model Handler (#23266)
Anand Inguva [Sun, 18 Sep 2022 02:16:57 +0000 (22:16 -0400)] 
Add drop_example flag to the RunInference and Model Handler (#23266)

* Add drop_example flag to the RunInference and Model Handler

* Pass drop_example to the _convert_to_result

* Refactor _convert_to_result

* Return _convert_to_result

* Fixup lint

* Code update based on Suggestions

* Refactor class name

* Refactor TensorRT to add drop_example

* Add typing.Optional to the type hint for drop_example

* Add _convert_to_result for tensorRT run_inference

* fixup lint

2 weeks agoupdated the pydoc for running a custom model on Beam (#23218)
liferoad [Sat, 17 Sep 2022 12:12:35 +0000 (08:12 -0400)] 
updated the pydoc for running a custom model on Beam (#23218)

* updated the pydoc for running a custom model on Beam

* Update website/www/site/content/en/documentation/sdks/python-machine-learning.md

Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>
* Update website/www/site/content/en/documentation/sdks/python-machine-learning.md

Co-authored-by: Danny McCormick <dannymccormick@google.com>
Co-authored-by: XQ Hu <xqhu@google.com>
Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>
2 weeks agoFix Kafka performance test sourceOption to match expected hash (#23274)
Yi Hu [Fri, 16 Sep 2022 18:17:27 +0000 (14:17 -0400)] 
Fix Kafka performance test sourceOption to match expected hash (#23274)

* Fix Kafka performance test expected hash

* Revert "Fix Kafka performance test expected hash"

This reverts commit e9b22494127def02a551f4feda8d5c3fabe9adcc.

* Unify key size of performance test and precommit test

2 weeks agoTensorRT Initial commit (#22131)
azhurkevich [Fri, 16 Sep 2022 16:37:38 +0000 (12:37 -0400)] 
TensorRT Initial commit (#22131)

* Initial commit

* Format

* Fix read of onnx file

* TensorRT Object Detection example

* Fix copyright; Add back pycuda

* Addressing noted and resolved comments

* Addressing noted and resolved comments

* Keeping mem alloc out of inference call + CUDA Python replaced PyCUDA + inference calls are now asynchronous

* Addressing comments

* Fixing header

* Format and lint

* Format and lint tensorrt example

* Sort imports

* Format docstrings

* Addressing some new comments

* Refactor tensorrt imports

* Address PR comment

* Adding stream sync

* Adding stream sync

* Getting rid of optional argument validation

* Setting experimental status

* Raising exception intead of exiting

* Update sdks/python/apache_beam/examples/inference/README.md

Co-authored-by: Andy Ye <andyye333@gmail.com>
* Update sdks/python/apache_beam/examples/inference/README.md

Co-authored-by: Andy Ye <andyye333@gmail.com>
* gradle task for tensor RT example E2E test

* Update input path and docker image

* Add no_use_multiple_sdk_containers

* Add disk_size_gb for the example to work on Dataflow with Custom Image

* Add inference test to Python 3.8. Also add TODO for py37, py39

* Apply suggestions from code review

* Fix lint issues

* Guard TensorRT context with a reentrant lock.

* Add dockerfile

* Apache license

* Apache license

Co-authored-by: Andy Ye <andyye333@gmail.com>
Co-authored-by: Anand Inguva <anandinguva98@gmail.com>
Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>
Co-authored-by: Valentyn Tymofieiev <valentyn@google.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>
2 weeks ago[Website] Correct spelling of structural (#23225)
Shane Hansen [Fri, 16 Sep 2022 13:19:19 +0000 (06:19 -0700)] 
[Website] Correct spelling of structural (#23225)

sturctural -> structural

2 weeks agoBump cloud.google.com/go/bigquery from 1.40.0 to 1.41.0 in /sdks (#23247)
dependabot[bot] [Fri, 16 Sep 2022 13:01:01 +0000 (09:01 -0400)] 
Bump cloud.google.com/go/bigquery from 1.40.0 to 1.41.0 in /sdks (#23247)

Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.40.0 to 1.41.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.40.0...bigquery/v1.41.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/bigquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks agoIssue#21430 Avoid pruning DataframeTransforms (#23069)
Ning Kang [Fri, 16 Sep 2022 01:15:50 +0000 (18:15 -0700)] 
Issue#21430 Avoid pruning DataframeTransforms (#23069)

Mitigated Issue#21430

* Avoided the pruning of composite transforms for Beam dataframe.
* Added an interactive collect merged DeferredDataFrame unittest

2 weeks agoRevert "Trying out property-based tests for Beam python coders (#22233)" 23265/head
Pablo E [Thu, 15 Sep 2022 21:27:49 +0000 (14:27 -0700)] 
Revert "Trying out property-based tests for Beam python coders (#22233)"

This reverts commit 9be9a43c1b85be9b2f78e2943f6092ccb88e13b4.

2 weeks agoImprove the performance of TextSource by reducing how many byte[]s are copied (fixes...
Lukasz Cwik [Thu, 15 Sep 2022 21:00:50 +0000 (14:00 -0700)] 
Improve the performance of TextSource by reducing how many byte[]s are copied (fixes #23193) (#23196)

* Improve the performance of TextSource by reducing how many byte[]s are copied (fixes #23193)

This makes TextSource take about 2.3x less CPU resources during decoding.

Before this change:
```
TextSourceBenchmark.benchmarkTextSource        thrpt    5  0.248 ± 0.029  ops/s
```

After this change:
```
TextSourceBenchmark.benchmarkHadoopLineReader  thrpt    5  0.465 ± 0.064  ops/s
TextSourceBenchmark.benchmarkTextSource        thrpt    5  0.575 ± 0.059  ops/s
```

* Write file in pieces instead of pre-allocating entire buffer

* Address PR comments

2 weeks agoBump up dataflow python container version to beam-master-20220914 (#23238)
Yi Hu [Thu, 15 Sep 2022 16:49:55 +0000 (12:49 -0400)] 
Bump up dataflow python container version to beam-master-20220914 (#23238)

2 weeks agoFix outdated code in python sdk install (#23231)
Yi Hu [Thu, 15 Sep 2022 16:49:26 +0000 (12:49 -0400)] 
Fix outdated code in python sdk install (#23231)

2 weeks agoRevert "Exclude protobuf 3.20.2" (#23237)
Yi Hu [Thu, 15 Sep 2022 16:47:54 +0000 (12:47 -0400)] 
Revert "Exclude protobuf 3.20.2" (#23237)

2 weeks agoTest fix Kafka Performance test batch (#23191)
Yi Hu [Thu, 15 Sep 2022 15:02:18 +0000 (11:02 -0400)] 
Test fix Kafka Performance test batch (#23191)

* Fix topic not exist

* Tune test parameters to avoid time out

2 weeks agoMerge pull request #22968: [Website] update site navigation #22902
Alexey Romanenko [Thu, 15 Sep 2022 14:54:35 +0000 (16:54 +0200)] 
Merge pull request #22968: [Website] update site navigation #22902

2 weeks agoGA migration - Base actions to use for precommit and postcommit workflows (#23109)
Benjamin Gonzalez [Thu, 15 Sep 2022 14:27:03 +0000 (09:27 -0500)] 
GA migration - Base actions to use for precommit and postcommit workflows (#23109)

* Base actions to use for precommit and postcommit workflows

* Adding Java, Flink, Spark and Golang supported versions to json file

* switching to temurin due to Apache License compatibility

* setting go version to 1.18

* Add environment as optionals in setup-base-action

Co-authored-by: Fernando Morales <fernando.morales@wizeline.com>
Co-authored-by: Fernando Morales <80284146+fernando-wizeline@users.noreply.github.com>
2 weeks ago[GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows...
Daniela Martín [Thu, 15 Sep 2022 14:17:02 +0000 (09:17 -0500)] 
[GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows) (#23158)

* Adding all changes under gh-actions-self-hosted-runners folder in PR#22703 in order to create a new PR only with Infra related work.

* Updated GH Actions Runner version to 2.296.1. Added github-actions-vpa file.

* Updating auth variable to use string interpolation.

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Code refactored for beamRunners filtering.

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Code refactored when looking for beam runners. Using find function instead of inner conditions.

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Updating self-hosted-linux README file.

Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Implementing PR#23158 requested changes

* Implementing Review Changes

* Refactoring to use find instead of plain loops and adding env var validation

* Adding REQUIRED_ENV_VARS as a constant

Co-authored-by: Elias Segundo <elias.segundo@luisrazo.local>
* Adding REQUIRED_ENV_VARS as a shared constant for the helper Cloud Functions.

* Implementing PR#23158 requested changes: octokit paginate

* Using octokit paginate

Co-authored-by: Elias Segundo <elias.segundo@luisrazo.local>
* Updating self-hosted runner version to v2.296.2.

Co-authored-by: Danny McCormick <dannymccormick@google.com>
Co-authored-by: Elias Segundo Antonio <eliassegundo.segundo@gmail.com>
Co-authored-by: Elias Segundo <elias.segundo@luisrazo.local>
2 weeks agoPass namespace through RunInference transform (#23182)
Anand Inguva [Thu, 15 Sep 2022 13:13:47 +0000 (09:13 -0400)] 
Pass namespace through RunInference transform (#23182)

* Pass namespace through RunInference transform

* Refactor namespace to metrics_namespace

* Update doc string
Testing an example with a custom namespace

* Add a stage name to the base RunInferene DoFn

* Unit test for custom namespace
Revert "Testing an example with a custom namespace"

This reverts commit 39163fdc311ba87403754a453efad008e48fbf77.
Update sdks/python/apache_beam/ml/inference/base_test.py

* Refactor RunInference default namespace

* Fixup unit test

* Refactor ModelHandler default namespace for consistency

2 weeks agoOptimize varint reading and writing for small ints. (#23192)
Robert Bradshaw [Thu, 15 Sep 2022 00:34:48 +0000 (17:34 -0700)] 
Optimize varint reading and writing for small ints. (#23192)

This is used as a primitive for several other coders.

Before:

tiny_row, RowCoder, 1000 element(s)    : p. element median time cost: 7.22051e-07 sec, relative std: 12.45%
large_row, RowCoder, 1000 element(s)   : p. element median time cost: 3.12304e-06 sec, relative std: 5.01%
nullable_row, RowCoder, 1000 element(s): p. element median time cost: 3.27158e-06 sec, relative std: 6.49%
diverse_row, RowCoder, 1000 element(s) : p. element median time cost: 2.33555e-06 sec, relative std: 7.48%
small_int, FastPrimitivesCoder, 1000 element(s)                   : p. element median time cost: 1.66655e-07 sec, relative std: 9.18%
large_int, FastPrimitivesCoder, 1000 element(s)                   : p. element median time cost: 3.39985e-07 sec, relative std: 3.05%
small_int, LengthPrefixCoder[FastPrimitivesCoder], 1000 element(s): p. element median time cost: 5.70536e-07 sec, relative std: 4.27%
small_list, FastPrimitivesCoder, 1000 element(s)               : p. element median time cost: 1.57607e-06 sec, relative std: 3.05%
large_list, FastPrimitivesCoder, 1000 element(s)               : p. element median time cost: 4.76551e-05 sec, relative std: 4.31%
small_list, IterableCoder[FastPrimitivesCoder], 1000 element(s): p. element median time cost: 1.59943e-06 sec, relative std: 2.11%
large_list, IterableCoder[FastPrimitivesCoder], 1000 element(s): p. element median time cost: 4.66548e-05 sec, relative std: 3.59%

After:

tiny_row, RowCoder, 1000 element(s)    : p. element median time cost: 6.68406e-07 sec, relative std: 13.95%
large_row, RowCoder, 1000 element(s)   : p. element median time cost: 2.88701e-06 sec, relative std: 6.26%
nullable_row, RowCoder, 1000 element(s): p. element median time cost: 3.19052e-06 sec, relative std: 8.06%
diverse_row, RowCoder, 1000 element(s) : p. element median time cost: 2.24805e-06 sec, relative std: 6.15%
small_int, FastPrimitivesCoder, 1000 element(s)                   : p. element median time cost: 1.65582e-07 sec, relative std: 13.75%
large_int, FastPrimitivesCoder, 1000 element(s)                   : p. element median time cost: 3.14236e-07 sec, relative std: 6.83%
small_int, LengthPrefixCoder[FastPrimitivesCoder], 1000 element(s): p. element median time cost: 5.85556e-07 sec, relative std: 4.98%
small_list, FastPrimitivesCoder, 1000 element(s)               : p. element median time cost: 1.48404e-06 sec, relative std: 42.11%
large_list, FastPrimitivesCoder, 1000 element(s)               : p. element median time cost: 4.77622e-05 sec, relative std: 4.20%
small_list, IterableCoder[FastPrimitivesCoder], 1000 element(s): p. element median time cost: 1.50216e-06 sec, relative std: 1.90%
large_list, IterableCoder[FastPrimitivesCoder], 1000 element(s): p. element median time cost: 4.57406e-05 sec, relative std: 3.11%

2 weeks agoUpdate google-cloud-bigquery requirement from <3,>=1.6.0 to >=1.6.0,<4 in /sdks/pytho...
dependabot[bot] [Wed, 14 Sep 2022 21:27:13 +0000 (14:27 -0700)] 
Update google-cloud-bigquery requirement from <3,>=1.6.0 to >=1.6.0,<4 in /sdks/python (#22378)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
Co-authored-by: Valentyn Tymofieiev <valentyn@google.com>
2 weeks agoBump github.com/google/go-cmp from 0.5.8 to 0.5.9 in /sdks (#23123)
dependabot[bot] [Wed, 14 Sep 2022 20:39:44 +0000 (16:39 -0400)] 
Bump github.com/google/go-cmp from 0.5.8 to 0.5.9 in /sdks (#23123)

Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](https://github.com/google/go-cmp/compare/v0.5.8...v0.5.9)

---
updated-dependencies:
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks agoBump cloud.google.com/go/bigquery from 1.39.0 to 1.40.0 in /sdks (#23122)
dependabot[bot] [Wed, 14 Sep 2022 20:11:59 +0000 (16:11 -0400)] 
Bump cloud.google.com/go/bigquery from 1.39.0 to 1.40.0 in /sdks (#23122)

Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.39.0 to 1.40.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.39.0...bigquery/v1.40.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/bigquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks agoResolve script parsing error when changing from bash to sh. (#23199)
Robert Burke [Wed, 14 Sep 2022 19:52:12 +0000 (12:52 -0700)] 
Resolve script parsing error when changing from bash to sh. (#23199)

* Update cut_release_branch.sh

* Remove space.git

* Update verify_release_build.sh

* Rm space change

2 weeks ago[Website] update site navigation #22902 22968/head
bulat safiullin [Wed, 31 Aug 2022 11:40:58 +0000 (17:40 +0600)] 
[Website] update site navigation  #22902

2 weeks agoUpdate to newest version of CloudPickle. (#23223)
tvalentyn [Wed, 14 Sep 2022 18:39:01 +0000 (11:39 -0700)] 
Update to newest version of CloudPickle. (#23223)

2 weeks ago(BQ Python) Perform job waits in finish_bundle to allow BQ streaming writes with...
Ahmed Abualsaud [Wed, 14 Sep 2022 18:26:59 +0000 (14:26 -0400)] 
(BQ Python) Perform job waits in finish_bundle to allow BQ streaming writes with large batch loads (#23012)

* remove WaitForBQJobs and perform waits at each step's finish_bundle. copy jobs will provide trigger for delete stage

* clarify why we emit a None value when no schema updates needed

* added test for copy jobs

* add test for dynamic destination streaming

* properly check results with matchers

* style fixes

* fixing test_one_job... test

* fixing test_wait_for_job_completion

* yield copy job references instead. having trouble returning and yielding to separate outputs in the same finish_bundle

* style fixes

2 weeks agoAdd section to docs on resource hints/RunInference (#23215)
Danny McCormick [Wed, 14 Sep 2022 17:07:40 +0000 (13:07 -0400)] 
Add section to docs on resource hints/RunInference (#23215)

* Add section to docs on resource hints/RunInference

* Formatting + wording

* Fix link

2 weeks agoUpdate cibuildwheel (#23024)
Anand Inguva [Wed, 14 Sep 2022 17:02:00 +0000 (13:02 -0400)] 
Update cibuildwheel (#23024)

2 weeks agoMerge pull request #23229: Fix assignees check in alert email
Kenn Knowles [Wed, 14 Sep 2022 16:18:58 +0000 (09:18 -0700)] 
Merge pull request #23229: Fix assignees check in alert email