pulsar.git
29 min ago[improve][broker] Add a message to a NullPointerException created in ManagedLedgerImp... master
Lari Hotari [Wed, 28 Sep 2022 08:32:07 +0000 (11:32 +0300)] 
[improve][broker] Add a message to a NullPointerException created in ManagedLedgerImpl (#17293)

- a NPE with no description is confusing

49 min ago[cleanup][broker][Modernizer] fix violations in pulsar-broker (#17691)
youzipi [Wed, 28 Sep 2022 08:12:43 +0000 (16:12 +0800)] 
[cleanup][broker][Modernizer] fix violations in pulsar-broker (#17691)

Co-authored-by: Marvin Cai <cai19930303@gmail.com>
4 hours ago[fix][flaky-test]Delete PersistentSubscriptionTest.testCanAcknowledgeAndCommitForTran...
fengyubiao [Wed, 28 Sep 2022 04:19:47 +0000 (12:19 +0800)] 
[fix][flaky-test]Delete PersistentSubscriptionTest.testCanAcknowledgeAndCommitForTransaction (#17845)

* scenario is already covered by PendingAckPersistentTest

4 hours agofix: SqliteJdbcSinkTest close in order (#17849)
tison [Wed, 28 Sep 2022 04:16:30 +0000 (12:16 +0800)] 
fix: SqliteJdbcSinkTest close in order (#17849)

* fix: delete sqlite files after jdbc connection closed

This closes #17713.

Signed-off-by: tison <wander4096@gmail.com>
* uses isolated db file

Signed-off-by: tison <wander4096@gmail.com>
* Revert "uses isolated db file"

This reverts commit 295db3cf9457b537f295e74f091cd7dde0c478de.

* close in order

Signed-off-by: tison <wander4096@gmail.com>
* strong order guarantee

Signed-off-by: tison <wander4096@gmail.com>
* factor out defer logic to avoid further bugs

Signed-off-by: tison <wander4096@gmail.com>
* Revert "factor out defer logic to avoid further bugs"

This reverts commit f7f4634f37da783e90c64d94a2fb0b5cb947ef33.

* Revert "strong order guarantee"

This reverts commit 747086f3c7a4608b764fe6023ebaceab5b1227d1.

* use awaitTermination

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>
6 hours ago[feature][client-cpp] Support inclusive seek for cpp client (#17209)
Kai Wang [Wed, 28 Sep 2022 02:08:24 +0000 (10:08 +0800)] 
[feature][client-cpp] Support inclusive seek for cpp client (#17209)

Fixes #17186

### Motivation

There are some cases in which it is useful to be able to include current
position of the message when reset of cursor was made.

### Modifications

* Support inclusive seek in c++ consumers.
* Add a unit test to verify.

12 hours ago[improve][common] Make Bookkeeper metadata options configurable (#17834)
Lari Hotari [Tue, 27 Sep 2022 20:28:10 +0000 (23:28 +0300)] 
[improve][common] Make Bookkeeper metadata options configurable (#17834)

- use Bookkeeper defaults by setting BK_METADATA_OPTIONS=none

12 hours ago[fix][tests] Fix Mockito mocks memory leak (#17851)
Lari Hotari [Tue, 27 Sep 2022 20:27:33 +0000 (23:27 +0300)] 
[fix][tests] Fix Mockito mocks memory leak (#17851)

* Call cleanup method in finally block to ensure it's not skipped

* Clear invocations for the mocks that are left around without cleanup

* Cleanup PulsarService and PulsarAdmin mocks/spies in MockedPulsarServiceBaseTest

* Don't record invocations at all for PulsarService and PulsarAdmin in MockedPulsarServiceBaseTest

* Don't record invocations for spies by default

* Simplify reseting mocks

* Fix PersistentTopicTest

* Fix TokenExpirationProducerConsumerTest

* Fix SimpleLoadManagerImplTest

* Fix FilterEntryTest

14 hours agoSkip creating a subscription replication snapshot if no messages have been published...
Lari Hotari [Tue, 27 Sep 2022 18:32:45 +0000 (21:32 +0300)] 
Skip creating a subscription replication snapshot if no messages have been published after the topic gets activated on a broker (#16618)

* Skip creating a replication snapshot if no messages have been published

* Adapt test to new behavior where replication snapshots happen only when there are new messages

23 hours ago[improve][pulsar-io-kafka] Add option to copy Kafka headers to Pulsar properties...
Ayman Khalil [Tue, 27 Sep 2022 09:17:03 +0000 (02:17 -0700)] 
[improve][pulsar-io-kafka] Add option to copy Kafka headers to Pulsar properties (#17829)

23 hours agodocs: add developers-landing page to sidebars (#17780)
tison [Tue, 27 Sep 2022 09:07:15 +0000 (17:07 +0800)] 
docs: add developers-landing page to sidebars (#17780)

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>
24 hours ago[fix][cli] Quit PerformanceConsumer after receiving numMessages messages (#17750)
Andras Beni [Tue, 27 Sep 2022 08:46:52 +0000 (10:46 +0200)] 
[fix][cli] Quit PerformanceConsumer after receiving numMessages messages (#17750)

24 hours agodocs: Updating Python installation section (#17796)
tison [Tue, 27 Sep 2022 08:15:51 +0000 (16:15 +0800)] 
docs: Updating Python installation section (#17796)

28 hours ago[Pulsar-init] Support cluster init using proxy url and protocol (#17844)
Rajan Dhabalia [Tue, 27 Sep 2022 04:02:36 +0000 (21:02 -0700)] 
[Pulsar-init] Support cluster init using proxy url and protocol (#17844)

30 hours ago[fix][metrics]wrong metrics text generated when label_cluster specified (#17704)
fengyubiao [Tue, 27 Sep 2022 02:43:38 +0000 (10:43 +0800)] 
[fix][metrics]wrong metrics text generated when label_cluster specified (#17704)

* [fix][metrics]wrong metrics text generated when label_cluster specified

* improve logic branch

* mark test group

34 hours ago[fix][doc] Fix maxNumberOfRejectedRequestPerConnection doc (#17821)
Cong Zhao [Mon, 26 Sep 2022 22:50:22 +0000 (06:50 +0800)] 
[fix][doc] Fix maxNumberOfRejectedRequestPerConnection doc (#17821)

* Fix maxNumberOfRejectedRequestPerConnection doc

* fix doc in 2.8.x docs

38 hours agoSet log level to `warn` when topic policy reader occurs AlreadyClosedException. ...
Jiwei Guo [Mon, 26 Sep 2022 18:52:17 +0000 (02:52 +0800)] 
Set log level to `warn` when topic policy reader occurs AlreadyClosedException. (#17841)

41 hours agoAdd link to Reactive Java client to README.md (#17839)
Lari Hotari [Mon, 26 Sep 2022 15:35:40 +0000 (18:35 +0300)] 
Add link to Reactive Java client to README.md (#17839)

42 hours agoFix broken logo image in README.md (#17838)
Lari Hotari [Mon, 26 Sep 2022 14:38:11 +0000 (17:38 +0300)] 
Fix broken logo image in README.md (#17838)

* Fix broken logo image in README.md

* Use https://pulsar.apache.org/img/pulsar.svg

45 hours ago[improve][CI] Support approval solution on maintenance branches (#17770)
Lari Hotari [Mon, 26 Sep 2022 11:33:41 +0000 (14:33 +0300)] 
[improve][CI] Support approval solution on maintenance branches (#17770)

46 hours ago[fix][connector] KCA: use reflection to get pulsar-client impl classes (#17835)
Nicolò Boschi [Mon, 26 Sep 2022 10:29:52 +0000 (12:29 +0200)] 
[fix][connector] KCA: use reflection to get pulsar-client impl classes (#17835)

47 hours ago[Improve][Standalone] Standalone Add param of --metadata-url for runing with metadata...
Lan [Mon, 26 Sep 2022 09:44:35 +0000 (17:44 +0800)] 
[Improve][Standalone] Standalone Add param of --metadata-url for runing with metadata (#17077)

2 days agodocs: remove unuseful info in docs generation (#17827)
Mercurio [Mon, 26 Sep 2022 07:03:11 +0000 (15:03 +0800)] 
docs: remove unuseful info in docs generation (#17827)

2 days ago[Improve][Auth]Update authentication failed metrics report (#17787)
Guangning E [Mon, 26 Sep 2022 05:02:08 +0000 (13:02 +0800)] 
[Improve][Auth]Update authentication failed metrics report (#17787)

2 days agocleanup unused sync method in NamespacesBase (#17778)
Xiaoyu Hou [Mon, 26 Sep 2022 05:00:36 +0000 (13:00 +0800)] 
cleanup unused sync method in NamespacesBase (#17778)

2 days ago[cleanup][site] remove blogs from the main repo (#17832)
tison [Mon, 26 Sep 2022 03:40:25 +0000 (11:40 +0800)] 
[cleanup][site] remove blogs from the main repo (#17832)

2 days ago[fix][doc] Add missing files linked in `io-connectors` (#17732)
momo-jun [Mon, 26 Sep 2022 03:06:31 +0000 (11:06 +0800)] 
[fix][doc] Add missing files linked in `io-connectors` (#17732)

* add missing files for 2.1.1-incubating

* add missing files for 2.2.0 and 2.2.1

* add missing files for 2.3.0, 2.3.1 and 2.3.2

* add missing files for 2.4.0, 2.4.1, and 2.4.2

* add missing files for 2.5.1 and 2.5.2

* preview fix

3 days agoMake BookieId work with PulsarRegistrationDriver (#17762)
Enrico Olivelli [Sat, 24 Sep 2022 20:07:49 +0000 (22:07 +0200)] 
Make BookieId work with PulsarRegistrationDriver (#17762)

* Make BookieId work with PulsarRegistrationDriver

* Switch to MetadataCache

* checkstyle

3 days agofix comments for exposeManagedLedgerMetricsInPrometheus field (#17792)
HuangZeGui [Sat, 24 Sep 2022 15:14:40 +0000 (23:14 +0800)] 
fix comments for exposeManagedLedgerMetricsInPrometheus field (#17792)

4 days ago[fix][flaky-test]NamespaceOwnershipListenerTests.testNamespaceBundleOwnershipListener...
fengyubiao [Sat, 24 Sep 2022 05:39:15 +0000 (13:39 +0800)] 
[fix][flaky-test]NamespaceOwnershipListenerTests.testNamespaceBundleOwnershipListener (#17807)

4 days ago[fix][sec] Bump snakeyaml to 1.32 for CVE-2022-38752 (#17779)
tison [Sat, 24 Sep 2022 04:44:43 +0000 (12:44 +0800)] 
[fix][sec] Bump snakeyaml to 1.32 for CVE-2022-38752 (#17779)

4 days ago[doc][proxy] Fix typo in "Document how to mitigate CVE-2022-24280" (#17830)
Lari Hotari [Fri, 23 Sep 2022 18:39:24 +0000 (21:39 +0300)] 
[doc][proxy] Fix typo in "Document how to mitigate CVE-2022-24280" (#17830)

4 days ago[improve][misc] Add IssueNavigationLink for IDEA git log (#17769)
Alexander Preuß [Fri, 23 Sep 2022 15:42:48 +0000 (17:42 +0200)] 
[improve][misc] Add IssueNavigationLink for IDEA git log (#17769)

This enables GitHub hyperlinks in IDEA for commits containing PR information

4 days agoDelete site2/website/static/img directory (#17818)
tison [Fri, 23 Sep 2022 15:33:13 +0000 (23:33 +0800)] 
Delete site2/website/static/img directory (#17818)

We no longer sync this folder and it's reasonable to keep the truth on the pulsar-site repo.

This refers to https://github.com/apache/pulsar/issues/17817.

4 days ago[improve][broker] Make MessageRedeliveryController work more efficiently (#17804)
Penghui Li [Fri, 23 Sep 2022 14:45:08 +0000 (22:45 +0800)] 
[improve][broker] Make MessageRedeliveryController work more efficiently (#17804)

4 days ago[doc][proxy] Document how to mitigate CVE-2022-24280 (#17825)
Lari Hotari [Fri, 23 Sep 2022 14:22:16 +0000 (17:22 +0300)] 
[doc][proxy] Document how to mitigate CVE-2022-24280 (#17825)

4 days ago[fix][doc] Remove redundant steps in doc generation (#17585)
Mercurio [Fri, 23 Sep 2022 13:37:08 +0000 (21:37 +0800)] 
[fix][doc] Remove redundant steps in doc generation (#17585)

4 days agodocs: Add ceveat to Pulsar SQL overview and update Presto to Trino (#17798)
tison [Fri, 23 Sep 2022 13:19:56 +0000 (21:19 +0800)] 
docs: Add ceveat to Pulsar SQL overview and update Presto to Trino (#17798)

* docs: Trino -> PrestoSQL

Signed-off-by: tison <wander4096@gmail.com>
* Trino -> PrestoSQL for sql-rest-api

Signed-off-by: tison <wander4096@gmail.com>
* Trino -> PrestoSQL for sql-deployment-configurations

Signed-off-by: tison <wander4096@gmail.com>
* Trino -> PrestoSQL for sql-getting-started

Signed-off-by: tison <wander4096@gmail.com>
* add caveat

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>
5 days agoadd `.md` to doc links (#17763)
momo-jun [Fri, 23 Sep 2022 06:22:05 +0000 (14:22 +0800)] 
add `.md` to doc links (#17763)

5 days ago[fix][docs] Fix link to DebeziumMsSqlSourceTester.java (#17775)
Alexander Preuß [Fri, 23 Sep 2022 01:52:02 +0000 (03:52 +0200)] 
[fix][docs] Fix link to DebeziumMsSqlSourceTester.java (#17775)

5 days ago[doc][message]Fix the description of failover subscription in the doc (#17748)
Xiangying Meng [Fri, 23 Sep 2022 01:27:06 +0000 (09:27 +0800)] 
[doc][message]Fix the description of failover subscription in the doc (#17748)

5 days ago[feat][broker] Support lower boundary shedding for ThresholdShedder (#17456)
feynmanlin [Fri, 23 Sep 2022 01:19:12 +0000 (09:19 +0800)] 
[feat][broker] Support lower boundary shedding for ThresholdShedder (#17456)

 Support lower boundary shedding for ThresholdShedder (#17456)

The existing ThresholdShedder has the following problems, for example:
There are 11 Brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
The average load is 80 * 10 / 11 = 72.73, and the threshold to unload is 72.73 + 10 = 82.73.
Since 80 < 82.73, unload will not be trigger, and there is one idle Broker with load of 0%.

On the basis of ThresholdShedder, we adds the lower boundary judgment of the load.
When 【current usage < average usage - threshold】, the broker with the highest load will be triggered to unload

5 days ago[cleanup][broker]remove unnecessary variable (#17768)
LinChen [Thu, 22 Sep 2022 18:06:24 +0000 (02:06 +0800)] 
[cleanup][broker]remove unnecessary variable (#17768)

* remove unused code

* check style

* remove unused bundleFactory

Co-authored-by: leolinchen <leolinchen@tencent.com>
5 days ago[improve][txn] Add getState in transaction for client API (#17423)
congbo [Thu, 22 Sep 2022 15:00:24 +0000 (23:00 +0800)] 
[improve][txn] Add getState in transaction for client API (#17423)

### Motivation
now `org.apache.pulsar.client.api.transaction.Transaction` dont have a interface for user to get the transaction state.

user can get the transaction state to do user's own op.
### Modifications
1. add the interface in `org.apache.pulsar.client.api.transaction.Transaction`     `getState`
2. TransactionImpl implement the interface
```
     * Get transaction state.
     *
     * @return {@link State} the state of the transaction.
     */
    State getState();
```
### Verifying this change
add the test

5 days ago[improve][client-c++]Support include message header size when check maxMessageSize...
Cong Zhao [Thu, 22 Sep 2022 14:59:32 +0000 (22:59 +0800)] 
[improve][client-c++]Support include message header size when check maxMessageSize (#17289)

### Motivation

See: #17188

### Modifications

Support include message header size when check maxMessageSize for cpp client

5 days agoManagedLedger: move to FENCED state in case of BadVersionException (#17736)
Enrico Olivelli [Thu, 22 Sep 2022 13:50:12 +0000 (15:50 +0200)] 
ManagedLedger: move to FENCED state in case of BadVersionException (#17736)

5 days ago[fix] [python client] Better Python garbage collection management for C++-owned objec...
Zac Bentley [Thu, 22 Sep 2022 13:47:53 +0000 (09:47 -0400)] 
[fix] [python client] Better Python garbage collection management for C++-owned objects (#16535)

Fixes https://github.com/apache/pulsar/issues/16527

5 days ago[fix][broker] Extract additional servlets to the default directory by… (#17477)
Andras Beni [Thu, 22 Sep 2022 11:27:09 +0000 (13:27 +0200)] 
[fix][broker] Extract additional servlets to the default directory by… (#17477)

5 days ago[doc][monitoring][metadata] metadata store metrics doc (#17088)
Tao Jiuming [Thu, 22 Sep 2022 10:30:36 +0000 (18:30 +0800)] 
[doc][monitoring][metadata] metadata store metrics doc (#17088)

5 days ago[fix][tableview] fixed ack failure in ReaderImpl due to null messageId (#17728)
Heesung Sohn [Thu, 22 Sep 2022 10:29:14 +0000 (03:29 -0700)] 
[fix][tableview] fixed ack failure in ReaderImpl due to null messageId (#17728)

5 days ago[fix][tests] Some unit tests are not run by the CI (#17767)
Nicolò Boschi [Thu, 22 Sep 2022 09:09:44 +0000 (11:09 +0200)] 
[fix][tests] Some unit tests are not run by the CI (#17767)

6 days ago[fix][cpp] Use weak ptr avoid circular references. (#17481)
Baodi Shi [Thu, 22 Sep 2022 07:49:12 +0000 (15:49 +0800)] 
[fix][cpp] Use weak ptr avoid circular references. (#17481)

### Motivation

Capturing shared ptr in the timer function will cause a circular reference.

### Modifications

- Use weak ptr instead shared ptr.

6 days ago[feat][broker]PIP-180 ShadowTopic - Part IV - Add Shadow Replicator (#17371)
JiangHaiting [Thu, 22 Sep 2022 02:57:00 +0000 (10:57 +0800)] 
[feat][broker]PIP-180 ShadowTopic - Part IV - Add Shadow Replicator (#17371)

6 days ago[improve][doc] Improve release process document (#17684)
momo-jun [Thu, 22 Sep 2022 00:58:25 +0000 (08:58 +0800)] 
[improve][doc] Improve release process document (#17684)

6 days agoAdd Pulsar Summit Asia 2022 CFP blog (#17757)
Sherlock Xu [Thu, 22 Sep 2022 00:56:22 +0000 (08:56 +0800)] 
Add Pulsar Summit Asia 2022 CFP blog (#17757)

6 days ago[monitoring][broker][fix] Fix EntryFilter stats (#17605)
Tao Jiuming [Wed, 21 Sep 2022 13:30:03 +0000 (21:30 +0800)] 
[monitoring][broker][fix] Fix EntryFilter stats (#17605)

* fix entryFilter stats

* fix test

* add test comment

* review fix

6 days ago[fix][broker]Consumer can't consume messages because there has two sames topics in...
fengyubiao [Wed, 21 Sep 2022 13:28:38 +0000 (21:28 +0800)] 
[fix][broker]Consumer can't consume messages because there has two sames topics in one broker (#17526)

6 days ago[fix][tests] Fix resource leak in tests. Call cleanup before setup (#17743)
Lari Hotari [Wed, 21 Sep 2022 12:58:37 +0000 (15:58 +0300)] 
[fix][tests] Fix resource leak in tests. Call cleanup before setup (#17743)

- when setup is called explicitly, cleanup should be issued before it
- remove unnecessary "resetConfig" method calls
  - resetConfig gets called as part of internalCleanup
  - when resetConfig is in cleanup, it's possible to override config before calling setup
    - this allows using setup & cleanup methods instead of using
      error prone internalSetup and internalCleanup methods which might leave something behind

7 days ago[cleanup][broker]remove unnecessary variable in MLTransactionMetadataStoreProvider...
fengyubiao [Wed, 21 Sep 2022 04:59:16 +0000 (12:59 +0800)] 
[cleanup][broker]remove unnecessary variable in MLTransactionMetadataStoreProvider (#17721)

7 days ago[fix][connector] Upgrade hadoop client to 3.3.3 and 2.10.2 (#17727)
Penghui Li [Wed, 21 Sep 2022 03:24:36 +0000 (11:24 +0800)] 
[fix][connector] Upgrade hadoop client to 3.3.3 and 2.10.2 (#17727)

7 days ago[feat][build] Support ARM64-based docker images (#17733)
tison [Wed, 21 Sep 2022 03:17:38 +0000 (11:17 +0800)] 
[feat][build] Support ARM64-based docker images (#17733)

7 days ago[fix][doc] fix schema get value doc problem (#17740)
congbo [Wed, 21 Sep 2022 03:14:07 +0000 (11:14 +0800)] 
[fix][doc] fix schema get value doc problem (#17740)

7 days agonamespace cli set-offload-policy's thresholdBytes support negative and 0 (#17502)
YingQun Zhong [Wed, 21 Sep 2022 03:07:24 +0000 (11:07 +0800)] 
namespace cli set-offload-policy's thresholdBytes support negative and 0 (#17502)

7 days ago[feat][build] Print out more info for bin/pulsar version (#17752)
Michael Marshall [Wed, 21 Sep 2022 02:41:53 +0000 (19:41 -0700)] 
[feat][build] Print out more info for bin/pulsar version (#17752)

### Motivation

The current `bin/pulsar version` script only prints out the version string. When running custom builds of Pulsar, this is sometimes not enough to know which version you have. Since the `PulsarVersion` class already has more information, I propose we print out all of its information in the same format that the broker logs the information.

Here is a sample output for this specific PR:

```shell
$ bin/pulsar version
Current version of pulsar is: 2.11.0-SNAPSHOT
Git Revision 4414e3d76c27e821e75f30eae853862b04e63200
Git Branch print-more-version-info
Built by Michael Marshall <mmarshall@apache.org> on mmarshall-rmbp16 at 2020-01-22T09:10:15-0600
```

### Modifications

* Add more output to the `bin/pulsar version` command.

### Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

### Documentation

- [x] `doc-not-needed`
(Please explain why)

### Matching PR in forked repository

PR in forked repository: https://github.com/michaeljmarshall/pulsar/pull/3

7 days ago[improve][test] Add integration test for pulsar-client (#17616)
Andras Beni [Wed, 21 Sep 2022 02:20:03 +0000 (04:20 +0200)] 
[improve][test] Add integration test for pulsar-client (#17616)

7 days ago[improve][test] Add integration test for broker interceptors (#17515)
Andras Beni [Wed, 21 Sep 2022 02:18:43 +0000 (04:18 +0200)] 
[improve][test] Add integration test for broker interceptors (#17515)

7 days ago[improve][doc] add a tip for doc contribution (#17744)
Anonymitaet [Wed, 21 Sep 2022 02:14:45 +0000 (10:14 +0800)] 
[improve][doc] add a tip for doc contribution (#17744)

7 days ago[improve][CI] Don't run CI when PR is in draft status or isn't mergeable (#17749)
Lari Hotari [Tue, 20 Sep 2022 19:54:36 +0000 (22:54 +0300)] 
[improve][CI] Don't run CI when PR is in draft status or isn't mergeable (#17749)

7 days ago[improve][ci] add back cpp-test name (#17742)
tison [Tue, 20 Sep 2022 16:23:40 +0000 (00:23 +0800)] 
[improve][ci] add back cpp-test name (#17742)

7 days ago[fix][security] Upgrade reload4j in file-system offloader (#17716)
Nicolò Boschi [Tue, 20 Sep 2022 15:37:36 +0000 (17:37 +0200)] 
[fix][security] Upgrade reload4j in file-system offloader (#17716)

7 days ago[Doc]--remove redundant words (#17741)
Huanli Meng [Tue, 20 Sep 2022 13:11:35 +0000 (21:11 +0800)] 
[Doc]--remove redundant words (#17741)

7 days ago[monitoring][broker][metadata] add metadata store metrics (#17041)
Tao Jiuming [Tue, 20 Sep 2022 11:01:41 +0000 (19:01 +0800)] 
[monitoring][broker][metadata] add metadata store metrics (#17041)

7 days ago[feature][pulsar-io-mongo] Add support for full message synchronization (#16003)
Yuanhao Ji [Tue, 20 Sep 2022 10:59:54 +0000 (18:59 +0800)] 
[feature][pulsar-io-mongo] Add support for full message synchronization (#16003)

### Motivation

Now, the MongoDB source connector only supports the incremental message synchronization.
This PR adds support for full message synchronization.

Since MongDB 4.0, we can set the starting point for the change stream by the `startAtOperationTime` field.
So, we can set it to `0` to make start point the earliest.
See https://www.mongodb.com/docs/v4.2/reference/method/db.collection.watch/ for more information.

7 days ago[improve][txn][PIP-196]Segmented transaction buffer snapshot configuration (#16917)
Xiangying Meng [Tue, 20 Sep 2022 10:18:39 +0000 (18:18 +0800)] 
[improve][txn][PIP-196]Segmented transaction buffer snapshot configuration (#16917)

* [improve][txn][PIP-196] Segmented transaction buffer snapshot
Master https://github.com/apache/pulsar/issues/16913
### Motivation
1. Add configurations.
2. Implement the system topic client for snapshot topic and index topic.
3. Implement AbortedTxnProcessor.
4. Add transaction buffer snapshot metrics.
### Modification
Add configuration for segment snaopshot.

8 days ago[cleanup][test] Delete unnecessary variables in SubscriptionMessageDispatchThrottling...
guan46 [Tue, 20 Sep 2022 08:32:31 +0000 (16:32 +0800)] 
[cleanup][test] Delete unnecessary variables in SubscriptionMessageDispatchThrottlingTest (#17702)

8 days ago[improve][doc] Improve the information architecture of encryption and cookbook (...
momo-jun [Tue, 20 Sep 2022 08:06:44 +0000 (16:06 +0800)] 
[improve][doc] Improve the information architecture of encryption and cookbook (#17666)

* remove duplicate file `cookbooks-encryption.md`

* Revert "remove duplicate file `cookbooks-encryption.md`"

This reverts commit a1e5a547c9ca5c65fc806aca3c6c606d555cc68d.

* streamline encryption cookbook and make it single-sourced

* Add code snippets for Java/C++/Go clients.

* use one svg image to replace two jpg files

* client version and title updates

* improve security overview

* Add note and link for TLS encryption

8 days ago[fix][client] Unwrap completion exception for Lookup Services (#17717)
Penghui Li [Tue, 20 Sep 2022 07:54:31 +0000 (15:54 +0800)] 
[fix][client] Unwrap completion exception for Lookup Services (#17717)

8 days ago[fix][metadata] Cleanup state when lock revalidation gets `LockBusyException` (#17700)
Qiang Zhao [Tue, 20 Sep 2022 07:41:15 +0000 (15:41 +0800)] 
[fix][metadata] Cleanup state when lock revalidation gets `LockBusyException` (#17700)

### Motivation

In the production environment,  we found two brokers holding the same valid locks. and one has an exceptional revalidate future with `lockBusyException`. after reading the code, there may forget the reset the cache and complete expire exception when getting lockBusyException.

8 days ago[improve][doc] Improve the information architecture of the security chapter (#17615)
momo-jun [Tue, 20 Sep 2022 05:14:16 +0000 (13:14 +0800)] 
[improve][doc] Improve the information architecture of the security chapter (#17615)

* Sidebar re-org

* Separate TLS encryption/authentication using Keystore to parent topics

* use language-specific tabs to show code snippets

* streamline headings

* Update client-libraries-java.md

* Add code snippet for python clients to support #17482

* Add code snippet for go clients

8 days ago[fix][flaky-test]CompactedTopicTest.testReadCompleteMessagesDuringTopicUnloading...
fengyubiao [Tue, 20 Sep 2022 05:13:15 +0000 (13:13 +0800)] 
[fix][flaky-test]CompactedTopicTest.testReadCompleteMessagesDuringTopicUnloading (#17380)

8 days ago[fix][schema]ledger handle leak when update schema (#17283)
fengyubiao [Tue, 20 Sep 2022 03:21:54 +0000 (11:21 +0800)] 
[fix][schema]ledger handle leak when update schema (#17283)

### Motivation

in the schema update, will create a `ledgerHandle` and write data to BK, after that `ledgerHandle` is no longer useful and no other object holds references to it. `ledgerHandle` will be recycled with GC, but `ledgerHandle` also hold external connections, which will cause leakage.

https://github.com/apache/pulsar/blob/40b9d7ea50cef54becb09f2543193e08375abe0b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java#L452-L456

### Modifications

after the schema is updated, close the `ledgerHandle`, just like schema-read:

https://github.com/apache/pulsar/blob/40b9d7ea50cef54becb09f2543193e08375abe0b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java#L519-L525

8 days agodocs: replace reference-configuration page with an external link (#17699)
Mercurio [Tue, 20 Sep 2022 01:31:16 +0000 (09:31 +0800)] 
docs: replace reference-configuration page with an external link (#17699)

8 days ago[fix][CI] Fix issues with approval solution for GitHub Actions (#17723)
Lari Hotari [Mon, 19 Sep 2022 21:30:11 +0000 (00:30 +0300)] 
[fix][CI] Fix issues with approval solution for GitHub Actions (#17723)

- The approval solution doesn't work as expected by approving the PR
  or by adding the ready-to-test label and adding a comment
  "/pulsarbot rerun-failure-checks".

- Fix the ready-to-test label check:
  - Refresh PR labels when re-running workflow
    - when re-running, the event JSON remains the same. The API
      must be used to fetch the up-to-date JSON for the PR.

- Fix the PR approval check:
  - set GITHUB_TOKEN for script so that retrieving the approval status could work

8 days ago[improve][ci] docbot comment only for open PRs (#17709)
tison [Mon, 19 Sep 2022 19:25:47 +0000 (03:25 +0800)] 
[improve][ci] docbot comment only for open PRs (#17709)

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>
8 days ago[fix][broker] Fix if dynamicConfig item in ZK do not exist in broker cause NPE (...
Lei Zhiyuan [Mon, 19 Sep 2022 12:33:05 +0000 (20:33 +0800)] 
[fix][broker] Fix if dynamicConfig item in ZK do not exist in broker cause NPE (#17705)

8 days ago[fix][common] Fix issue where logs get truncated when Runtime.halt is used to termina...
Lari Hotari [Mon, 19 Sep 2022 11:37:13 +0000 (14:37 +0300)] 
[fix][common] Fix issue where logs get truncated when Runtime.halt is used to terminate the process (#17589)

* [fix][common] Fix issue where logs get truncated when Pulsar process gets terminated with Runtime.halt

- call Log4j2's LogManager.shutdown before terminating the process with Runtime.halt

* Don't log warning when status code is 0

9 days ago[improvement][client-java] Avoid too large memory preallocation for batch message...
Tao Jiuming [Mon, 19 Sep 2022 07:05:39 +0000 (15:05 +0800)] 
[improvement][client-java] Avoid too large memory preallocation for batch message. (#15033)

9 days ago[feat][CI] Add approval solution to reduce GitHub Actions resource consumption (...
Lari Hotari [Mon, 19 Sep 2022 06:18:42 +0000 (09:18 +0300)] 
[feat][CI] Add approval solution to reduce GitHub Actions resource consumption (#17693)

Please check the proposal email on dev mailing list: https://lists.apache.org/thread/gwfmxmxlhtsjn17sxxc367jcs4pcwofz

Co-authored-by: tison <wander4096@gmail.com>
9 days ago[fix][metadata] Handle session events in separate thread (#17638)
Lari Hotari [Mon, 19 Sep 2022 02:40:18 +0000 (05:40 +0300)] 
[fix][metadata] Handle session events in separate thread (#17638)

9 days ago[fix][broker] Fix namespace backlog quota check with retention. (#17706)
JiangHaiting [Mon, 19 Sep 2022 02:29:51 +0000 (10:29 +0800)] 
[fix][broker] Fix namespace backlog quota check with retention. (#17706)

9 days ago[flaky-test]Add information in ManagedLedgerBkTest to determine the problem (#17441)
fengyubiao [Mon, 19 Sep 2022 02:24:47 +0000 (10:24 +0800)] 
[flaky-test]Add information in ManagedLedgerBkTest to determine the problem (#17441)

9 days ago[improve][client-c++] support Exclusive Producer access mode for c++ (#17439)
Cong Zhao [Mon, 19 Sep 2022 02:16:00 +0000 (10:16 +0800)] 
[improve][client-c++] support Exclusive Producer access mode for c++ (#17439)

9 days ago[improve][broker] Make cursor properties support modify single value concurrently...
Cong Zhao [Mon, 19 Sep 2022 01:42:00 +0000 (09:42 +0800)] 
[improve][broker] Make cursor properties support modify single value concurrently. (#17164)

9 days ago[fix][metadata] Set revalidateAfterReconnection true for certain failures (#17664)
Michael Marshall [Sun, 18 Sep 2022 13:27:59 +0000 (06:27 -0700)] 
[fix][metadata] Set revalidateAfterReconnection true for certain failures (#17664)

10 days ago[improve][cli] Pulsar shell: add command to set/get property of a config (#17651)
Nicolò Boschi [Sat, 17 Sep 2022 09:27:59 +0000 (11:27 +0200)] 
[improve][cli] Pulsar shell: add command to set/get property of a config (#17651)

10 days ago[improve][test] remove powermock-reflect dependency (#17696)
tison [Sat, 17 Sep 2022 09:27:34 +0000 (17:27 +0800)] 
[improve][test] remove powermock-reflect dependency (#17696)

11 days ago[fix][functions] Ensure InternalConfigurationData data model is compatible across...
Nicolò Boschi [Fri, 16 Sep 2022 18:03:00 +0000 (20:03 +0200)] 
[fix][functions] Ensure InternalConfigurationData data model is compatible across different versions (#17690)

* [fix][functions] Ensure InternalConfigurationData data model is compatible across different versions

* style

* fix the other way

### Motivation

After https://github.com/apache/pulsar/pull/14384, the broker and the client expects that the `InternalConfigurationData` contains `metadataStoreUrl` and `configurationMetadataStoreUrl` fields.
However the broker is no more compatible with old clients.

https://github.com/apache/pulsar/pull/14384 is landed to branch-2.11 and [2.10.2](https://github.com/apache/pulsar/pull/17291)

Example scenario:
- broker on 2.10.1
- function worker on 2.10.1

1. upgrade fn worker to 2.11.0 or 2.10.2
2. the fn worker starts and download the internal config from the broker
3. broker serves a json with old fields (`zookeeperServers` and `configurationStoreServers`)
4. fn worker reads the json and convert it to a `InternalConfigurationData` instance. It expects to see the fields filled `metadataStoreUrl` and `configurationMetadataStoreUrl` but they aren't
5. NPE on fn worker
```
2022-09-15T17:42:16,072+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - Initializing Pulsar Functions namespace...
2022-09-15T17:42:16,192+0000 [main] ERROR org.apache.pulsar.functions.worker.FunctionWorkerStarter - Encountered error in function worker.
java.lang.NullPointerException: null
    at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.removeIdentifierFromMetadataURL(MetadataStoreFactoryImpl.java:73)
    at org.apache.pulsar.functions.worker.WorkerUtils.initializeDlogNamespace(WorkerUtils.java:188)
    at org.apache.pulsar.functions.worker.PulsarWorkerService.initializeStandaloneWorkerService(PulsarWorkerService.java:281)
    at org.apache.pulsar.functions.worker.PulsarWorkerService.initAsStandalone(PulsarWorkerService.java:208)
    at org.apache.pulsar.functions.worker.Worker.start(Worker.java:54)
    at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:76)
```

Additionaly there's the same issue if we upgrade the broker before the fn worker:
1. the broker gets the upgrade. won't serve `zookeeperServers` field
2. fn worker restarts for some reasons.
3. fn worker gets the internal config and look for  `zookeeperServers` field which is empty in the json
4. NPE

### Modifications

* Restore old fields in `InternalConfigurationData` and add fallback the old values in the new fields getters
* Added unit test

- [x] `doc-not-needed`

11 days ago[fix][cli] Fix mbeans to json (#17676)
Zixuan Liu [Fri, 16 Sep 2022 15:56:29 +0000 (23:56 +0800)] 
[fix][cli] Fix mbeans to json (#17676)

Signed-off-by: Zixuan Liu <nodeces@gmail.com>
11 days ago[fix][common] Fix parsing partitionedKey with Base64 encode issue. (#17687)
Jiwei Guo [Fri, 16 Sep 2022 15:56:03 +0000 (23:56 +0800)] 
[fix][common] Fix parsing partitionedKey with Base64 encode issue. (#17687)

* Fix parsing partitionedKey with Base64 encode issue.

* release the buf

* fix checkstyle issue.

11 days ago[doc] change 0.2.0 to 0.3.0 version pulsar-manager (#17633)
Naresh [Fri, 16 Sep 2022 13:42:35 +0000 (19:12 +0530)] 
[doc] change 0.2.0 to 0.3.0 version pulsar-manager (#17633)

11 days ago[fix][broker] Fix unexpected subscription deletion caused by the cursor last active...
Shen Liu [Fri, 16 Sep 2022 13:34:45 +0000 (21:34 +0800)] 
[fix][broker] Fix unexpected subscription deletion caused by the cursor last active time not updated in time (#17573)