airflow.git
6 hours agoFixed triple quotes in task group example (#26829) main
Mati O [Sun, 2 Oct 2022 16:38:17 +0000 (19:38 +0300)] 
Fixed triple quotes in task group example (#26829)

16 hours agoDocumentation fixes (#26819)
Karthikeyan Singaravelan [Sun, 2 Oct 2022 06:44:45 +0000 (12:14 +0530)] 
Documentation fixes (#26819)

19 hours agoRetry on Airflow Schedule DAG Run DB Deadlock (#26347)
Anthony Panat [Sun, 2 Oct 2022 03:33:59 +0000 (23:33 -0400)] 
Retry on Airflow Schedule DAG Run DB Deadlock (#26347)

Co-authored-by: Anthony Panat <anthonypanat@Anthonys-MacBook-Pro-2.local>
Co-authored-by: Anthony Panat <anthonypanat@anthonys-mbp-2.mynetworksettings.com>
20 hours agomake consistency on markup title string level (#26696)
Changhoon Oh [Sun, 2 Oct 2022 02:59:23 +0000 (11:59 +0900)] 
make consistency on markup title string level (#26696)

Co-authored-by: ok9897 <ok9897@krafton.com>
20 hours agoAdd information about Amazon Elastic MapReduce Connection (#26687)
Andrey Anshin [Sun, 2 Oct 2022 02:57:23 +0000 (06:57 +0400)] 
Add information about Amazon Elastic MapReduce Connection (#26687)

* Added information about Amazon Elastic MapReduce Connection

22 hours agoSagemaker and Sagemaker Endpoint System Tests (AIP-47) (#26684)
D. Ferruzzi [Sun, 2 Oct 2022 01:12:42 +0000 (18:12 -0700)] 
Sagemaker and Sagemaker Endpoint System Tests (AIP-47) (#26684)

* Sagemaker System Test fixes

22 hours agoFix null strings bug in SqlToS3Operator in non parquet formats (#26676)
harryplumer [Sun, 2 Oct 2022 01:12:21 +0000 (18:12 -0700)] 
Fix null strings bug in SqlToS3Operator in non parquet formats (#26676)

22 hours agoCorrecting the transfer config name. (#25719)
Prajyot Naik [Sun, 2 Oct 2022 00:56:11 +0000 (06:26 +0530)] 
Correcting the transfer config name. (#25719)

22 hours agoFix Cloud Build assets & system tests migration (AIP-47) (#26672)
Beata Kossakowska [Sun, 2 Oct 2022 00:43:13 +0000 (02:43 +0200)] 
Fix Cloud Build assets & system tests migration (AIP-47) (#26672)

* Add a missing file.

* Fix example_cloud_build.py

25 hours agoA few docs fixups (#26788)
blag [Sat, 1 Oct 2022 22:16:08 +0000 (15:16 -0700)] 
A few docs fixups (#26788)

31 hours agoFix an infallible test of triggerer job (#26816)
Lyuwen Yan [Sat, 1 Oct 2022 16:14:02 +0000 (09:14 -0700)] 
Fix an infallible test of triggerer job (#26816)

Co-authored-by: yanmulin <lyuweny@gmail.com>
33 hours agoFix parsing of optional `mode` field in BigQuery Result Schema (#26786)
Peter Wicks [Sat, 1 Oct 2022 13:47:53 +0000 (07:47 -0600)] 
Fix parsing of optional `mode` field in BigQuery Result Schema (#26786)

33 hours agoFix MaxID logic for GCSToBigQueryOperator (#26768)
Peter Wicks [Sat, 1 Oct 2022 13:39:16 +0000 (07:39 -0600)] 
Fix MaxID logic for GCSToBigQueryOperator (#26768)

33 hours agoUpdate message with new breeze subcommand (#26817)
Ross Turk [Sat, 1 Oct 2022 13:38:31 +0000 (09:38 -0400)] 
Update message with new breeze subcommand (#26817)

Signed-off-by: Ross Turk <ross@rossturk.com>
Signed-off-by: Ross Turk <ross@rossturk.com>
46 hours agoAdd "Clean" to the Amazon System Test Contrib Guide (#26811)
D. Ferruzzi [Sat, 1 Oct 2022 00:35:55 +0000 (17:35 -0700)] 
Add "Clean" to the Amazon System Test Contrib Guide (#26811)

2 days agoApache Airflow 2.4.1 has been released (#26810)
Jed Cunningham [Fri, 30 Sep 2022 23:18:52 +0000 (16:18 -0700)] 
Apache Airflow 2.4.1 has been released (#26810)

2 days agoFix running debuggers inside `airflow tasks test` (#26806)
Ash Berlin-Taylor [Fri, 30 Sep 2022 17:46:14 +0000 (18:46 +0100)] 
Fix running debuggers inside `airflow tasks test` (#26806)

As part of 2.3.3 we added redaction to output from the tasks test
command, but unfortunately that broke using a debugger with this error:

```
  File "/usr/lib/python3.10/pdb.py", line 262, in user_line
    self.interaction(frame, None)
  File "/home/ash/.virtualenvs/airflow/lib/python3.10/site-packages/pdb.py", line 231, in interaction
    self._cmdloop()
  File "/usr/lib/python3.10/pdb.py", line 322, in _cmdloop
    self.cmdloop()
  File "/usr/lib/python3.10/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
TypeError: 'NoneType' object cannot be interpreted as an integer
```

(ipdb has a similar but different error)

The "fix" is to assign a fileno attribute to the object. `input()` needs
this to write the prompt. It feels like a "bug" that it doesn't work
without it, but as this class is only used in `tasks test` this is a
safe change

2 days agoAdd BatchOperator template fields (#26805)
D. Ferruzzi [Fri, 30 Sep 2022 17:35:10 +0000 (10:35 -0700)] 
Add BatchOperator template fields (#26805)

2 days agoFix warning when using xcomarg dependencies (#26801)
Ash Berlin-Taylor [Fri, 30 Sep 2022 16:03:07 +0000 (17:03 +0100)] 
Fix warning when using xcomarg dependencies (#26801)

This warning was invisible before 2.4 due to a bug in our logging config
(fixed by commit 7363e35) and AIP-45 which suddenly made this appear.

The problem was being caused by set_xcomargs_dependencies being called
once for each class in the hierarchy, and each of them doing the same
logic.

The fix is to look at the _actual_ function of `self.__init__` and
compare it to the function we're about to call so that we don't set
dependencies until we have finished the "outer" most class's
apply_defaults invocation.

2 days agodemote Removed state in priority for displaying task summaries (#26789)
Brent Bovenzi [Fri, 30 Sep 2022 15:52:28 +0000 (11:52 -0400)] 
demote Removed state in priority for displaying task summaries (#26789)

2 days agoUpdate KIND used in tests to v0.16.0 (#26791)
raphaelauv [Fri, 30 Sep 2022 13:39:30 +0000 (15:39 +0200)] 
Update KIND used in tests to v0.16.0 (#26791)

2 days agoFix separator getting added to variables_prefix when empty (#26749)
Rajath [Fri, 30 Sep 2022 13:15:48 +0000 (18:45 +0530)] 
Fix separator getting added to variables_prefix when empty (#26749)

Fix separator getting added to variables_prefix

azure key vault adds a - between the key prefix and the secret name when it reads it from azure key vault. It does this even when the key prefix is "" so secrets end up named -my-secret instead of my-secret.

2 days agoEnsure the log messages from operators during parsing go somewhere (#26779)
Ash Berlin-Taylor [Fri, 30 Sep 2022 13:08:41 +0000 (14:08 +0100)] 
Ensure the log messages from operators during parsing go somewhere (#26779)

* Ensure the log messages from operators during parsing go somewhere

While investigating #26599 and the change from AIP-45, I noticed that
these warning messages weren't new! The only thing that was new was that
we started seeing them.

This is because the logger for BaseOperator and all subclasses is
`airflow.task.operators`, and the `airflow.task` logger is not
configured (with `set_context()`) until we have a TaskInstance, so it
just dropped all messages on the floor!

This changes it so that log messages are propagated to parent loggers by
default, but when we configure a context (and thus have a file to write
to) we stop that. A similar change was made for the `airflow.processor`
(but that is unlikely to suffer the same fate)

* Give a real row count value so logs don't fail

The ArangoDB sensor test was logging a mock object, which previously was
getting dropped before emitting, but with this change now fails with
"Mock is not an integer" when attempting the  `%d` interpolation.

To avoid making the mock overly specific (`arangodb_client_for_test.db.`
`return_value.aql.execute.return_value.count.return_value`!) I have
changed the test to mock the hook entirely (which is already tested)

2 days agoAdd restarting state to TaskState Enum in REST API (#26776)
Ephraim Anierobi [Thu, 29 Sep 2022 23:28:12 +0000 (00:28 +0100)] 
Add restarting state to TaskState Enum in REST API (#26776)

3 days agoRemove DAG parsing from StandardTaskRunner (#26750)
Ephraim Anierobi [Thu, 29 Sep 2022 21:51:46 +0000 (22:51 +0100)] 
Remove DAG parsing from StandardTaskRunner (#26750)

This makes the starting of StandardTaskRunner faster as the parsing of DAG will now be done once at task_run.
Also removed parsing of example dags when running a task

3 days agoadd icon legend to datasets graph (#26781)
Brent Bovenzi [Thu, 29 Sep 2022 18:14:45 +0000 (14:14 -0400)] 
add icon legend to datasets graph (#26781)

3 days agoAllow retrieving error message from data.detail (#26762)
pierrejeambrun [Thu, 29 Sep 2022 14:06:52 +0000 (16:06 +0200)] 
Allow retrieving error message from data.detail (#26762)

3 days agoReduce log verbosity in KubernetesExecutor. (#26582)
HTErik [Thu, 29 Sep 2022 03:14:25 +0000 (05:14 +0200)] 
Reduce log verbosity in KubernetesExecutor. (#26582)

4 days agoFix airflow tasks run --local when dags_folder differs from that of processor (#26509)
Daniel Standish [Wed, 28 Sep 2022 22:56:14 +0000 (15:56 -0700)] 
Fix airflow tasks run --local when dags_folder differs from that of processor (#26509)

Previously the code used the dags_folder of the "current" process (e.g. the celery worker, or k8s executor worker pod) to calculate the relative fileloc based on the full fileloc stored in the serialized dag.  But if the worker dags_folder folder is different from the dags folder configured on the dag processor, then airflow can't calculate the relative path, so it will just use the full path, which in this case will be a bad path.  We can fix this by keeping track of the dags_folder from the dag processor that serialized the dag, and using this for figuring out the relative path.

4 days agoAdd all dependent providers when running selective checks (#26736)
Jed Cunningham [Wed, 28 Sep 2022 20:27:40 +0000 (13:27 -0700)] 
Add all dependent providers when running selective checks (#26736)

* Add all dependent providers when running selective checks

Like we run tests for providers that use a changing provider, we need to
run tests for any providers we use directly.

So for example, if we change the `postgres` provider, both `common.sql`
(which `postgres` uses) and `google` (which uses `postgres`) should be
run.

4 days ago[docs] Improve documentation on helm hooks disabling (#26747)
Matthew Aguerreberry [Wed, 28 Sep 2022 19:15:10 +0000 (16:15 -0300)] 
[docs] Improve documentation on helm hooks disabling (#26747)

Copy and paste the older code snippet into the values.yml did not work, the migrations jobs were not triggered when using ArgoCD. I had to place the code in this format in order to function.

4 days agoImprove provider release process (#26748)
Jarek Potiuk [Wed, 28 Sep 2022 18:02:08 +0000 (20:02 +0200)] 
Improve provider release process (#26748)

Found during September release:

* fix process steps to be copy&pasteable
* add -m to tag generation to account for signed/annotated tags
* added twine check to the CI workflow

4 days agoFix errors in CHANGELOGS for slack and amazon (#26746) providers-alibaba/2.1.0 providers-alibaba/2.1.0rc1 providers-amazon/6.0.0 providers-amazon/6.0.0rc1 providers-apache-drill/2.2.1 providers-apache-drill/2.2.1rc1 providers-apache-druid/3.2.1 providers-apache-druid/3.2.1rc1 providers-apache-hive/4.0.1 providers-apache-hive/4.0.1rc1 providers-apache-pinot/3.2.1 providers-apache-pinot/3.2.1rc1 providers-atlassian-jira/1.0.0 providers-atlassian-jira/1.0.0rc1 providers-databricks/3.3.0 providers-databricks/3.3.0rc1 providers-dbt-cloud/2.2.0 providers-dbt-cloud/2.2.0rc1 providers-docker/3.2.0 providers-docker/3.2.0rc1 providers-elasticsearch/4.2.1 providers-elasticsearch/4.2.1rc1 providers-exasol/4.0.1 providers-exasol/4.0.1rc1 providers-google/8.4.0rc1 providers-jdbc/3.2.1 providers-jdbc/3.2.1rc1 providers-jira/3.1.0 providers-jira/3.1.0rc1 providers-microsoft-azure/4.3.0 providers-microsoft-azure/4.3.0rc1 providers-microsoft-mssql/3.2.1 providers-microsoft-mssql/3.2.1rc1 providers-mysql/3.2.1 providers-mysql/3.2.1rc1 providers-odbc/3.1.2 providers-odbc/3.1.2rc1 providers-oracle/3.4.0 providers-oracle/3.4.0rc1 providers-postgres/5.2.2 providers-postgres/5.2.2rc1 providers-qubole/3.2.1 providers-qubole/3.2.1rc1 providers-sftp/4.1.0 providers-sftp/4.1.0rc1 providers-slack/6.0.0rc1 providers-snowflake/3.3.0 providers-snowflake/3.3.0rc1 providers-ssh/3.2.0 providers-ssh/3.2.0rc1 providers-trino/4.1.0 providers-trino/4.1.0rc1 providers-vertica/3.2.1 providers-vertica/3.2.1rc1
Jarek Potiuk [Wed, 28 Sep 2022 12:31:00 +0000 (14:31 +0200)] 
Fix errors in CHANGELOGS for slack and amazon (#26746)

4 days agoBump sphinx and sphinx-autoapi (#26743)
Jarek Potiuk [Wed, 28 Sep 2022 11:42:43 +0000 (13:42 +0200)] 
Bump sphinx and sphinx-autoapi (#26743)

The sphinx-autoapi bug https://github.com/readthedocs/sphinx-autoapi/issues/352
has been resolved and 2.0.0 version has been released. Since we
are using sphinx/sphinx-autoapi only for `doc` extra, we can
safely bump the minimum version of both to latest released minor
versions.

4 days agoUpdate docs for September Provider's release (#26731)
Jarek Potiuk [Wed, 28 Sep 2022 11:02:00 +0000 (13:02 +0200)] 
Update docs for September Provider's release (#26731)

4 days agoFix GCS sensor system tests failing with DebugExecutor (#26742)
BartÅ‚omiej Hirsz [Wed, 28 Sep 2022 09:46:21 +0000 (11:46 +0200)] 
Fix GCS sensor system tests failing with DebugExecutor (#26742)

4 days agoDon't iterate tis list twice for no reason (#26740)
Daniel Standish [Wed, 28 Sep 2022 06:31:47 +0000 (23:31 -0700)] 
Don't iterate tis list twice for no reason (#26740)

Just don't need this line.

4 days agoHandle mapped tasks in task duration chart (#26722)
Daniel Standish [Wed, 28 Sep 2022 06:31:22 +0000 (23:31 -0700)] 
Handle mapped tasks in task duration chart (#26722)

Previously mapped tasks each existed as their own independent-but-unlabeled points.  Makes sense to aggregate them somehow.

4 days agoAdd oracledb thick mode support for oracle provider (#26576)
Paul Williams [Wed, 28 Sep 2022 06:14:46 +0000 (02:14 -0400)] 
Add oracledb thick mode support for oracle provider (#26576)

4 days agoRemove TaskFail duplicates check (#26714)
Daniel Standish [Wed, 28 Sep 2022 05:12:54 +0000 (22:12 -0700)] 
Remove TaskFail duplicates check (#26714)

This check was added in error.  The duplicates are supposed to be there.  The task duration
and gantt views process TaskFail records with the apparent intention of including durations
from non-final failed attempts to the overall duration of the run.

4 days agoAdd documentation to dag test function (#26713)
Daniel Imberman [Wed, 28 Sep 2022 03:54:25 +0000 (20:54 -0700)] 
Add documentation to dag test function (#26713)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Daniel Imberman <danielimberman@Daniels-MBP.lan>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
5 days agoRevert "Rename schema to database in `PostgresHook` (#26436)" (#26734)
Jed Cunningham [Tue, 27 Sep 2022 23:06:18 +0000 (16:06 -0700)] 
Revert "Rename schema to database in `PostgresHook` (#26436)" (#26734)

This reverts commit 642375f97de133edba1a6c1fa9397d840e8b5936.

The common sql tests didn't run in #26436, and there are failures there from this change.

I think they might point toward compatibility problems with the change, so reverting for now.

5 days agoFix task duration cumulative chart (#26717)
Daniel Standish [Tue, 27 Sep 2022 22:54:59 +0000 (15:54 -0700)] 
Fix task duration cumulative chart (#26717)

Previously the chart could go down, which should not happen for a cumulative chart.  We fix that here.

Also, we add task fail durations to the non-cumulative chart for consistency.

5 days agoFix non-hidden cumulative chart on duration view (#26716)
Daniel Standish [Tue, 27 Sep 2022 22:54:04 +0000 (15:54 -0700)] 
Fix non-hidden cumulative chart on duration view (#26716)

When you first load the page, cumulative is unchecked, but the chart itself is unhidden and appears below the non-cumulative chart.  Adding this line ensures that the initial value of the checkbox is respected.

5 days agoUpdate Airflow release docs (#26727)
Jed Cunningham [Tue, 27 Sep 2022 21:54:00 +0000 (14:54 -0700)] 
Update Airflow release docs (#26727)

Just some things I ran across when releasing 2.4.1rc1.

5 days agoFix broken URL for `docker-compose.yaml` (#26726)
Kaxil Naik [Tue, 27 Sep 2022 21:42:31 +0000 (22:42 +0100)] 
Fix broken URL for `docker-compose.yaml` (#26726)

This actually fixes it. There was some attempts of it on #26721, #26695, #26711

5 days agoChange links to `boto3` documentation (#26708)
Andrey Anshin [Tue, 27 Sep 2022 21:08:48 +0000 (01:08 +0400)] 
Change links to `boto3` documentation (#26708)

5 days agoFix static checks on main (#26728)
Jed Cunningham [Tue, 27 Sep 2022 20:49:21 +0000 (13:49 -0700)] 
Fix static checks on main (#26728)

#26423 broke static checks, looks like the full CI suite didn't run.

5 days agoRemove S3KeysUnchangedSensor from system test because it can only be run in poke...
syedahsn [Tue, 27 Sep 2022 19:16:23 +0000 (12:16 -0700)] 
Remove S3KeysUnchangedSensor from system test because it can only be run in poke mode, which causes it to fail in the system tests (#26606)

5 days agoAdd statsd overrideMappings in helm chart values (#26598)
Bob Du [Tue, 27 Sep 2022 18:42:12 +0000 (02:42 +0800)] 
Add statsd overrideMappings in helm chart values (#26598)

* add statsd overrideMappings in helm chart values

* add statsd overrideMappings unit test in helm chart

* fix statsd configmap unit test in helm chart

5 days agoFixed broken URL for docker-compose.yaml (#26721)
Kamesh Kotwani [Tue, 27 Sep 2022 18:36:02 +0000 (00:06 +0530)] 
Fixed broken URL for docker-compose.yaml (#26721)

The previous URL was pointing to a parent directory which resulted in a 404 error as file was not present in parent directory, rather the file is present in the same directory as the page, so the URL has been fixed to point to correct location.

5 days agoRename schema to database in `PostgresHook` (#26436)
Felix Uellendall [Tue, 27 Sep 2022 16:18:54 +0000 (18:18 +0200)] 
Rename schema to database in `PostgresHook` (#26436)

In PostgresHook the "schema" field is only being called like that to make it compatible with the underlying DbApiHook which uses the schema for the sql alchemy connector. The postgres connector library however does not allow setting a schema in the connection instead a database can be set. To clarify that, we change all references in the PostgresHook code and documentation.

5 days agoUpdate index.rst (#26711)
Marcelo Majer Franceschini [Tue, 27 Sep 2022 15:12:52 +0000 (12:12 -0300)] 
Update index.rst (#26711)

https://github.com/apache/airflow/pull/26695
Now it will work. Sorry

5 days agoAdd 'schedule' to example_video_intelligence.py (#26701)
Beata Kossakowska [Tue, 27 Sep 2022 15:06:14 +0000 (17:06 +0200)] 
Add 'schedule' to example_video_intelligence.py (#26701)

5 days agoFix pod_template imagePullPolicy (#26423)
Musaed Albrikan [Tue, 27 Sep 2022 15:05:45 +0000 (18:05 +0300)] 
Fix pod_template imagePullPolicy (#26423)

Co-authored-by: Musaed Albrikan <musaed@mrsool.co>
5 days agoAdd logging options to docker operator (#26653)
Nischith Shetty [Tue, 27 Sep 2022 14:42:37 +0000 (20:12 +0530)] 
Add logging options to docker operator (#26653)

5 days agoRefactor `SlackWebhookOperator`: Get rid of mandatory http-provider dependency (...
Andrey Anshin [Tue, 27 Sep 2022 14:41:20 +0000 (18:41 +0400)] 
Refactor `SlackWebhookOperator`: Get rid of mandatory http-provider dependency (#26648)

* Refactor `SlackWebhookOperator`: Get rid of mandatory http-provider dependency

* Fix tests selective tests for slack

5 days agoRemove duplicated connection-type within the provider (#26628)
Andrey Anshin [Tue, 27 Sep 2022 13:25:30 +0000 (17:25 +0400)] 
Remove duplicated connection-type within the provider (#26628)

5 days agoremove "ssh://" prefix from git repo value (#26632)
Will Herrmann [Tue, 27 Sep 2022 13:05:12 +0000 (07:05 -0600)] 
remove "ssh://" prefix from git repo value (#26632)

5 days agoWhen rendering template, unmap task in context (#26702)
Tzu-ping Chung [Tue, 27 Sep 2022 12:52:52 +0000 (20:52 +0800)] 
When rendering template, unmap task in context (#26702)

5 days agodocs: Fix typo (#26706)
Marcos Bressan [Tue, 27 Sep 2022 11:51:46 +0000 (08:51 -0300)] 
docs: Fix typo (#26706)

5 days agoMake build-images workflow name dynamic (#26690)
Jarek Potiuk [Tue, 27 Sep 2022 11:11:58 +0000 (13:11 +0200)] 
Make build-images workflow name dynamic (#26690)

As of today you can dynamically name your build-images workflow.
This will be nice change to be able to distinguish the "tests"
from build images workflows.

5 days agoFix PinotDB dependencies (#26705)
Jarek Potiuk [Tue, 27 Sep 2022 11:11:32 +0000 (13:11 +0200)] 
Fix PinotDB dependencies (#26705)

Our canary builds detected that PinotDB released 7 hours ago lacks
ciso8601 dependency, which make import fail during test collection.

This PR adds the dependency in provider until the issue is fixed.
PR has been raised to PinotDB

https://github.com/python-pinot-dbapi/pinot-dbapi/issues/47

5 days agoUpdate index.rst (#26695)
Marcelo Majer Franceschini [Tue, 27 Sep 2022 07:36:31 +0000 (04:36 -0300)] 
Update index.rst (#26695)

* Update index.rst

* Update index.rst

5 days agoSFTPOperator - add support for list of file paths (#26666)
Paul Williams [Tue, 27 Sep 2022 05:52:04 +0000 (01:52 -0400)] 
SFTPOperator - add support for list of file paths (#26666)

6 days agoFix intermittently failing tests for SFTPOperator (#26685)
Paul Williams [Mon, 26 Sep 2022 23:06:02 +0000 (19:06 -0400)] 
Fix intermittently failing tests for SFTPOperator (#26685)

6 days agoFix scroll overflow for ConfirmDialog (#26681)
pierrejeambrun [Mon, 26 Sep 2022 22:07:33 +0000 (00:07 +0200)] 
Fix scroll overflow for ConfirmDialog (#26681)

6 days agoAdd a note against use of top level code in timetable (#26649)
Ephraim Anierobi [Mon, 26 Sep 2022 22:01:14 +0000 (23:01 +0100)] 
Add a note against use of top level code in timetable (#26649)

Accessing Variables, Connections at top level of code is bad practice

Add a section in best practices ref for timetable

6 days agoCloud Data Loss Prevention Operators assets (#26618)
Beata Kossakowska [Mon, 26 Sep 2022 14:21:54 +0000 (16:21 +0200)] 
Cloud Data Loss Prevention Operators assets (#26618)

6 days agoRedshift to S3 and S3 to Redshift System test (AIP-47) (#26613)
syedahsn [Mon, 26 Sep 2022 14:17:24 +0000 (07:17 -0700)] 
Redshift to S3 and S3 to Redshift System test (AIP-47) (#26613)

* System test for Redshift to S3 using SystemTestContextBuilder pattern.

* Combine S3 to Redshift, and Redshift to S3 system tests

6 days agoAdd `DbtCloudListJobsOperator` (#26475)
Phani Kumar [Mon, 26 Sep 2022 13:49:54 +0000 (19:19 +0530)] 
Add `DbtCloudListJobsOperator` (#26475)

* Add dbt list jobs operator

* Apply review suggestion

6 days agoSimplify RTIF.delete_old_records() (#26667)
Tzu-ping Chung [Mon, 26 Sep 2022 13:49:17 +0000 (21:49 +0800)] 
Simplify RTIF.delete_old_records() (#26667)

A lot of the codes and comments are actually not relevant since we've
removed the funky execution_date based filtering in 2.3, and we can
simply the implementation quite a bit now.

6 days agoUse COALESCE when ordering runs to handle NULL (#26626)
Tzu-ping Chung [Mon, 26 Sep 2022 12:59:08 +0000 (20:59 +0800)] 
Use COALESCE when ordering runs to handle NULL (#26626)

Data interval columns are NULL for runs created before 2.3, but SQL's
NULL-sorting logic would make those old runs always appear first. In a
perfect world we'd want to sort by get_run_data_interval(), but that's
not efficient, so instead the columns are coalesced into logical date,
which is good enough in most cases.

6 days agoSkip sdist providers installation for non-canary builds (#26671)
Jarek Potiuk [Mon, 26 Sep 2022 09:22:14 +0000 (11:22 +0200)] 
Skip sdist providers installation for non-canary builds (#26671)

The sdist providers installation takes a long time and it is unlikely
to fail if wheel installation succeeds. We should skip running the
providers builds for PRs to speed them up. In case there will be
any problem with sdist installation, we will catch it in our
"canary" builds anyway.

6 days agoFix xcom arg.py .zip bug (#26636)
Robert J. McGinness [Mon, 26 Sep 2022 09:02:55 +0000 (05:02 -0400)] 
Fix xcom arg.py .zip bug (#26636)

6 days agoRemove asset compilation for CI image builds (#26665)
Jarek Potiuk [Mon, 26 Sep 2022 08:31:39 +0000 (10:31 +0200)] 
Remove asset compilation for CI image builds (#26665)

The asset compilation is not actually used in CI image so it is not
really needed to be run in CI.

6 days agoAdd limit to Sphinx < 5.2.0 until autoapi incompatibilities are fixed (#26670)
Jarek Potiuk [Mon, 26 Sep 2022 08:30:57 +0000 (10:30 +0200)] 
Add limit to Sphinx < 5.2.0 until autoapi incompatibilities are fixed (#26670)

The new (released 2 days ago) Sphinx 5.2.0 introduced deprecation
for he way properties are documented and sphinx-autoapi needs
to catch-up with it. Until that, we limit Sphinx to < 5.2.0

Tracked in https://github.com/readthedocs/sphinx-autoapi/issues/352

6 days agoClearer code for PodGenerator.deserialize_model_file (#26641)
Xiaodong DENG [Mon, 26 Sep 2022 07:59:24 +0000 (09:59 +0200)] 
Clearer code for PodGenerator.deserialize_model_file (#26641)

Especially for how it handles non-existent file.

When the file path received doesn't exist, the current way is to use yaml.safe_load()
to process it, and it returns the path as a string.

Then this string is passed to deserialize_model_dict() and results in an empty object.
Passing 'None' to deserialize_model_dict() will do the same, but the code will become clearer, and less misleading.

Meanwhile, when the model file path received doesn't exist, there should be a warning in the log.

(This change shouldn't cause any behaviour change)

6 days agoAdd Vroom to the list of companies using Apache Airflow (#26643)
LI LIU [Mon, 26 Sep 2022 02:11:22 +0000 (22:11 -0400)] 
Add Vroom to the list of companies using Apache Airflow (#26643)

7 days agoAdd requirements.txt example to "extending docker" (#26663)
Jarek Potiuk [Sun, 25 Sep 2022 22:19:08 +0000 (00:19 +0200)] 
Add requirements.txt example to "extending docker" (#26663)

When extending the image, there was an example of adding individual
packages via PyPI, but not requirements.txt. This PR fixes it.

7 days agoDo not restart breeze containers after restart (#26647)
Jarek Potiuk [Sun, 25 Sep 2022 19:52:50 +0000 (21:52 +0200)] 
Do not restart breeze containers after restart (#26647)

In order to avoid initialisation of database and other integrations
when you are entering/leaving breeze, those are started when breeze
starts but not stopped when you leave. Then stopping such running
auxiliary containers should be done with `breeze stop` after you
are done. However those who do not do it, and will restart their
machine will find that the containers get restarted.

This has been added as part of #13446 where health-checks are added.
However "always" was not a good choice. It should have been "on-failure"

8 days agoRemove comon provider docs from ad-hoc releases (#26645)
Jarek Potiuk [Sat, 24 Sep 2022 00:24:30 +0000 (02:24 +0200)] 
Remove comon provider docs from ad-hoc releases (#26645)

When we are releasing ad-hoc providers rather than all of them,
we rarely add new stuff (they are mostly bugfixes), so we should
rather not update the "common provider documentation", because
for example when we add (but not yet release) a new provider,
the common indexes might include it and confuse the users.

9 days agoFix one_done trigger rule tests (#26642)
Jed Cunningham [Fri, 23 Sep 2022 22:48:01 +0000 (15:48 -0700)] 
Fix one_done trigger rule tests (#26642)

9 days agoAllow MapXComArg to resolve after serialization (#26591)
Tzu-ping Chung [Fri, 23 Sep 2022 20:33:01 +0000 (04:33 +0800)] 
Allow MapXComArg to resolve after serialization  (#26591)

This is useful for cases where we want to resolve an XCom without
running a worker, e.g. to display the value in UI.

Since we don't want to actually call the mapper function in this case
(the function is arbitrary code, and not running it is the entire point
to serialize operators), "resolving" the XComArg in this case would
merely produce some kind of quasi-meaningful string representation,
instead of the actual value we'd get in the worker.

Also note that this only affects a very small number of cases, since
once a worker is run for the task instance, RenderedTaskInstanceFields
would store the real resolved value and take over UI representation,
avoiding this fake resolving logic to be accessed at all.

9 days agoResolve deprecation warning re Table.exists() (#26616)
Daniel Standish [Fri, 23 Sep 2022 20:30:38 +0000 (13:30 -0700)] 
Resolve deprecation warning re Table.exists() (#26616)

The deprecation warning tells us we need to change Table.exists() to Inspector.has_table().  It turns out we're already using the right method, but we weren't passing it the right object.  It wants a table name but we were giving it a table.

9 days agoUpdate docs when not to use helm hooks for starting jobs (#26604)
Hans Kristian Flaatten [Fri, 23 Sep 2022 20:29:09 +0000 (22:29 +0200)] 
Update docs when not to use helm hooks for starting jobs (#26604)

9 days agoCheck user is active (#26635)
Jed Cunningham [Fri, 23 Sep 2022 20:28:33 +0000 (13:28 -0700)] 
Check user is active (#26635)

9 days agoFormat the note in datasets docs (#26637)
Mark Norman Francis [Fri, 23 Sep 2022 18:13:31 +0000 (19:13 +0100)] 
Format the note in datasets docs (#26637)

9 days agoAdd ``one_done`` trigger rule (#26146)
Pankaj Singh [Fri, 23 Sep 2022 17:05:28 +0000 (22:35 +0530)] 
Add ``one_done`` trigger rule (#26146)

closes: #26099

9 days agoResolve warning about DISTINCT ON query on dags view (#26608)
Daniel Standish [Fri, 23 Sep 2022 08:52:26 +0000 (01:52 -0700)] 
Resolve warning about DISTINCT ON query on dags view (#26608)

Closes #26607

9 days agoBetter @task_group typing powered by ParamSpec and pre-commit (#26366)
Tzu-ping Chung [Fri, 23 Sep 2022 08:16:17 +0000 (16:16 +0800)] 
Better @task_group typing powered by ParamSpec and pre-commit (#26366)

9 days agoSuppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)
Daniel Standish [Fri, 23 Sep 2022 07:18:52 +0000 (00:18 -0700)] 
Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)

We already have it set false in the main flask app config, but now that we do db init from orm, and we create a distinct flask app for that, and it doesn't get the same setting.

Closes #26544

9 days agoFix Deferrable stuck as "scheduled" during backfill (#26205)
Ephraim Anierobi [Fri, 23 Sep 2022 05:08:28 +0000 (06:08 +0100)] 
Fix Deferrable stuck as "scheduled" during backfill (#26205)

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
9 days agoNo missing user warning for public admin (#26611)
Daniel Standish [Fri, 23 Sep 2022 05:05:53 +0000 (22:05 -0700)] 
No missing user warning for public admin (#26611)

If airflow has been configured such that public role is admin, there's no need to add users, so we shouldn't warn if there are none.

10 days agoAdd Pierre to committers (#26609)
Jarek Potiuk [Thu, 22 Sep 2022 21:45:15 +0000 (23:45 +0200)] 
Add Pierre to committers (#26609)

10 days agoAdd unit test for log retrieval url (#26603)
Jarek Potiuk [Thu, 22 Sep 2022 20:05:34 +0000 (22:05 +0200)] 
Add unit test for log retrieval url (#26603)

* Add unit test for log retrieval url

Added unit test to #26493

10 days agoRefactor SlackWebhookHook in order to use `slack_sdk` instead of HttpHook methods...
Andrey Anshin [Thu, 22 Sep 2022 19:48:02 +0000 (23:48 +0400)] 
Refactor SlackWebhookHook in order to use `slack_sdk` instead of HttpHook methods (#26452)

* Refactor SlackWebhookHook in order to use `slack_sdk` instead of HttpHook methods

10 days agoFix grid view log try numbers (#26556)
Brent Bovenzi [Thu, 22 Sep 2022 19:39:14 +0000 (15:39 -0400)] 
Fix grid view log try numbers (#26556)

* redo how try numbers are displayed in grid view logs

* rename attempt -> tryNumber

10 days agoFix 404 taskInstance errors and split into two tables (#26575)
pierrejeambrun [Thu, 22 Sep 2022 19:38:43 +0000 (21:38 +0200)] 
Fix 404 taskInstance errors and split into two tables (#26575)

10 days agoCreate a more efficient airflow dag test command that also has better local logging...
Daniel Imberman [Thu, 22 Sep 2022 16:14:29 +0000 (09:14 -0700)] 
Create a more efficient  airflow dag test command that also has better local logging (#26400)

The current `airflow dag test` command is insufficient for an iterative local workflow when developing dags. The current system relies on running a backfill job which a) is extremely slow and b) only shows scheduler-level logs, so the user would need to search for log files to see how their tasks actually ran!

This new system

## Speed

With this new system, we see a notable speedup in between-task speed. The new airflow dag test command can load tasks immediately and not wait for a scheduler heartbeat or other irrelevant airflow processes.

### Before:

example bash operator: 13 seconds
example python operator: 26 seconds

### After

example bash operator: 6 seconds
example python operator: 16 seconds

## Logging

Previously, the `airflow dag test` command would pass on scheduler-level logs. This gives essentially 0 useful information for a DAG writer, and would force them to seek out task.log files in their filesystem every time they run a test dag. To improve this, we now surface task-level logs to the command line. We also no longer have any scheduler-level logs as we are no longer using a scheduler!

### Before

```
[2022-09-14 14:37:48,310] {dagbag.py:522} INFO - Filling up the DagBag from /files/dags
[2022-09-14 14:37:48,967] {base_executor.py:93} INFO - Adding to queue: ['<TaskInstance: example_bash_operator.runme_0 backfill__2022-01-02T00:00:00+00:00 [queued]>']
[2022-09-14 14:37:48,973] {base_executor.py:93} INFO - Adding to queue: ['<TaskInstance: example_bash_operator.runme_1 backfill__2022-01-02T00:00:00+00:00 [queued]>']
[2022-09-14 14:37:48,979] {base_executor.py:93} INFO - Adding to queue: ['<TaskInstance: example_bash_operator.runme_2 backfill__2022-01-02T00:00:00+00:00 [queued]>']
[2022-09-14 14:37:48,985] {base_executor.py:93} INFO - Adding to queue: ['<TaskInstance: example_bash_operator.also_run_this backfill__2022-01-02T00:00:00+00:00 [queued]>']
[2022-09-14 14:37:48,991] {base_executor.py:93} INFO - Adding to queue: ['<TaskInstance: example_bash_operator.this_will_skip backfill__2022-01-02T00:00:00+00:00 [queued]>']
[2022-09-14 14:37:54,045] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:37:54,045] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "example_bash_operator__runme_0__20220102" && sleep 1']
[2022-09-14 14:37:54,053] {subprocess.py:85} INFO - Output:
[2022-09-14 14:37:54,055] {subprocess.py:92} INFO - example_bash_operator__runme_0__20220102
[2022-09-14 14:37:55,061] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:37:55,117] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:37:55,117] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "example_bash_operator__runme_1__20220102" && sleep 1']
[2022-09-14 14:37:55,124] {subprocess.py:85} INFO - Output:
[2022-09-14 14:37:55,125] {subprocess.py:92} INFO - example_bash_operator__runme_1__20220102
[2022-09-14 14:37:56,131] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:37:56,186] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:37:56,186] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "example_bash_operator__runme_2__20220102" && sleep 1']
[2022-09-14 14:37:56,193] {subprocess.py:85} INFO - Output:
[2022-09-14 14:37:56,194] {subprocess.py:92} INFO - example_bash_operator__runme_2__20220102
[2022-09-14 14:37:57,199] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:37:57,257] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:37:57,257] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "ti_key=example_bash_operator__also_run_this__20220102"']
[2022-09-14 14:37:57,265] {subprocess.py:85} INFO - Output:
[2022-09-14 14:37:57,266] {subprocess.py:92} INFO - ti_key=example_bash_operator__also_run_this__20220102
[2022-09-14 14:37:57,266] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:37:57,307] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:37:57,308] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "hello world"; exit 99;']
[2022-09-14 14:37:57,316] {subprocess.py:85} INFO - Output:
[2022-09-14 14:37:57,317] {subprocess.py:92} INFO - hello world
[2022-09-14 14:37:57,317] {subprocess.py:96} INFO - Command exited with return code 99
[2022-09-14 14:37:57,340] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 2 | succeeded: 4 | running: 0 | failed: 0 | skipped: 1 | deadlocked: 0 | not ready: 2
[2022-09-14 14:37:57,351] {base_executor.py:93} INFO - Adding to queue: ['<TaskInstance: example_bash_operator.run_after_loop backfill__2022-01-02T00:00:00+00:00 [queued]>']
[2022-09-14 14:37:58,918] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:37:58,918] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo 1']
[2022-09-14 14:37:58,925] {subprocess.py:85} INFO - Output:
[2022-09-14 14:37:58,926] {subprocess.py:92} INFO - 1
[2022-09-14 14:37:58,927] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:37:58,945] {dagrun.py:595} INFO - Marking run <DagRun example_bash_operator @ 2022-01-02T00:00:00+00:00: backfill__2022-01-02T00:00:00+00:00, state:running, queued_at: None. externally triggered: False> successful
[2022-09-14 14:37:58,946] {dagrun.py:657} INFO - DagRun Finished: dag_id=example_bash_operator, execution_date=2022-01-02T00:00:00+00:00, run_id=backfill__2022-01-02T00:00:00+00:00, run_start_date=2022-09-14 14:37:48.865989+00:00, run_end_date=2022-09-14 14:37:58.946180+00:00, run_duration=10.080191, state=success, external_trigger=False, run_type=backfill, data_interval_start=2022-01-02T00:00:00+00:00, data_interval_end=2022-01-03T00:00:00+00:00, dag_hash=None
[2022-09-14 14:37:58,946] {backfill_job.py:381} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 5 | running: 0 | failed: 0 | skipped: 2 | deadlocked: 0 | not ready: 0
[2022-09-14 14:37:58,949] {backfill_job.py:875} INFO - Backfill done for DAG <DAG: example_bash_operator>. Exiting.
```

### After:

```
root@613d334d9c9d:/opt/airflow# airflow dags test example_bash_operator  2022-01-03
[2022-09-14 14:43:45,232] {dagbag.py:522} INFO - Filling up the DagBag from /files/dags
[2022-09-14 14:43:45,997] {dag_command.py:556} INFO - dagrun id:example_bash_operator
/opt/airflow/airflow/cli/commands/dag_command.py:573 RemovedInAirflow3Warning: Calling `DAG.create_dagrun()` without an explicit data interval is deprecated
created dagrun <DagRun example_bash_operator @ 2022-01-03T00:00:00+00:00: manual__2022-01-03T00:00:00+00:00, state:running, queued_at: None. externally triggered: False>
[2022-09-14 14:43:46,013] {dag_command.py:496} INFO - starting dagrun
[2022-09-14 14:43:46,138] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:46,138] {dag_command.py:543} INFO - Running task runme_0
[2022-09-14 14:43:46,144] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:43:46,145] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "example_bash_operator__runme_0__20220103" && sleep 1']
[2022-09-14 14:43:46,152] {subprocess.py:85} INFO - Output:
[2022-09-14 14:43:46,154] {subprocess.py:92} INFO - example_bash_operator__runme_0__20220103
[2022-09-14 14:43:47,158] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:43:47,179] {dag_command.py:547} INFO - runme_0 ran successfully!
[2022-09-14 14:43:47,180] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:47,197] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:47,197] {dag_command.py:543} INFO - Running task runme_1
[2022-09-14 14:43:47,203] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:43:47,204] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "example_bash_operator__runme_1__20220103" && sleep 1']
[2022-09-14 14:43:47,211] {subprocess.py:85} INFO - Output:
[2022-09-14 14:43:47,213] {subprocess.py:92} INFO - example_bash_operator__runme_1__20220103
[2022-09-14 14:43:48,219] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:43:48,239] {dag_command.py:547} INFO - runme_1 ran successfully!
[2022-09-14 14:43:48,239] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:48,253] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:48,253] {dag_command.py:543} INFO - Running task runme_2
[2022-09-14 14:43:48,259] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:43:48,259] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "example_bash_operator__runme_2__20220103" && sleep 1']
[2022-09-14 14:43:48,266] {subprocess.py:85} INFO - Output:
[2022-09-14 14:43:48,267] {subprocess.py:92} INFO - example_bash_operator__runme_2__20220103
[2022-09-14 14:43:49,273] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:43:49,294] {dag_command.py:547} INFO - runme_2 ran successfully!
[2022-09-14 14:43:49,295] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:49,374] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:49,374] {dag_command.py:543} INFO - Running task also_run_this
[2022-09-14 14:43:49,380] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:43:49,380] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "ti_key=example_bash_operator__also_run_this__20220103"']
[2022-09-14 14:43:49,389] {subprocess.py:85} INFO - Output:
[2022-09-14 14:43:49,390] {subprocess.py:92} INFO - ti_key=example_bash_operator__also_run_this__20220103
[2022-09-14 14:43:49,390] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:43:49,395] {dag_command.py:547} INFO - also_run_this ran successfully!
[2022-09-14 14:43:49,395] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:49,406] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:49,407] {dag_command.py:543} INFO - Running task this_will_skip
[2022-09-14 14:43:49,412] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:43:49,413] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo "hello world"; exit 99;']
[2022-09-14 14:43:49,420] {subprocess.py:85} INFO - Output:
[2022-09-14 14:43:49,421] {subprocess.py:92} INFO - hello world
[2022-09-14 14:43:49,422] {subprocess.py:96} INFO - Command exited with return code 99
[2022-09-14 14:43:49,422] {dag_command.py:549} INFO - Task Skipped, continuing
[2022-09-14 14:43:49,422] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:49,440] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:49,440] {dag_command.py:543} INFO - Running task run_after_loop
[2022-09-14 14:43:49,445] {subprocess.py:62} INFO - Tmp dir root location:
 /tmp
[2022-09-14 14:43:49,446] {subprocess.py:74} INFO - Running command: ['/bin/bash', '-c', 'echo 1']
[2022-09-14 14:43:49,453] {subprocess.py:85} INFO - Output:
[2022-09-14 14:43:49,454] {subprocess.py:92} INFO - 1
[2022-09-14 14:43:49,454] {subprocess.py:96} INFO - Command exited with return code 0
[2022-09-14 14:43:49,458] {dag_command.py:547} INFO - run_after_loop ran successfully!
[2022-09-14 14:43:49,458] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:49,471] {dag_command.py:542} INFO - *****************************************************
[2022-09-14 14:43:49,471] {dag_command.py:543} INFO - Running task run_this_last
[2022-09-14 14:43:49,475] {dag_command.py:547} INFO - run_this_last ran successfully!
[2022-09-14 14:43:49,475] {dag_command.py:550} INFO - *****************************************************
[2022-09-14 14:43:49,480] {dagrun.py:595} INFO - Marking run <DagRun example_bash_operator @ 2022-01-03T00:00:00+00:00: manual__2022-01-03T00:00:00+00:00, state:running, queued_at: None. externally triggered: False> successful
[2022-09-14 14:43:49,480] {dagrun.py:657} INFO - DagRun Finished: dag_id=example_bash_operator, execution_date=2022-01-03T00:00:00+00:00, run_id=manual__2022-01-03T00:00:00+00:00, run_start_date=2022-01-03T00:00:00+00:00, run_end_date=2022-09-14 14:43:49.480895+00:00, run_duration=21998629.480895, state=success, external_trigger=False, run_type=manual, data_interval_start=2022-01-03T00:00:00+00:00, data_interval_end=2022-01-04T00:00:00+00:00, dag_hash=None
```

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).