More documentation update for June providers release (#16405)

The provider changelogs were already merged but they could not be released due to unavailability of signing key in remote location The documentation has once more been updated, including latest merges - documentation-only changes were removed from bugs/feetures/breaking changes lists. Few other improvements: * Pre-commit was added to make sure that the documentation in provider's index.rst files includes the latest changelog. * Index.rst now contain includes of the CHANGELOG.rst rather than copy of the CHANGELOG.rst * The `prepare-provider-package` breeze command has --non-interactive flag now * generated provider package README.rst contain changelog so you can see the changelog directly in PyPI * "Suggest change on this page" link in documentation is fixed.
apache · Jun 15, 2021 · 1fba540 · 1fba540
1 parent e8d3de8
commit 1fba540
Show file tree

Hide file tree

Showing 214 changed files with 944 additions and 4,419 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -320,6 +320,8 @@ jobs:
         uses: actions/checkout@v2
         with:
           persist-credentials: false
+          # Needs to fetch all history to be able to verify changelog generation
+          fetch-depth: 0
       - name: "Setup python"
         uses: actions/setup-python@v2
         with:
@@ -519,7 +521,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
       VERSION_SUFFIX_FOR_PYPI: ".dev0"
       GITHUB_REGISTRY: ${{ needs.ci-images.outputs.githubRegistry }}
-      NO_INTERACTIVE: "true"
+      NON_INTERACTIVE: "true"
     if: needs.build-info.outputs.image-build == 'true' && needs.build-info.outputs.default-branch == 'main'
     steps:
       - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -543,6 +543,24 @@ repos:
         pass_filenames: false
         require_serial: true
         additional_dependencies: ['rich']
+      - id: providers-changelogs
+        name: Update changelogs generated for providers
+        language: python
+        entry: ./dev/provider_packages/prepare_provider_packages.py
+        args: ['update-changelogs']
+        files: providers/.*/CHANGELOG.rst
+        pass_filenames: true
+        additional_dependencies:
+          - 'click~=7.0'
+          - 'jinja2~=2.10'
+          - 'keyring==10.1'
+          - 'PyGithub'
+          - 'jsonpath_ng'
+          - 'jsonschema'
+          - 'pyyaml'
+          - 'packaging'
+          - 'rich'
+          - 'tabulate'
       - id: markdownlint
         name: Run markdownlint
         description: Checks the style of Markdown files.

diff --git a/BREEZE.rst b/BREEZE.rst
@@ -2137,6 +2137,10 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
           Default: both
 
+  --non-interactive
+
+          Runs the command in non-interactive mode.
+
   -v, --verbose
           Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
           debugging - when you run breeze with --verbose flags you will be able to see the commands
@@ -2242,12 +2246,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
                  language-matters lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending
                  mypy mypy-helm no-providers-in-core-examples no-relative-imports
                  pre-commit-descriptions pre-commit-hook-names pretty-format-json
-                 provide-create-sessions providers-init-file provider-yamls pydevd pydocstyle pylint
-                 pylint-tests python-no-log-warn pyupgrade restrict-start_date rst-backticks
-                 setup-order setup-extra-packages shellcheck sort-in-the-wild sort-spelling-wordlist
-                 stylelint trailing-whitespace ui-lint update-breeze-file update-extras
-                 update-local-yml-file update-setup-cfg-file verify-db-migrations-documented
-                 version-sync www-lint yamllint
+                 provide-create-sessions providers-changelogs providers-init-file provider-yamls
+                 pydevd pydocstyle pylint pylint-tests python-no-log-warn pyupgrade
+                 restrict-start_date rst-backticks setup-order setup-extra-packages shellcheck
+                 sort-in-the-wild sort-spelling-wordlist stylelint trailing-whitespace ui-lint
+                 update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
+                 verify-db-migrations-documented version-sync www-lint yamllint
 
         You can pass extra arguments including options to the pre-commit framework as
         <EXTRA_ARGS> passed after --. For example:

diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
@@ -146,6 +146,8 @@ require Breeze Docker images to be installed locally:
 ----------------------------------- ---------------------------------------------------------------- ------------
 ``provide-create-sessions``           Make sure provide-session and create-session imports are OK
 ----------------------------------- ---------------------------------------------------------------- ------------
+``providers-changelogs``              Updates documentation for providers changelogs
+----------------------------------- ---------------------------------------------------------------- ------------
 ``providers-init-file``               Check that provider's __init__.py file is removed
 ----------------------------------- ---------------------------------------------------------------- ------------
 ``provider-yamls``                    Checks that provider.yaml files have the correct content

diff --git a/airflow/providers/airbyte/CHANGELOG.rst b/airflow/providers/airbyte/CHANGELOG.rst
@@ -27,17 +27,26 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 Features
 ~~~~~~~~
 
 * ``Add test_connection method to Airbyte hook (#16236)``
-* ``Add missing docstring params (#15741)``
 
 Bug Fixes
 ~~~~~~~~~
 
 * ``Fix hooks extended from http hook (#16109)``
 
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Updated documentation for June 2021 provider release (#16294)``
+   * ``Add missing docstring params (#15741)``
+
 1.0.0
 .....
 

diff --git a/airflow/providers/amazon/CHANGELOG.rst b/airflow/providers/amazon/CHANGELOG.rst
@@ -27,28 +27,41 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 Features
 ~~~~~~~~
 
-* ``Add Connection Documentation for the Hive Provider (#15704)``
 * ``CloudwatchTaskHandler reads timestamp from Cloudwatch events (#15173)``
 * ``remove retry for now (#16150)``
 * ``Remove the 'not-allow-trailing-slash' rule on S3_hook (#15609)``
+* ``Add support of capacity provider strategy for ECSOperator (#15848)``
+* ``Update copy command for s3 to redshift (#16241)``
 
 Bug Fixes
 ~~~~~~~~~
 
 * ``Fix S3 Select payload join (#16189)``
 * ``Fix spacing in 'AwsBatchWaitersHook' docstring (#15839)``
-* ``Fix spelling (#15699)``
 * ``MongoToS3Operator failed when running with a single query (not aggregate pipeline) (#15680)``
+* ``fix: AwsGlueJobOperator change order of args for load_file (#16216)``
 
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Check synctatic correctness for code-snippets (#16005)``
    * ``Bump pyupgrade v2.13.0 to v2.18.1 (#15991)``
    * ``Rename example bucket names to use INVALID BUCKET NAME by default (#15651)``
    * ``Docs: Replace 'airflow' to 'apache-airflow' to install extra (#15628)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
+   * ``Add Connection Documentation for the Hive Provider (#15704)``
+   * ``Update Docstrings of Modules with Missing Params (#15391)``
+   * ``Fix spelling (#15699)``
+   * ``Add Connection Documentation for Providers (#15499)``
+
+.. Review and move the new changes to one of the sections above:
 
 1.4.0
 .....
@@ -57,14 +70,12 @@ Features
 ~~~~~~~~
 
 * ``S3Hook.load_file should accept Path object in addition to str (#15232)``
-* ``Add Connection Documentation for Providers (#15499)``
 
 Bug fixes
 ~~~~~~~~~
 
 * ``Fix 'logging.exception' redundancy (#14823)``
 * ``Fix AthenaSensor calling AthenaHook incorrectly (#15427)``
-* ``Update Docstrings of Modules with Missing Params (#15391)``
 * ``Add links to new modules for deprecated modules (#15316)``
 * ``Fixes doc for SQSSensor (#15323)``
 

diff --git a/airflow/providers/apache/beam/CHANGELOG.rst b/airflow/providers/apache/beam/CHANGELOG.rst
@@ -27,11 +27,17 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Rename the main branch of the Airflow repo to be main (#16149)``
    * ``Check synctatic correctness for code-snippets (#16005)``
    * ``Rename example bucket names to use INVALID BUCKET NAME by default (#15651)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 2.0.0
 .....

diff --git a/airflow/providers/apache/cassandra/CHANGELOG.rst b/airflow/providers/apache/cassandra/CHANGELOG.rst
@@ -27,12 +27,18 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Adds interactivity when generating provider documentation. (#15518)``
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Remove Backport Providers (#14886)``
    * ``Update documentation for broken package releases (#14734)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 1.0.1
 .....

diff --git a/airflow/providers/apache/druid/CHANGELOG.rst b/airflow/providers/apache/druid/CHANGELOG.rst
@@ -27,13 +27,19 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Bump pyupgrade v2.13.0 to v2.18.1 (#15991)``
    * ``Adds interactivity when generating provider documentation. (#15518)``
    * ``Fix string concatenation using 'f-strings' (#15200)``
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Remove Backport Providers (#14886)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 1.1.0
 .....

diff --git a/airflow/providers/apache/hdfs/CHANGELOG.rst b/airflow/providers/apache/hdfs/CHANGELOG.rst
@@ -27,18 +27,20 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
-Features
-~~~~~~~~
-
-* ``Created initial guide for HDFS operators  (#11212)``
-* ``Update Docstrings of Modules with Missing Params (#15391)``
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
 
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Remove python2 related handlings and dependencies (#15301)``
    * ``Remove Backport Providers (#14886)``
    * ``Update documentation for broken package releases (#14734)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
+   * ``Update Docstrings of Modules with Missing Params (#15391)``
+   * ``Created initial guide for HDFS operators  (#11212)``
 
 1.0.1
 .....

diff --git a/airflow/providers/apache/hive/CHANGELOG.rst b/airflow/providers/apache/hive/CHANGELOG.rst
@@ -27,10 +27,10 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
-Features
-~~~~~~~~
-
-* ``Add Connection Documentation for the Hive Provider (#15704)``
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
 
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
@@ -39,6 +39,8 @@ Features
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Make Airflow code Pylint 2.8 compatible (#15534)``
    * ``Use Pip 21.* to install airflow officially (#15513)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
+   * ``Add Connection Documentation for the Hive Provider (#15704)``
 
 1.0.3
 .....

diff --git a/airflow/providers/apache/kylin/CHANGELOG.rst b/airflow/providers/apache/kylin/CHANGELOG.rst
@@ -27,17 +27,19 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
-Features
-~~~~~~~~
-
-* ``Update KylinHook docstring (#15602)``
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
 
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Adds interactivity when generating provider documentation. (#15518)``
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Remove Backport Providers (#14886)``
    * ``Update documentation for broken package releases (#14734)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
+   * ``Update KylinHook docstring (#15602)``
 
 1.0.1
 .....

diff --git a/airflow/providers/apache/livy/CHANGELOG.rst b/airflow/providers/apache/livy/CHANGELOG.rst
@@ -27,8 +27,14 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 1.1.0
 .....

diff --git a/airflow/providers/apache/pig/CHANGELOG.rst b/airflow/providers/apache/pig/CHANGELOG.rst
@@ -27,12 +27,18 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Adds interactivity when generating provider documentation. (#15518)``
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Remove Backport Providers (#14886)``
    * ``Update documentation for broken package releases (#14734)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 1.0.1
 .....

diff --git a/airflow/providers/apache/pinot/CHANGELOG.rst b/airflow/providers/apache/pinot/CHANGELOG.rst
@@ -27,12 +27,18 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Adds interactivity when generating provider documentation. (#15518)``
    * ``Prepares provider release after PIP 21 compatibility (#15576)``
    * ``Make Airflow code Pylint 2.8 compatible (#15534)``
    * ``Remove Backport Providers (#14886)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 1.0.1
 .....

diff --git a/airflow/providers/apache/spark/CHANGELOG.rst b/airflow/providers/apache/spark/CHANGELOG.rst
@@ -27,8 +27,21 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
+Bug fixes
+~~~~~~~~~
+
+* ``Make SparkSqlHook use Connection (#15794)``
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
+   * ``Updated documentation for June 2021 provider release (#16294)``
+
+.. Review and move the new changes to one of the sections above:
 
 1.0.3
 .....

diff --git a/airflow/providers/apache/sqoop/CHANGELOG.rst b/airflow/providers/apache/sqoop/CHANGELOG.rst
@@ -27,6 +27,11 @@ Breaking changes
 
 * ``Auto-apply apply_default decorator (#15667)``
 
+.. warning:: Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
+   If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
+   Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
+   automatically and you will have to manually run ``airflow upgrade db`` to complete the migration.
+
 .. Below changes are excluded from the changelog. Move them to
    appropriate section above if needed. Do not delete the lines(!):
    * ``Fixes failing static checks after recent pre-commit upgrade (#16183)``
@@ -35,6 +40,7 @@ Breaking changes
    * ``Make Airflow code Pylint 2.8 compatible (#15534)``
    * ``Remove Backport Providers (#14886)``
    * ``Update documentation for broken package releases (#14734)``
+   * ``Updated documentation for June 2021 provider release (#16294)``
 
 1.0.1
 .....