Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add .sql template extension rendering for BigQueryInsertJobOperator #22859

Closed
2 tasks done
LennyKLB opened this issue Apr 8, 2022 · 4 comments
Closed
2 tasks done

Add .sql template extension rendering for BigQueryInsertJobOperator #22859

LennyKLB opened this issue Apr 8, 2022 · 4 comments
Assignees
Labels
kind:feature Feature Requests provider:google Google (including GCP) related issues

Comments

@LennyKLB
Copy link
Contributor

LennyKLB commented Apr 8, 2022

Description

Add .sql as value to template_ext in airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator

Use case/motivation

It would be helpful if Jinja would also read .sql templates in airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator, especially for its configuration.query.query parameter. It will render that parameter as sql correctly, but only the outer configuration body -- which is based on Bigquery/rest/v2/Job#jobconfiguration -- is considered for extension-based rendering and only json is considered.

I think there is a clear use-case to support templating SQL-only files. I know that BigQueryExecuteQueryOperator has this but it's deprecated in favour of BigQueryInsertJobOperator.

I currently work around this using a really basic override:

class BigQueryTemplateFileOperator(BigQueryInsertJobOperator):
    template_ext = (".json", ".sql")

However, I'd like to avoid having to replicate this operator everywhere in our codebase if this might actually already be implemented soon.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@LennyKLB LennyKLB added the kind:feature Feature Requests label Apr 8, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Apr 8, 2022

Thanks for opening your first issue here! Be sure to follow the issue template!

@josh-fell
Copy link
Contributor

@LennyKLB Assigned to you!

@mik-laj mik-laj added the provider:google Google (including GCP) related issues label Apr 9, 2022
LennyKLB added a commit to LennyKLB/airflow that referenced this issue Apr 11, 2022
@LennyKLB
Copy link
Contributor Author

The change itself is so small that I refrained from adding/editing tests. I found that the BigQuery-related tests didn't feature any file-based templating.
If this is nonetheless still necessary, could someone point me in the right direction in terms of inspiration?

@josh-fell
Copy link
Contributor

Closing via #22920.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feature Feature Requests provider:google Google (including GCP) related issues
Projects
None yet
Development

No branches or pull requests

3 participants
  翻译: