You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add .sql as value to template_ext in airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator
Use case/motivation
It would be helpful if Jinja would also read .sql templates in airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator, especially for its configuration.query.query parameter. It will render that parameter as sql correctly, but only the outer configuration body -- which is based on Bigquery/rest/v2/Job#jobconfiguration -- is considered for extension-based rendering and only json is considered.
I think there is a clear use-case to support templating SQL-only files. I know that BigQueryExecuteQueryOperator has this but it's deprecated in favour of BigQueryInsertJobOperator.
I currently work around this using a really basic override:
class BigQueryTemplateFileOperator(BigQueryInsertJobOperator):
template_ext = (".json", ".sql")
However, I'd like to avoid having to replicate this operator everywhere in our codebase if this might actually already be implemented soon.
The change itself is so small that I refrained from adding/editing tests. I found that the BigQuery-related tests didn't feature any file-based templating.
If this is nonetheless still necessary, could someone point me in the right direction in terms of inspiration?
Description
Add
.sql
as value totemplate_ext
inairflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator
Use case/motivation
It would be helpful if Jinja would also read
.sql
templates inairflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator
, especially for itsconfiguration.query.query
parameter. It will render that parameter assql
correctly, but only the outerconfiguration
body -- which is based on Bigquery/rest/v2/Job#jobconfiguration -- is considered for extension-based rendering and onlyjson
is considered.I think there is a clear use-case to support templating SQL-only files. I know that
BigQueryExecuteQueryOperator
has this but it's deprecated in favour ofBigQueryInsertJobOperator
.I currently work around this using a really basic override:
However, I'd like to avoid having to replicate this operator everywhere in our codebase if this might actually already be implemented soon.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: