-
Notifications
You must be signed in to change notification settings - Fork 14k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Enable individual trigger logging (#27758)
FileTaskHandler is updated so that it can be used by triggerer job to write messages from distinct triggers such that they are visible in web UI task logs. Messages from all relevant sources are merged and interleaved in the same task log pane. Due to variation in FileTaskHandler implementations, we needed some extra params set so triggerer job can use them appropriately. For some we need to use QueueListener pattern, for some we need to put many instances (one per trigger) behind a wrapper class. For this purpose we introduce some config params you can set on handler instance or class. Context vars are used so that handler can map messages to the correct TI. And these changes to log handling in triggerer job are entirely disableable. Also add log serving capability to triggerer service for live logs with non-streaming handlers. Added a TaskReturnCode for signaling so that log messages can be more accurate when deferring. And when resuming we now have better messaging i.e. "resuming from deferral". New method in FileTaskHandler which is for reading remote logs. It's called along with other sources in _read and interleaved there. Meta messages and actual log content are now separated so that messages are displayed at top after interleaving. For interleaving we need to parse log timestamp so we make this pluggable in case user formatter is weird. TI.next_kwargs is used to infer "has deferred" for "resuming from deferral messaging". TabWithTooltip class is added in web UI js. It can be used for adding tooltips for tabs. We didn't end up needing it, but Brent went through the trouble to write it and it may come in handy. In the chart, we need triggerer to be a service (instead of deployment) in order for logs serving to work out of the box. This is only enabled when airflowVersion in your chart values is >= 2.6.0. --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
- Loading branch information
1 parent
8338926
commit 1b18a50
Showing
54 changed files
with
2,390 additions
and
488 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.