KubernetesPodOperator stops tailing logs in long living task

[michalmisiewicz]

KubernetesPodOperator stops tailing logs after 4 hours in long living task. When log tailing is interrupted, task log is flooded by Event: pod had an event of type Running messages. Messages are generated every 2 seconds.
Log tail interruption occurs every time after 4 hours. Interruption can be related to timeout on AKS control plane. Nevertheless, KubernetesPodOperator should continue to tail logs after interruption. I'm going to submit fix.

Apache Airflow version: 1.10.12
Kubernetes version (if you are using kubernetes) (use kubectl version): 1.17.7
Environment: Azure Kubernetes Service

[2020-09-23 13:35:43,077] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 600: percentage: 2.00\n'
[2020-09-23 13:51:01,477] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 900: percentage: 3.00\n'
[2020-09-23 14:05:50,749] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 1200: percentage: 4.00\n'
[2020-09-23 14:20:42,323] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 1500: percentage: 5.00\n'
[2020-09-23 14:35:42,412] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 1800: percentage: 6.00\n'
[2020-09-23 14:50:52,880] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 2100: percentage: 7.00\n'
[2020-09-23 15:06:05,138] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 2400: percentage: 8.00\n'
[2020-09-23 15:21:24,700] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 2700: percentage: 9.00\n'
[2020-09-23 15:36:52,563] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3000: percentage: 10.00\n'
[2020-09-23 15:52:32,510] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3300: percentage: 11.00\n'
[2020-09-23 16:08:53,489] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3600: percentage: 12.00\n'
[2020-09-23 16:24:13,732] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900: percentage: 13.00\n'
[2020-09-23 16:24:14,633] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900: percentage: 13.00\n'
[2020-09-23 16:24:14,690] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900: percentage: 13.00\n'
[2020-09-23 16:24:14,883] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900: percentage: 13.00\n'
[2020-09-23 16:39:40,816] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 4200: percentage: 14.00\n'
[2020-09-23 16:55:21,781] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 4500: percentage: 15.00\n'
[2020-09-23 17:10:58,901] {pod_launcher.py:171} INFO - Event: some-pod had an event of type Running
[2020-09-23 17:10:58,927] {pod_launcher.py:166} INFO - Pod some-pod has state running
[2020-09-23 17:11:01,148] {pod_launcher.py:171} INFO - Event: some-pod had an event of type Running
[2020-09-23 17:11:01,154] {pod_launcher.py:166} INFO - Pod some-pod has state running
[2020-09-23 17:11:01,154] {pod_launcher.py:166} INFO - Pod some-pod has state running
...
the same message for next 10 hours every 2 seconds