Airflow task instances disappearing, unable to restart

[lihan]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

This happens on v2.6.2, but likely also on other earlier versions.

Following are the bug observations:

• Task instances from already successful dag runs went disappearing (Refer to below screenshot). The dag run itself stays successful (green block), but there was no task instances exist in the meta database.

• Attempt to click "Clear existing tasks" dag run to retry the dag, but there was again no task instances created, and the dag run state transitioned from queued to succeed immediately

• Same thing happens if click "Queue up new tasks" button, the dag run immediately went back to normal state
  

• This issue happened after Airflow tasks went disappear after airflow db clean #32596

Also see a video recording.

airflow-bug-short.mov

What you think should happen instead

No response

How to reproduce

I cannot easily reproduce this locally, but it happens on production env.

Operating System

Docker python 3.9

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

Unlikely relate to the deployment.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Can you please check if the DagRun in question has no Task Instance entries? That would confirm that the root cause is #32684

[lihan]

Yes, dag_run object exists in the meta DB, but no associated task instances.

[hussein-awala]

You can check my comment, you can edit the query I provided to clean your history, you just need to detect the empty dag runs, delete them, then recreate them using the backfill command.

[potiuk]

Closing as duplicate. Good we have it as it will serve as a guide for others who will have similar issue.