Use the Dataflow job monitoring interface

When you run your pipeline by using Dataflow, you can view that job and any others by using the Dataflow monitoring interface. The monitoring interface lets you see and interact with your Dataflow jobs.

You can access the Dataflow monitoring interface in the Google Cloud console.

Tasks that you can perform by using the monitoring interface include the following:

  • See a list of running, completed, and failed jobs.
  • View a graphical representation of a job's stages and the progress of each stage
  • View graphs of job metrics, such as data freshness, resource utilization, and I/O requests.
  • Monitor the estimated cost of a job.
  • View pipeline logs.
  • Identify which steps might cause pipeline lag.
  • Identify causes of latency in your sources and sinks.
  • Understand pipeline errors.

Monitoring interface components

The monitoring interface contains the following visualizers and charts:

Project monitoring dashboard
A dashboard that monitors your Dataflow jobs at the project level.
Jobs list
A list of all running Dataflow jobs and all jobs run within the last 30 days, along with their status, region, elapsed time, and other information.
Job graph
A graphical representation of a pipeline. The job graph also provides a job summary, a job log, and information about each step in the pipeline.
Execution details
Shows the execution stages of a job, data freshness for streaming jobs, and worker progress for batch jobs.
Job metrics
Charts that display metrics over the duration of a job.
Estimated cost
The estimated cost of your Dataflow job, based on resource usage metrics.
Recommendations
Recommendations for improving job performance, reducing cost, and troubleshooting errors.
Autoscaling
A set of charts that help you to understand the autoscaling behavior of streaming jobs.
Pipeline logs
Logs emitted by your pipeline and by the Dataflow service.
Data sampling
A tool that lets you observe sampled data at each step of a pipeline.

What's next