We present ART-V an efficient framework for auto-regressive video generation with diffusion models. Unlike existing methods that generate entire videos in one-shot ART-V generates a single frame at a time conditioned on the previous ones. The framework offers three distinct advantages. First it only learns simple continual motions between adjacent frames therefore avoiding modeling complex long-range motions that require huge training data. Second it preserves the high-fidelity generation ability of the pre-trained image diffusion models by making only minimal network modifications. Third it can generate arbitrarily long videos conditioned on a variety of prompts such as text image or their combinations making it highly versatile and flexible. To combat the common drifting issue in AR models we propose masked diffusion model which implicitly learns which information can be drawn from reference images rather than network predictions in order to reduce the risk of generating inconsistent appearances that cause drifting. Moreover we further enhance generation coherence by conditioning it on the initial frame which typically contains minimal noise. This is particularly useful for long video generation. When trained for only two weeks on four GPUs ART-V already can generate videos with natural motions rich details and a high level of aesthetic quality. Besides it enables various appealing applications eg composing a long video from multiple text prompts.