Video Models: Why Motion Is So Much Harder
Temporal consistency, world models, and what current video generators actually learned.
A video model can produce single frames that look as good as a still image model's. Ask it to keep the same face across 48 frames, though, and eyes can flicker, hair can warp and the background can slide apart. Motion is a constraint on every frame at once: each pixel has to agree with the ones before and after it.
Your photo app can generate a still of a sunset with ease. A two-second loop of the same scene is a different story: clouds wobble, the sun pulses, the ocean texture dances. This article explains why motion is so much harder than appearance, how today's video generators are built to handle it, where they still fail, and which systems lead as of October 2026.
Why is temporal consistency harder than getting one frame right?
Think of drawing a flipbook of a bouncing ball. One ball is a shaded circle, easy. Now every page must show the same ball, obey gravity and land in the right spot, and any mismatch between pages jumps out at the eye.
Video generators face the same challenge. A frame of a person is an arrangement of colors and textures. Motion adds three demands: identity across time (the blob in frame 5 is the same nose as in frame 4), physical plausibility, and causal order (later frames follow from earlier ones). If the model does not keep track of which object is which from frame to frame, the face morphs. That is the root of temporal flicker: features that drift in look or position because nothing tied them together through time.
Physics makes it harder. Ask for a ball thrown in the air and a video model usually draws a believable arc, because such arcs fill its training videos. It is matching patterns, not running physics. The training loss only asks for videos that look like the training data, so physics is a side effect rather than a rule the model follows, and the gap shows when a scene departs from what the model has seen.
Ambiguity is the other wall. Given one frame of a person standing still, the next frame could show them turning, walking or sitting. A model trained to predict the next frame with a pixel by pixel loss such as mean squared error averages over all those futures and produces a blurry smudge instead of one sharp motion. Mathieu, Couprie and LeCun documented this blur and proposed adversarial and image gradient losses to fight it (2015, Deep multi-scale video prediction beyond mean square error). Feed blurry frames back in as input and the blur tends to compound. That is why modern generators use diffusion, which learns a distribution over possible videos and samples one of them, instead of predicting a single average.
How do modern video generators encode and process motion?
Your app decides to add a clip generator. It cannot work directly on pixels: a few seconds of HD video is dozens of frames of millions of numbers each. Instead it compresses the clip with a video autoencoder, Part 3's learned compressor extended to time. The open models agree on the recipe. Wan's autoencoder shrinks height and width by 8 and time by 4, with 16 numbers per latent position, and compresses the first frame only in space, so a still image is just a one-frame video (Team Wan 2025, Wan: Open and Advanced Large-Scale Video Generative Models). HunyuanVideo and CogVideoX use the same 8 × 8 × 4 compression.
Two generations of architecture sit on top of that, and it helps to keep them apart.
The older approach: an image model plus temporal layers
The first wave (2022 to 2023) started from a trained image model and added layers that look across time. Make-A-Video split a full spatiotemporal U-Net into separate space and time parts (Singer et al. 2022, Make-A-Video: Text-to-Video Generation without Text-Video Data). Video LDM turned Stable Diffusion into a video model by adding temporal layers and training them on videos (Blattmann et al. 2023, Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models). AnimateDiff trained a plug-in motion module that can be inserted into personalized image models (Guo et al. 2023, AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning). This is what people mean by an "inflated" U-Net: the image layers still look within each frame, and temporal attention layers between them compare each spatial position across frames.
Splitting attention into a space part and a time part came from video understanding. TimeSformer, a video classifier, found that this "divided attention" worked best among the designs it compared (Bertasius et al. 2021, Is Space-Time Attention All You Need for Video Understanding?). Some generators make the temporal part adaptive. One 2026 method attends locally across frames when a clip has a lot of motion, to keep fast details, and globally when it has little, to keep the scene stable (Hong et al. 2026, Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion).
The current approach: one transformer attending across space and time
The current generation uses a diffusion transformer, the DiT from Part 3, over spacetime patches. OpenAI's Sora report described the pattern in February 2024: compress the video into a latent space, cut it into spacetime patches, and use those patches as transformer tokens (OpenAI 2024, Video generation models as world simulators). The report gave no sizes; it says model and implementation details are not included. The open models publish theirs. Wan's patches cover 2 × 2 latent positions in a single latent time step, so each token stands for a 16 × 16 pixel square across 4 frames.
Every token carries a position in space and time, and self-attention runs across all the tokens at once, so the collar in frame 3 can attend directly to the same collar in frame 12. Wan, HunyuanVideo and CogVideoX all use this full spatiotemporal attention, and HunyuanVideo's authors report that it beat divided attention in their tests (Kong et al. 2024, HunyuanVideo: A Systematic Framework For Large Video Generative Models). The price is compute: attention cost grows with the square of the number of tokens. That is why the autoencoder compresses so hard, and why LTX-Video pushes compression to 1:192 so its transformer can afford full attention (HaCohen et al. 2024, LTX-Video: Realtime Video Latent Diffusion).
Generation then runs the same loop as in Part 3, over the whole clip at once:
What actually breaks when motion goes wrong?
Your photo app tries to generate a slow-motion loop of a jumping dog. Each frame looks photorealistic, but the legs flicker between positions with no motion in between. That is identity drift: the model does not hold on to the leg as the same object, so it reinvents it frame by frame.
In the older design, identity drift points at the temporal layers. They link corresponding patches across frames, and their reach is a trade-off. If they mix too broadly, fast motion smears; if they look too locally, an object's identity cannot survive an occlusion or a big jump. That is the trade the motion-adaptive method above tries to manage. Full spacetime attention removes the hard limit on reach, but it does not guarantee that the model learned to use it well.
Another common failure: a clip shows a man holding a cup, and the cup slowly shrinks and disappears. The model treated the cup as a passing detail rather than a lasting object. One reason is that what is in the scene and how it moves end up tangled together inside the model, so a change meant for the motion can also alter an object's size or texture. The idea that video splits into content and motion goes back at least to MoCoGAN, which generated videos from a fixed content code and a changing motion code (Tulyakov et al. 2017, MoCoGAN: Decomposing Motion and Content for Video Generation).
Large systems are not immune. OpenAI's Sora report said the model could often, though not always, keep people and objects consistent even when they leave the frame, in clips up to a minute long. The same report listed glass that does not shatter correctly, food that does not change when bitten, incoherence in long clips and objects that appear from nowhere. Controlled tests point the same way. Kang et al. trained video models on simulated physics and found perfect generalization inside the training distribution but failure outside it: the models copied the closest training example instead of learning the rule, matching on color before size, velocity or shape (2024, How Far is Video Generation from World Model: A Physical Law Perspective). The Physics-IQ benchmark tested Sora, Runway, Pika, Lumiere, Stable Video Diffusion and VideoPoet and found their physical understanding severely limited and unrelated to how realistic the videos looked (Motamed et al. 2025, Do generative video models understand physical principles?).
Can sampling fix motion without retraining?
A trained video model holds more information about motion than its default sampling uses, and two lines of research show it.
FlowMo is a training-free guidance method. At each denoising step it measures the distance between the latents of consecutive frames, which gives a motion signal with the appearance taken out, then measures how much that signal varies over time for each patch, and nudges the sampler to reduce the variance. Its authors report better motion coherence across several text-to-video models without losing visual quality or prompt alignment (Shaulov et al. 2025, FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation).
Motion by Queries looked inside the attention layers. It found that the self-attention queries (Q, the vectors each position uses to decide what to attend to) carry the layout and motion of a clip but also the subject's identity, so copying them from a source video transfers the motion along with unwanted identity. Controlling when and how the queries are injected gave a zero-shot motion transfer method about ten times more efficient than earlier ones, and a training-free way to keep characters consistent across several shots (Atzmon et al. 2024, Motion by Queries: Identity-Motion Trade-offs in Text-to-Video Generation).
Why can guidance help at all? Each denoising step updates every frame together, but nothing in the sampler explicitly checks that the motion is smooth. Methods like FlowMo add that check during sampling, using the model's own predictions. That is also their limit: because they rely on what the model already represents, they cannot add motion knowledge the model never learned.
Which video models lead in late 2026?
OpenAI's Sora, the system that put spacetime patches in the headlines, is gone. Sora 2 launched on 30 September 2025 with synchronized dialogue and sound effects (Sora 2 is here). On 24 March 2026 OpenAI told developers it would remove the Sora 2 models and its Videos API, and its help article on the Sora discontinuation confirms that the Sora app and website closed on 26 April 2026. The API followed on 24 September 2026 with no replacement, as OpenAI's deprecations page records.
These are the leading systems as of October 2026, each from the maker's own pages:
| System | Maker | Latest version | Length per generation | Sound | Open weights |
|---|---|---|---|---|---|
| Veo | Veo 3.1 (October 2025) and Veo 3.1 Lite (March 2026) | 4, 6 or 8 s, extendable to 148 s | Yes | No | |
| Gemini Omni | Gemini Omni 1.1 Flash (August 2026) | Scenes extendable to 40 s | Yes | No | |
| Kling | Kuaishou | Kling 4.0 (early access from September 2026, wider rollout in October) | Up to 30 s | Yes | No |
| Seedance | ByteDance | Seedance 2.5 (July 2026) | Up to 30 s in one pass | Yes | No |
| Runway | Runway | Gen-4.5 (December 2025) | 2 to 10 s | Not stated | No |
| Wan | Alibaba | Wan2.2, the latest open weights (July 2025); Wan3.0 in beta on Alibaba Cloud (August 2026) | 5 s at 720p (Wan2.2 TI2V-5B); up to 30 s (Wan3.0) | Not in Wan2.2's text or image to video | Wan2.2 yes, Apache 2.0 |
| HunyuanVideo | Tencent | HunyuanVideo-1.5, 8.3B parameters (November 2025) | About 5 s by default | Separate Foley model | Yes, community license |
| LTX | Lightricks | LTX-2.5 (August 2026) | Varies by mode | Yes | Yes, community license |
Three things changed since 2024. Most leading models now generate sound together with the picture. Several generate up to 30 seconds in a single pass (Kling 4.0, Seedance 2.5), and others extend a clip step by step to well over a minute (Veo 3.1 reaches 148 seconds). And many models accept an image as the first frame (image to video) as well as a text prompt, which gives you far more control over what the clip shows.
Quick Reference
| Property | Value |
|---|---|
| Latent compression (open models) | 8× in height and width, 4× in time (Wan, HunyuanVideo, CogVideoX); LTX-Video 1:192 |
| Tokens (Wan) | Patches of 2 × 2 latent positions in one latent time step: 16 × 16 pixels across 4 frames |
| Attention in current models | Full attention across space and time in one diffusion transformer |
| Attention in older models | Image U-Net with added temporal layers (divided space and time) |
| Why next-frame MSE predictors blur | They average many possible futures |
| FlowMo guidance signal | Patch-wise variance over time of an appearance-debiased latent difference |
| Motion by Queries finding | Self-attention queries carry motion and identity together |
| Sora | Spacetime patches in a diffusion transformer (2024); no sizes published; discontinued in 2026 |
Frequently Asked Questions
Q: Can I just add 3D convolutions to an image generator and get smooth video? 3D convolutions only see a few neighboring frames at a time. They smooth short-range motion, but they cannot keep an object's identity across long gaps, which is why video models added temporal attention, and why current ones attend across all frames at once.
Q: Why do video models fail on prompts with many moving characters? Each character needs its own consistent motion, and nothing in training checks who does what. OpenAI's own Sora page listed as a weakness that animals or people can appear spontaneously, especially in scenes with many entities.
Q: Is training-free motion guidance ready for production use? It is research, tested on open models. Hosted video services do not expose their samplers, so you cannot apply it there; with an open model such as Wan you can try a published method like FlowMo yourself.
Q: Why do video models sometimes lose small objects like cups or hands? Compression is part of it. Each latent position stands for an 8 × 8 pixel block over 4 frames, so a small, fast object covers only a few tokens, and the model can treat it as texture rather than a lasting object.
Q: Could an explicit physics engine inside the model solve the problem? It is an open research direction. The physics tests above suggest that scale alone does not make a model discover physical laws, so explicit structure may help. The engineering is hard, though: a physics engine needs exact object states, which are hard to read from noisy latents.
Test yourself
Your photo app wants to generate a short clip of a vase falling from a table. The model produces a vase that slides sideways along the table instead of dropping. You use a hosted video model, so you cannot retrain it. What would you try, and why does this happen?
Answer: Start with the prompt: describe the motion and its physics explicitly, such as "the vase tips over the edge, falls straight down and shatters on the tile floor", and describe a camera that keeps the fall in frame. If the model supports image to video, give it a first frame with the vase already tipping past the edge, so the most likely continuation is a fall. Generate with several seeds, since each sample is one draw from the model's distribution, and keep the best. If you can run an open model instead, a published sampling method such as FlowMo can improve motion coherence without retraining. The sliding happens because the model matches patterns from its training clips, and vases resting on or moving along tables are probably far more common there than vases falling and breaking. Studies of these models find that they copy the closest familiar case instead of applying a physical rule like gravity.
Next: Document AI: How Models Read PDFs, Tables, and Receipts. It turns from making images to reading them: how models pull text, tables and fields out of scanned pages.
If you want a breakdown like this every week, how real systems actually work, what fails, and how to fix it, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Deep multi-scale video prediction beyond mean square error (Mathieu et al. 2015)
- Wan: Open and Advanced Large-Scale Video Generative Models (Team Wan 2025)
- HunyuanVideo: A Systematic Framework For Large Video Generative Models (Kong et al. 2024)
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (Yang et al. 2024)
- LTX-Video: Realtime Video Latent Diffusion (HaCohen et al. 2024)
- Make-A-Video: Text-to-Video Generation without Text-Video Data (Singer et al. 2022)
- Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (Blattmann et al. 2023)
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (Guo et al. 2023)
- Is Space-Time Attention All You Need for Video Understanding? (Bertasius et al. 2021, TimeSformer)
- Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion (Hong et al. 2026)
- Video generation models as world simulators (OpenAI, February 2024) and Sora (OpenAI)
- MoCoGAN: Decomposing Motion and Content for Video Generation (Tulyakov et al. 2017)
- How Far is Video Generation from World Model: A Physical Law Perspective (Kang et al. 2024)
- Do generative video models understand physical principles? (Motamed et al. 2025, Physics-IQ)
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (Shaulov et al. 2025)
- Motion by Queries: Identity-Motion Trade-offs in Text-to-Video Generation (Atzmon et al. 2024)
- Sora 2 is here, What to know about the Sora discontinuation and Deprecations (OpenAI)
- Vendor pages for the landscape table, read October 2026: Veo, Gemini Omni 1.1 Flash, Kling 4.0, Seedance 2.5, Runway Gen-4.5, Wan2.2, Wan3.0, HunyuanVideo-1.5, LTX-2.5
