There is provided a video encoder including a processing hardware, and a system memory storing a software code. The processing hardware is configured to execute the software code to receive an uncompressed video content and a motion compensated video content corresponding to the uncompressed video content, transform the uncompressed video content to a first latent space representation of the uncompressed video content, transform the motion compensated video content to a second latent space representation of the uncompressed video content, determine, using the first latent space representation and the second latent space representation, a latent space residual, and generate, using the latent space residual, a compressed video content corresponding to the uncompressed video content.