An AR client device generates and uses a background model to identify portions of images that depict the sky. A background model is a model that represents where the sky is visible for the client device. To identify a sky background portion of an image, a client device can map an image onto the background model and thereby determine which portion of the image represents the sky. The client device can use the identified sky background portion to augment the image to include AR content in the sky. To generate the background model, the client device applies a background detection model to a set of images to generate background probability images. The background probability images are mapped onto a background model using orientation data captured by the client device to update the background model based on the background probability image.