Methods and systems are described herein for protecting client data while training machine learning models. The system may transmit, to client devices, simple models to be trained on a respective client device to generate predictions based on a respective subset of respective client data of the respective client device. The system may receive the trained simple models from the client devices. The system may input, into an ensemble model including the simple models, an unlabeled synthetic dataset. This may cause the ensemble model to aggregate a set of predictions generated by each simple model to generate labels for the unlabeled synthetic dataset. The system may then input, into a new model, the unlabeled synthetic dataset and the labels to train the new model to predict the labels for the unlabeled synthetic dataset.