Federated Learning Averages Model Updates, But Heterogeneous Data Can Break It

Most machine learning assumes all data is centralized, yet real data often lives on phones, in hospitals, banks, and factories where moving it is costly or restricted. Federated learning instead sends the model to the data: each client trains locally and returns only an update, which a server combines into a new global model. The standard method, Federated Averaging (FedAvg), forms this global model as a weighted average of client models, with weights based on dataset size. The approach generally works because every client starts each round from the same model, keeping local models close enough that their average stays sensible, but this breaks down when clients train for too long. The main difficulty is non-IID data, since different clients see different distributions, which makes local updates less representative of the whole population.