Scenario A. Spatial baseline. The first scenario we present is the
baseline. In this scenario, each client owns 1200 samples from each
of the 5 domains, so the situation is totally IID, which may seem
unrealistic. As illustrated in
7(a)
and
7(b)
, in this scenario both FedAvg and FedProx perform excellently well, reaching an accuracy of 90%.
Scenario B. Input space heterogeneity with homogeneous test. In this scenario, the dataset of each client only has data from 1 domain,
and the 15 clients selected for testing are randomly selected. In
7(c)
and
7(d)
we can see that the global model from FedAvg and FedProx achieve a performance similar to the baseline scenario (A), although
the situation presented here is more realistic, since each client presents
its own kind of data. Both methods take a couple more training rounds
to converge, but the difference is not significant, meaning that both
FedAvg and FedProx can deal with this kind of non-IID data.
Scenario C. Input space heterogeneity with biased test. This
scenario is very similar to Scenario 𝐵, with the only difference that all
of the clients that have MNIST data samples are selected for testing, and
for that reason, there are no MNIST data samples in the training clients
datasets. This emulates the situation presented in the Domain Adaptation techniques (See Section
3.3.1
), i.e., a new domain appears in testing
time. In
7(e)
and
7(f)
we can see that the accuracy drops significantly
in this scenario for both FedAvg and FedProx, which is very reasonable
because most of the data used for testing the model belongs to a domain
what was unseen during training. At the same time, FedProx performs
significantly better than FedAvg (68% and 48% respectively), because
it is prepared to face this kind of heterogeneity.
