Welcome > Artificial Intelligence > ML > Reading Notes on Dr. Mi Zhang's Publications > Idea of improving FL using NAS

Idea of improving FL using NAS

References:

reference

NAS for Better Federated Learning Models in Different FL Topologies?

(Another direction might also be enticing: using FL to improve NAS.)

In Mi Zhang’s FL benchmark paper ArXiv’20: FedML: A Research Library and Benchmark for Federated Learning, I found several important features that an FL can have, such as computing paradigms, topology, exchanged information, and training procedures.

It also mentions FedNAS as one category of FL algorithm (in Section 4.1). However, based on a survey paper about Fed + NAS, I am thinking there can be many different ways to improve FL using NAS. Recalling Song Han’s Once for All paper, what he do is to adopt NAS to find the best model for machine constraints. Similarly, we probably can use NAS to find the best model for FL constraints, and these constraints seems more complicated than just the machine resource constraints – FL has multiple constraints (or features?), such as computing paradigms, topology, exchanged information and training procedures.

So tracking down this direction, let’s see whether there are anything has been done in improving FL models using NAS.

arXiv’2020, from Cisco, Direct Federated Neural Architecture Search
- Aim to provide NAS to meet the heterogeneous data and resource distribution of FL.
- Find read-to-deploy network to any type of client hardware by searching on a super network (basd on DSNAS, similar to once for all?)
arXiv’2020, from USC, Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
- A collaborated Neural Arch search on edge servers;
- Put NAS at each client and get the weight $w$ and arch $\alpha$
- Then merge them in the coordinator as normally done in FL.
arXiv’2020, from Peking, Federated Neural Architecture Search
- Wrong paper? It seems they are improving NAS using FL, instead of improving FL using NAS, right?

New ideas:

Where do we search? How about at the server level instead of at the client level?
- Need a new training method without data (since data is only available at client)
- Can we use only the trained $w$ as the data for training in the NAS?
What do we search for? For optimization goals (or search objectives) of the NAS, we might have more choices than just the accuracy or device resource constraints:
- Can we search by balancing more factors, such as computing paradigms, topology, or communication cost, or training procedure?

If you could revise
the fundmental principles of
computer system design
to improve security...

... what would you change?

Idea of improving FL using NAS

More