In this work, we study to release the potential of massive heterogeneous weak computing power to collaboratively train large-scale models on dispersed datasets. In order to improve both efficiency and accuracy in resource-adaptive collaborative learning, we take the first step to consider the \textit{ unstructured pruning}, \textit{ varying submodel architectures}, \textit{ knowledge loss}, and \textit{ straggler} challenges simultaneously. We propose a novel semi-asynchronous collaborative training framework, namely ${ Co\text{ -}S}^2{ P}$, with data distribution-aware structured pruning and cross-block knowledge transfer mechanism to address the above concerns. Furthermore, we provide theoretical proof that ${ Co\text{ -}S}^2{ P}$ can achieve asymptotic optimal convergence rate of $O(1/\sqrt{ N^*EQ})$. Finally, we conduct extensive experiments on two types of tasks with a real-world hardware testbed including diverse IoT devices.The experimental results demonstrate that $Co\text{ -}S^2P$ improves accuracy by up to 8.8\% and resource utilization by up to 1.2$\times$ compared to state-of-the-art methods, while reducing memory consumption by approximately 22\% and training time by about 24\% on all resource-limited devices.
Categories: cs.DC cs.LG