p7cs.DC

Category

cs.DC

218 papers

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, Xiaoqiang Zheng

1603.04467

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica

2309.06180

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He

1706.02677

The Hidden Vulnerability of Distributed Learning in Byzantium

El Mahdi El Mhamdi, Rachid Guerraoui, Sébastien Rouault

1802.07927

GSPMD: General and Scalable Parallelization for ML Computation Graphs

Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, Zhifeng Chen

2105.04663

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, Shen Li

2304.11277

Regularized Evolution for Image Classifier Architecture Search

Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le

1802.01548

Sparse GPU Kernels for Deep Learning

Trevor Gale, Matei Zaharia, Cliff Young, Erich Elsen

2006.10901

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He

2207.00032

Summarizing CPU and GPU Design Trends with Product Data

Yifan Sun, Nicolas Bohm Agostini, Shi Dong, David Kaeli

1911.11313

A Little Is Enough: Circumventing Defenses For Distributed Learning

Moran Baruch, Gilad Baruch, Yoav Goldberg

1902.06156

Towards Federated Learning at Scale: System Design

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, Jason Roselander

1902.01046

Mesh-TensorFlow: Deep Learning for Supercomputers

Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake Hechtman

1811.02084

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia

2104.04473

signSGD: Compressed Optimisation for Non-Convex Problems

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, Anima Anandkumar

1802.04434

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

Dong Yin, Yudong Chen, Kannan Ramchandran, Peter Bartlett

1803.01498

Mitigating Sybils in Federated Learning Poisoning

Clement Fung, Chris J. M. Yoon, Ivan Beschastnikh

1808.04866

Beyond Data and Model Parallelism for Deep Neural Networks

Zhihao Jia, Matei Zaharia, Alex Aiken

1807.05358

Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

Sebastian Caldas, Jakub Konečny, H. Brendan McMahan, Ameet Talwalkar

1812.07210

Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

Zhihao Jia, Sina Lin, Charles R. Qi, Alex Aiken

1802.04924

Large-Scale Evolution of Image Classifiers

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, Alex Kurakin

1703.01041

Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Takuya Akiba, Shuji Suzuki, Keisuke Fukuda

1711.04325

Communication-Computation Efficient Gradient Coding

Min Ye, Emmanuel Abbe

1802.03475

Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

Hao Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, Eric P. Xing

1706.03292

Fast Convolutional Nets With fbfft: A GPU Performance Evaluation

Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, Yann LeCun

1412.7580

Don't Decay the Learning Rate, Increase the Batch Size

Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le

1711.00489

Using Optimization to Obtain a Width-Independent, Parallel, Simpler, and Faster Positive SDP Solver

Zeyuan Allen-Zhu, Yin Tat Lee, Lorenzo Orecchia

1507.02259

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He

2201.05596

When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions

Weiming Zhuang, Chen Chen, Jingtao Li, Chaochao Chen, Yaochu Jin, Lingjuan Lyu

2306.15546

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, Ang Li

2409.05976

Pathways: Asynchronous Distributed Dataflow for ML

Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Dan Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, Brennan Saeta, Parker Schuh, Ryan Sepassi, Laurent El Shafey, Chandramohan A. Thekkath, Yonghui Wu

2203.12533

Randomized Distributed Mean Estimation: Accuracy vs Communication

Jakub Konečný, Peter Richtárik

1611.07555

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He

2104.07857

Using Optimization to Solve Positive LPs Faster in Parallel

Zeyuan Allen-Zhu, Lorenzo Orecchia

1407.1925

Implementing Randomized Matrix Algorithms in Parallel and Distributed Environments

Jiyan Yang, Xiangrui Meng, Michael W. Mahoney

1502.03032

Faster and Simpler Width-Independent Parallel Algorithms for Positive Semidefinite Programming

Richard Peng, Kanat Tangwongsan, Peng Zhang

1201.5135

Distributed Coordinate Descent Method for Learning with Big Data

Peter Richtárik, Martin Takáč

1310.2059

Principal Component Analysis and Higher Correlations for Distributed Data

Ravindran Kannan, Santosh Vempala, David Woodruff

1304.3162

ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing

Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, Burkhard Rost

2007.06225

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Yujun Lin, Song Han, Huizi Mao, Yu Wang, William J. Dally

1712.01887

TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li

1705.07878

A Distributed Newton Method for Large Scale Consensus Optimization

Rasul Tutunov, Haitham Bou Ammar, Ali Jadbabaie

1606.06593

AI and Memory Wall

Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer

2403.14123

A Decentralized Second-Order Method with Exact Linear Convergence Rate for Consensus Optimization

Aryan Mokhtari, Wei Shi, Qing Ling, Alejandro Ribeiro

1602.00596

Robust Communication-Optimal Distributed Clustering Algorithms

Pranjal Awasthi, Ainesh Bakshi, Maria-Florina Balcan, Colin White, David Woodruff

1703.00830

Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration

Congliang Chen, Li Shen, Fangyu Zou, Wei Liu

2101.05471

TensorFlow: A system for large-scale machine learning

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, Xiaoqiang Zheng

1605.08695

Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

Jianyu Wang, Gauri Joshi

1808.07576

Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning

Hao Yu, Sen Yang, Shenghuo Zhu

1807.06629

Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD

Jianyu Wang, Gauri Joshi

1810.08313

Local SGD Converges Fast and Communicates Little

Sebastian U. Stich

1805.09767

Sparsified SGD with Memory

Sebastian U. Stich, Jean-Baptiste Cordonnier, Martin Jaggi

1809.07599

On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization

Fan Zhou, Guojing Cong

1708.01012

Randomized Composable Core-sets for Distributed Submodular Maximization

Vahab Mirrokni, Morteza Zadimoghaddam

1506.06715

Accelerated, Parallel and Proximal Coordinate Descent

Olivier Fercoq, Peter Richtárik

1312.5799

Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization

Angelia Nedić, Alex Olshevsky, Michael G. Rabbat

1709.08765

Optimal Distributed Online Prediction using Mini-Batches

Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao

1012.1367

D$^2$: Decentralized Training over Decentralized Data

Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, Ji Liu

1803.07068

A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates

Zhi Li, Wei Shi, Ming Yan

1704.07807

On Optimal Probabilities in Stochastic Coordinate Descent Methods

Peter Richtárik, Martin Takáč

1310.3438

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, Ji Liu

1705.09056

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang

1512.01274

A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

Shun Zheng, Jialei Wang, Fen Xia, Wei Xu, Tong Zhang

1604.03763

Perturbed Iterate Analysis for Asynchronous Stochastic Optimization

Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, Michael I. Jordan

1507.06970

TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, Ion Stoica

2102.07988

Reproducing GW150914: the first observation of gravitational waves from a binary black hole merger

Duncan A. Brown, Karan Vahi, Michela Taufer, Von Welch, Ewa Deelman

2010.07244

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, Wei Lin

2007.01045

TF.Learn: TensorFlow's High-level Module for Distributed Machine Learning

Yuan Tang

1612.04251

NSML: Meet the MLaaS platform with a real-world case study

Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, Nako Sung, Jung-Woo Ha

1810.09957

MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

Trevor Gale, Deepak Narayanan, Cliff Young, Matei Zaharia

2211.15841

Distributed Deep Learning Using Synchronous Stochastic Gradient Descent

Dipankar Das, Sasikanth Avancha, Dheevatsa Mudigere, Karthikeyan Vaidynathan, Srinivas Sridharan, Dhiraj Kalamkar, Bharat Kaul, Pradeep Dubey

1602.06709

Approximate Gradient Coding via Sparse Random Graphs

Zachary Charles, Dimitris Papailiopoulos, Jordan Ellenberg

1711.06771

Communication Compression for Decentralized Training

Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, Ji Liu

1803.06443

Near-Optimal Straggler Mitigation for Distributed Gradient Methods

Songze Li, Seyed Mohammadreza Mousavi Kalan, A. Salman Avestimehr, Mahdi Soltanolkotabi

1710.09990

Data Access for LIGO on the OSG

Derek Weitzel, Brian Bockelman, Duncan A. Brown, Peter Couvares, Frank Würthwein, Edgar Fajardo Hernandez

1705.06202

Papaya: Practical, Private, and Scalable Federated Learning

Dzmitry Huba, John Nguyen, Kshitiz Malik, Ruiyu Zhu, Mike Rabbat, Ashkan Yousefpour, Carole-Jean Wu, Hongyuan Zhan, Pavel Ustinov, Harish Srinivas, Kaikai Wang, Anthony Shoumikhin, Jesik Min, Mani Malek

2111.04877

ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates

Zhimin Peng, Yangyang Xu, Ming Yan, Wotao Yin

1506.02396

Quantized Adam with Error Feedback

Congliang Chen, Li Shen, Haozhi Huang, Wei Liu

2004.14180

Distributed Nonconvex Multiagent Optimization Over Time-Varying Networks

Ying Sun, Gesualdo Scutari, Daniel Palomar

1607.00249

RecShard: Statistical Feature-Based Memory Optimization for Industry-Scale Neural Recommendation

Geet Sethi, Bilge Acun, Niket Agarwal, Christos Kozyrakis, Caroline Trippel, Carole-Jean Wu

2201.10095

Coded Fourier Transform

Qian Yu, Mohammad Ali Maddah-Ali, A. Salman Avestimehr

1710.06471

Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training

Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, Shibo Wang

2004.13336

ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning

Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, Leshem Choshen

2212.01378

PipeDream: Fast and Efficient Pipeline Parallel DNN Training

Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, Phil Gibbons

1806.03377

Efficient-Adam: Communication-Efficient Distributed Adam

Congliang Chen, Li Shen, Wei Liu, Zhi-Quan Luo

2205.14473

ZeRO-Offload: Democratizing Billion-Scale Model Training

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He

2101.06840

Memory-Efficient Pipeline-Parallel DNN Training

Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia

2006.09503

FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads

Zhen Zheng, Pengzhan Zhao, Guoping Long, Feiwen Zhu, Kai Zhu, Wenyi Zhao, Lansong Diao, Jun Yang, Wei Lin

2009.10924

Straggler Mitigation in Distributed Optimization Through Data Encoding

Can Karakus, Yifan Sun, Suhas Diggavi, Wotao Yin

1711.04969

Toward Communication Efficient Adaptive Gradient Method

Xiangyi Chen, Xiaoyun Li, Ping Li

2109.05109

Adaptive Federated Optimization

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, H. Brendan McMahan

2003.00295

Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, Ping Li

2003.05622

ATOMO: Communication-efficient Learning via Atomic Sparsification

Hongyi Wang, Scott Sievert, Zachary Charles, Shengchao Liu, Stephen Wright, Dimitris Papailiopoulos

1806.04090

ClusterCluster: Parallel Markov Chain Monte Carlo for Dirichlet Process Mixtures

Dan Lovell, Jonathan Malmaud, Ryan P. Adams, Vikash K. Mansinghka

1304.2302

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He

2309.14509

Distributed optimization over time-varying directed graphs

Angelia Nedic, Alex Olshevsky

1303.2289

Massively Parallel Methods for Deep Reinforcement Learning

Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, David Silver

1507.04296

Coding for Distributed Fog Computing

Songze Li, Mohammad Ali Maddah-Ali, A. Salman Avestimehr

1702.06082

A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale

Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, Michael Rabbat

2309.06497

KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics

Evan R. Sparks, Shivaram Venkataraman, Tomer Kaftan, Michael J. Franklin, Benjamin Recht

1610.09451

Graph Expansion and Communication Costs of Fast Matrix Multiplication

Grey Ballard, James Demmel, Olga Holtz, Oded Schwartz

1109.1693

Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study

Suyog Gupta, Wei Zhang, Fei Wang

1509.04210

Online Tensor Methods for Learning Latent Variable Models

Furong Huang, U. N. Niranjan, Mohammad Umar Hakeem, Animashree Anandkumar

1309.0787

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi, Xiaowen Chu

1807.11205

torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

Chiheon Kim, Heungsub Lee, Myungryong Jeong, Woonhyuk Baek, Boogeon Yoon, Ildoo Kim, Sungbin Lim, Sungwoong Kim

2004.09910

Speeding Up Distributed Machine Learning Using Codes

Kangwook Lee, Maximilian Lam, Ramtin Pedarsani, Dimitris Papailiopoulos, Kannan Ramchandran

1512.02673

MediaPipe: A Framework for Building Perception Pipelines

Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, Matthias Grundmann

1906.08172

Accelerated Methods for Deep Reinforcement Learning

Adam Stooke, Pieter Abbeel

1803.02811

CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency

Walid A. Hanafy, Qianlin Liang, Noman Bashir, David Irwin, Prashant Shenoy

2302.08681

Stochastic Gradient Push for Distributed Deep Learning

Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, Michael Rabbat

1811.10792

Improving Distributed Gradient Descent Using Reed-Solomon Codes

Wael Halbawi, Navid Azizan-Ruhi, Fariborz Salehi, Babak Hassibi

1706.05436

Local algorithms for independent sets are half-optimal

Mustazee Rahman, Balint Virag

1402.0485

Understanding Capacity-Driven Scale-Out Neural Recommendation Inference

Michael Lui, Yavuz Yetim, Özgür Özkan, Zhuoran Zhao, Shin-Yeh Tsai, Carole-Jean Wu, Mark Hempstead

2011.02084

Asynchronous Federated Learning on Heterogeneous Devices: A Survey

Chenhao Xu, Youyang Qu, Yong Xiang, Longxiang Gao

2109.04269

On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent

Noah Golmant, Nikita Vemuri, Zhewei Yao, Vladimir Feinberg, Amir Gholami, Kai Rothauge, Michael W. Mahoney, Joseph Gonzalez

1811.12941

Image Classification at Supercomputer Scale

Chris Ying, Sameer Kumar, Dehao Chen, Tao Wang, Youlong Cheng

1811.06992

When the Curious Abandon Honesty: Federated Learning Is Not Private

Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, Nicolas Papernot

2112.02918

COLA: Decentralized Linear Learning

Lie He, An Bian, Martin Jaggi

1808.04883

IBM Federated Learning: an Enterprise Framework White Paper V0.1

Heiko Ludwig, Nathalie Baracaldo, Gegi Thomas, Yi Zhou, Ali Anwar, Shashank Rajamoni, Yuya Ong, Jayaram Radhakrishnan, Ashish Verma, Mathieu Sinn, Mark Purcell, Ambrish Rawat, Tran Minh, Naoise Holohan, Supriyo Chakraborty, Shalisha Whitherspoon, Dean Steuer, Laura Wynter, Hifaz Hassan, Sean Laguna, Mikhail Yurochkin, Mayank Agarwal, Ebube Chuba, Annie Abay

2007.10987

Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback

Shuai Zheng, Ziyue Huang, James T. Kwok

1905.10936

Decentralized Deep Learning with Arbitrary Communication Compression

Anastasia Koloskova, Tao Lin, Sebastian U. Stich, Martin Jaggi

1907.09356

Dual-Free Stochastic Decentralized Optimization with Variance Reduction

Hadrien Hendrikx, Francis Bach, Laurent Massoulié

2006.14384

Coordinate Friendly Structures, Algorithms and Applications

Zhimin Peng, Tianyu Wu, Yangyang Xu, Ming Yan, Wotao Yin

1601.00863

Coded Computation over Heterogeneous Clusters

Amirhossein Reisizadeh, Saurav Prakash, Ramtin Pedarsani, Amir Salman Avestimehr

1701.05973

PyTorch-BigGraph: A Large-scale Graph Embedding System

Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, Alex Peysakhovich

1903.12287

Gradient Coding

Rashish Tandon, Qi Lei, Alexandros G. Dimakis, Nikos Karampatziakis

1612.03301

NSML: A Machine Learning Platform That Enables You to Focus on Your Models

Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, Sunghun Kim

1712.05902

Parallel Tensor Compression for Large-Scale Scientific Data

Woody Austin, Grey Ballard, Tamara G. Kolda

1510.06689

Distributed Optimization Based on Gradient-tracking Revisited: Enhancing Convergence Rate via Surrogation

Ying Sun, Amir Daneshmand, Gesualdo Scutari

1905.02637

AutoDNNchip: An Automated DNN Chip Predictor and Builder for Both FPGAs and ASICs

Pengfei Xu, Xiaofan Zhang, Cong Hao, Yang Zhao, Yongan Zhang, Yue Wang, Chaojian Li, Zetong Guan, Deming Chen, Yingyan Lin

2001.03535

Lagrange Coded Computing: Optimal Design for Resiliency, Security and Privacy

Qian Yu, Songze Li, Netanel Raviv, Seyed Mohammadreza Mousavi Kalan, Mahdi Soltanolkotabi, Salman Avestimehr

1806.00939

The War of the Efficiencies: Understanding the Tension between Carbon and Energy Optimization

Walid A. Hanafy, Roozbeh Bostandoost, Noman Bashir, David Irwin, Mohammad Hajiesmaili, Prashant Shenoy

2306.16948

A Fundamental Tradeoff between Computation and Communication in Distributed Computing

Songze Li, Mohammad Ali Maddah-Ali, Qian Yu, A. Salman Avestimehr

1604.07086

Strong Scaling of Matrix Multiplication Algorithms and Memory-Independent Communication Lower Bounds

Grey Ballard, James Demmel, Olga Holtz, Benjamin Lipshitz, Oded Schwartz

1202.3177

Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems

Maxim Naumov, John Kim, Dheevatsa Mudigere, Srinivas Sridharan, Xiaodong Wang, Whitney Zhao, Serhat Yilmaz, Changkyu Kim, Hector Yuen, Mustafa Ozdal, Krishnakumar Nair, Isabel Gao, Bor-Yiing Su, Jiyan Yang, Mikhail Smelyanskiy

2003.09518

Fundamental Limits of Coded Linear Transform

Sinong Wang, Jiashang Liu, Ness Shroff, Pengyu Yang

1804.09791

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Shaohuai Shi, Qiang Wang, Xiaowen Chu

1711.05979

Block-Diagonal and LT Codes for Distributed Computing With Straggling Servers

Albin Severinson, Alexandre Graell i Amat, Eirik Rosnes

1712.08230

Stochastic Controlled Averaging for Federated Learning with Communication Compression

Xinmeng Huang, Ping Li, Xiaoyun Li

2308.08165

Distributed Low-rank Subspace Segmentation

Ameet Talwalkar, Lester Mackey, Yadong Mu, Shih-Fu Chang, Michael I. Jordan

1304.5583

Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols

Nicolas Loizou, Peter Richtárik

1905.08645

Robust and Scalable Bayes via a Median of Subset Posterior Measures

Stanislav Minsker, Sanvesh Srivastava, Lizhen Lin, David B. Dunson

1403.2660

Acceleration of the shiftable O(1) algorithm for bilateral filtering and non-local means

Kunal N. Chaudhury

1203.5128

Parallel Algorithms for Constrained Tensor Factorization via the Alternating Direction Method of Multipliers

Athanasios P. Liavas, Nicholas D. Sidiropoulos

1409.2383

Efficient Straggler Replication in Large-scale Parallel Computing

Da Wang, Gauri Joshi, Gregory Wornell

1503.03128

A Unified Coded Deep Neural Network Training Strategy Based on Generalized PolyDot Codes for Matrix Multiplication

Sanghamitra Dutta, Ziqian Bai, Haewon Jeong, Tze Meng Low, Pulkit Grover

1811.10751

Scheduling and Aggregation Design for Asynchronous Federated Learning over Wireless Networks

Chung-Hsuan Hu, Zheng Chen, Erik G. Larsson

2212.07356

Hybrid static/dynamic scheduling for already optimized dense matrix factorization

Simplice Donfack, Laura Grigori, William D. Gropp, Vivek Kale

1110.2677

On Efficient Training of Large-Scale Deep Learning Models: A Literature Review

Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, Dacheng Tao

2304.03589

Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H. Vincent Poor

2007.07481

Splitwise: Efficient generative LLM inference using phase splitting

Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini

2311.18677

Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training

Liang Luo, Jacob Nelson, Luis Ceze, Amar Phanishayee, Arvind Krishnamurthy

1805.07891

RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing

Liu Ke, Udit Gupta, Carole-Jean Wu, Benjamin Youngjae Cho, Mark Hempstead, Brandon Reagen, Xuan Zhang, David Brooks, Vikas Chandra, Utku Diril, Amin Firoozshahian, Kim Hazelwood, Bill Jia, Hsien-Hsin S. Lee, Meng Li, Bert Maher, Dheevatsa Mudigere, Maxim Naumov, Martin Schatz, Mikhail Smelyanskiy, Xiaodong Wang

1912.12953

Pangolin: An Efficient and Flexible Graph Pattern Mining System on CPU and GPU

Xuhao Chen, Roshan Dathathri, Gurbinder Gill, Keshav Pingali

1911.06969

The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory

Dan Alistarh, Christopher De Sa, Nikola Konstantinov

1803.08841

A Unified Coding Framework for Distributed Computing with Straggling Servers

Songze Li, Mohammad Ali Maddah-Ali, A. Salman Avestimehr

1609.01690

Distributed Strongly Convex Optimization

Konstantinos I. Tsianos, Michael G. Rabbat

1207.3031

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica

2311.03285

Federated Learning With Quantized Global Model Updates

Mohammad Mohammadi Amiri, Deniz Gunduz, Sanjeev R. Kulkarni, H. Vincent Poor

2006.10672

Communication-Optimal Parallel Algorithm for Strassen's Matrix Multiplication

Grey Ballard, James Demmel, Olga Holtz, Benjamin Lipshitz, Oded Schwartz

1202.3173

Ray: A Distributed Framework for Emerging AI Applications

Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, Ion Stoica

1712.05889

Asynchronous Federated Optimization

Cong Xie, Sanmi Koyejo, Indranil Gupta

1903.03934

Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations

Ranggi Hwang, Taehun Kim, Youngeun Kwon, Minsoo Rhu

2005.05968

Computing Graph Neural Networks: A Survey from Algorithms to Accelerators

Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alonso, Eduard Alarcón

2010.00130

BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, Eytan Bakshy

1910.06403

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh

1910.06378

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning

Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, Dimitris Papailiopoulos

2007.05084

On Large-Cohort Training for Federated Learning

Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, Virginia Smith

2106.07820

Byzantine-Tolerant Machine Learning

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, Julien Stainer

1703.02757

AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning

Qijing Huang, Ameer Haj-Ali, William Moses, John Xiang, Ion Stoica, Krste Asanovic, John Wawrzynek

2003.00671

Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More

Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao, Yong Wu, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava

2103.10891

The Architectural Implications of Facebook's DNN-based Personalized Recommendation

Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Bill Jia, Hsien-Hsin S. Lee, Andrey Malevich, Dheevatsa Mudigere, Mikhail Smelyanskiy, Liang Xiong, Xuan Zhang

1906.03109

Information Theoretic Limits of Data Shuffling for Distributed Learning

Mohamed Attia, Ravi Tandon

1609.05181

PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units

Yujeong Choi, Minsoo Rhu

1909.04548

When Do Redundant Requests Reduce Latency ?

Nihar B. Shah, Kangwook Lee, Kannan Ramchandran

1311.2851

Verified Instruction-Level Energy Consumption Measurement for NVIDIA GPUs

Yehia Arafa, Ammar ElWazir, Abdelrahman ElKanishy, Youssef Aly, Ayatelrahman Elsayed, Abdel-Hameed Badawy, Gopinath Chennupati, Stephan Eidenbenz, Nandakishore Santhi

2002.07795

Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, Zhihao Jia

2312.15234

Salvaging Federated Learning by Local Adaptation

Tao Yu, Eugene Bagdasaryan, Vitaly Shmatikov

2002.04758

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

Zheng Zhang, Donglin Yang, Yaqi Xia, Liang Ding, Dacheng Tao, Xiaobo Zhou, Dazhao Cheng

2506.22175

Tune: A Research Platform for Distributed Model Selection and Training

Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, Ion Stoica

1807.05118

Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, Leandros Tassiulas

2112.11256

CYCLADES: Conflict-free Asynchronous Machine Learning

Xinghao Pan, Maximilian Lam, Stephen Tu, Dimitris Papailiopoulos, Ce Zhang, Michael I. Jordan, Kannan Ramchandran, Chris Re, Benjamin Recht

1605.09721

Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning

Haibo Yang, Minghong Fang, Jia Liu

2101.11203

Distributed Nesterov gradient methods over arbitrary graphs

Ran Xin, Dusan Jakovetic, Usman A. Khan

1901.06995

Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning

Zachary Charles, Jakub Konečný

2103.05032

Parallel resampling in the particle filter

Lawrence M. Murray, Anthony Lee, Pierre E. Jacob

1301.4019

An Application of Storage-Optimal MatDot Codes for Coded Matrix Multiplication: Fast k-Nearest Neighbors Estimation

Utsav Sheth, Sanghamitra Dutta, Malhar Chaudhari, Haewon Jeong, Yaoqing Yang, Jukka Kohonen, Teemu Roos, Pulkit Grover

1811.11811

On the Promise and Pitfalls of Optimizing Embodied Carbon

Noman Bashir, David Irwin, Prashant Shenoy

2306.15816

Federated Learning Based on Dynamic Regularization

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, Venkatesh Saligrama

2111.04263

PolyShard: Coded Sharding Achieves Linearly Scaling Efficiency and Security Simultaneously

Songze Li, Mingchao Yu, Chien-Sheng Yang, A. Salman Avestimehr, Sreeram Kannan, Pramod Viswanath

1809.10361

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim

2406.19707

Data-Quality Based Scheduling for Federated Edge Learning

Afaf Taik, Hajar Moudoud, Soumaya Cherkaoui

2201.11247

Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis

Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, Peter Michaleas

1807.07814

Asynchronous Distributed Optimization using a Randomized Alternating Direction Method of Multipliers

Franck Iutzeler, Pascal Bianchi, Philippe Ciblat, Walid Hachem

1303.2837

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Jinming Xu, Ye Tian, Ying Sun, Gesualdo Scutari

1910.10666

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, Sebastian U. Stich

2003.10422

Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization

Zhize Li, Dmitry Kovalev, Xun Qian, Peter Richtárik

2002.11364

Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

Ammar Ahmad Awan, Ching-Hsiang Chu, Hari Subramoni, Dhabaleswar K. Panda

1707.09414

A Class of Parallel Tiled Linear Algebra Algorithms for Multicore Architectures

Alfredo Buttari, Julien Langou, Jakub Kurzak, Jack Dongarra

0709.1272

A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning

T. Nathan Mundhenk, Goran Konjevod, Wesam A. Sakla, Kofi Boakye

1609.04453

Federated Learning over Wireless IoT Networks with Optimized Communication and Resources

Hao Chen, Shaocheng Huang, Deyou Zhang, Ming Xiao, Mikael Skoglund, H. Vincent Poor

2110.11775

Coded MapReduce

Songze Li, Mohammad Ali Maddah-Ali, A. Salman Avestimehr

1512.01625

Think Locally, Act Globally: Federated Learning with Local and Global Representations

Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy P. Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency

2001.01523

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, George Karypis

2010.05337

Byzantine-Resilient Secure Federated Learning

Jinhyun So, Basak Guler, A. Salman Avestimehr

2007.11115

Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies

Yae Jee Cho, Jianyu Wang, Gauri Joshi

2010.01243

The Online Pause and Resume Problem: Optimal Algorithms and An Application to Carbon-Aware Load Shifting

Adam Lechowicz, Nicolas Christianson, Jinhang Zuo, Noman Bashir, Mohammad Hajiesmaili, Adam Wierman, Prashant Shenoy

2303.17551

Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations

Debraj Basu, Deepesh Data, Can Karakus, Suhas Diggavi

1906.02367

ByRDiE: Byzantine-resilient distributed coordinate descent for decentralized learning

Zhixiong Yang, Waheed U. Bajwa

1708.08155

Optimal Client Sampling for Federated Learning

Wenlin Chen, Samuel Horvath, Peter Richtarik

2010.13723

FeatGraph: A Flexible and Efficient Backend for Graph Neural Network Systems

Yuwei Hu, Zihao Ye, Minjie Wang, Jiali Yu, Da Zheng, Mu Li, Zheng Zhang, Zhiru Zhang, Yida Wang

2008.11359

Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search

Chaoyang He, Murali Annavaram, Salman Avestimehr

2004.08546

Improved Convergence Rates for Distributed Resource Allocation

Angelia Nedić, Alex Olshevsky, Wei Shi

1706.05441

Planar Graph Perfect Matching is in NC

Nima Anari, Vijay V. Vazirani

1709.07822

Adaptive Federated Learning in Resource Constrained Edge Computing Systems

Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K. Leung, Christian Makaya, Ting He, Kevin Chan

1804.05271

Federated Learning: Challenges, Methods, and Future Directions

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, Virginia Smith

1908.07873

Matching is as Easy as the Decision Problem, in the NC Model

Nima Anari, Vijay V. Vazirani

1901.10387

Counting Triangles in Massive Graphs with MapReduce

Tamara G. Kolda, Ali Pinar, Todd Plantenga, C. Seshadhri, Christine Task

1301.5887