1 Search Results for "Goldstein, Tom"

Document

Invited Talk

DOI: 10.4230/LIPIcs.DISC.2018.2

Challenges for Machine Learning on Distributed Platforms (Invited Talk)

Authors: Tom Goldstein

Published in: LIPIcs, Volume 121, 32nd International Symposium on Distributed Computing (DISC 2018)

Abstract

Deep neural networks are trained by solving huge optimization problems with large datasets and millions of variables. On the surface, it seems that the size of these problems makes them a natural target for distributed computing. Despite this, most deep learning research still takes place on a single compute node with a small number of GPUs, and only recently have researchers succeeded in unlocking the power of HPC. In this talk, we'll give a brief overview of how deep networks are trained, and use HPC tools to explore and explain deep network behaviors. Then, we'll explain the problems and challenges that arise when scaling deep nets over large system, and highlight reasons why naive distributed training methods fail. Finally, we'll discuss recent algorithmic innovations that have overcome these limitations, including "big batch" training for tightly coupled clusters and supercomputers, and "variance reduction" strategies to reduce communication in high latency settings.

Cite as

Tom Goldstein. Challenges for Machine Learning on Distributed Platforms (Invited Talk). In 32nd International Symposium on Distributed Computing (DISC 2018). Leibniz International Proceedings in Informatics (LIPIcs), Volume 121, pp. 2:1-2:3, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2018)

Copy BibTex To Clipboard

@InProceedings{goldstein:LIPIcs.DISC.2018.2,
  author =	{Goldstein, Tom},
  title =	{{Challenges for Machine Learning on Distributed Platforms}},
  booktitle =	{32nd International Symposium on Distributed Computing (DISC 2018)},
  pages =	{2:1--2:3},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-092-7},
  ISSN =	{1868-8969},
  year =	{2018},
  volume =	{121},
  editor =	{Schmid, Ulrich and Widder, Josef},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops-dev.dagstuhl.de/entities/document/10.4230/LIPIcs.DISC.2018.2},
  URN =		{urn:nbn:de:0030-drops-97910},
  doi =		{10.4230/LIPIcs.DISC.2018.2},
  annote =	{Keywords: Machine learning, distributed optimization}
}

Refine by Author
1 Goldstein, Tom

Refine by Classification
1 Computing methodologies → Machine learning

Refine by Keyword
1 Machine learning
1 distributed optimization

Refine by Type
1 document

Refine by Publication Year
1 2018

1 Search Results for "Goldstein, Tom"

Challenges for Machine Learning on Distributed Platforms (Invited Talk)

Abstract

Cite as

Thanks for your feedback!

Could not send message