{"@context":"https:\/\/schema.org\/","@type":"ScholarlyArticle","@id":"#article9373","name":"Compression in a Distributed Setting","abstract":"Motivated by an attempt to understand the formation and development of (human) language, we introduce a \"distributed compression\" problem. In our problem a sequence of pairs of players from a set of K players are chosen and tasked to communicate messages drawn from an unknown distribution Q. \r\nArguably languages are created and evolve to compress frequently occurring messages, and we focus on this aspect.\r\nThe only knowledge that players have about the distribution Q is from previously drawn samples, but these samples differ from player to player.\r\nThe only common knowledge between the players is restricted to a common prior distribution P and some constant number\r\nof bits of information (such as a learning algorithm). \r\nLetting T_epsilon denote the number of iterations it would take for a typical player\r\nto obtain an epsilon-approximation to Q in total variation distance, we ask\r\nwhether T_epsilon iterations suffice to compress the messages down roughly to their\r\nentropy and give a partial positive answer.\r\n\r\nWe show that a natural uniform algorithm can compress the communication down to an average cost per\r\nmessage of O(H(Q) + log (D(P || Q)) in tilde{O}(T_epsilon) iterations\r\nwhile allowing for O(epsilon)-error,\r\nwhere D(. || .) denotes the KL-divergence between distributions.\r\nFor large divergences\r\nthis compares favorably with the static algorithm that ignores all samples and\r\ncompresses down to H(Q) + D(P || Q) bits, while not requiring T_epsilon * K iterations that it would take players to develop optimal but separate compressions for \r\neach pair of players.\r\nAlong the way we introduce a \"data-structural\" view of the task of\r\ncommunicating with a natural language and show that our natural algorithm can also be\r\nimplemented by an efficient data structure, whose storage is comparable to the storage requirements of Q and whose query complexity is comparable to the lengths of the message to be\r\ncompressed.\r\nOur results give a plausible mathematical analogy to the mechanisms by which\r\nhuman languages get created and evolve, and in particular highlights the\r\npossibility of coordination towards a joint task (agreeing on a language)\r\nwhile engaging in distributed learning.","keywords":["Distributed Compression","Communication","Language Evolution","Isolating Hash Families"],"author":[{"@type":"Person","name":"Ghazi, Badih","givenName":"Badih","familyName":"Ghazi"},{"@type":"Person","name":"Haramaty, Elad","givenName":"Elad","familyName":"Haramaty"},{"@type":"Person","name":"Kamath, Pritish","givenName":"Pritish","familyName":"Kamath"},{"@type":"Person","name":"Sudan, Madhu","givenName":"Madhu","familyName":"Sudan"}],"position":19,"pageStart":"19:1","pageEnd":"19:22","dateCreated":"2017-11-28","datePublished":"2017-11-28","isAccessibleForFree":true,"license":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/legalcode","copyrightHolder":[{"@type":"Person","name":"Ghazi, Badih","givenName":"Badih","familyName":"Ghazi"},{"@type":"Person","name":"Haramaty, Elad","givenName":"Elad","familyName":"Haramaty"},{"@type":"Person","name":"Kamath, Pritish","givenName":"Pritish","familyName":"Kamath"},{"@type":"Person","name":"Sudan, Madhu","givenName":"Madhu","familyName":"Sudan"}],"copyrightYear":"2017","accessMode":"textual","accessModeSufficient":"textual","creativeWorkStatus":"Published","inLanguage":"en-US","sameAs":"https:\/\/doi.org\/10.4230\/LIPIcs.ITCS.2017.19","publisher":"Schloss Dagstuhl \u2013 Leibniz-Zentrum f\u00fcr Informatik","isPartOf":{"@type":"PublicationVolume","@id":"#volume6270","volumeNumber":67,"name":"8th Innovations in Theoretical Computer Science Conference (ITCS 2017)","dateCreated":"2017-11-28","datePublished":"2017-11-28","editor":{"@type":"Person","name":"Papadimitriou, Christos H.","givenName":"Christos H.","familyName":"Papadimitriou"},"isAccessibleForFree":true,"publisher":"Schloss Dagstuhl \u2013 Leibniz-Zentrum f\u00fcr Informatik","hasPart":"#article9373","isPartOf":{"@type":"Periodical","@id":"#series116","name":"Leibniz International Proceedings in Informatics","issn":"1868-8969","isAccessibleForFree":true,"publisher":"Schloss Dagstuhl \u2013 Leibniz-Zentrum f\u00fcr Informatik","hasPart":"#volume6270"}}}