Document Open Access Logo

Bayesian Perspective on Memorization and Reconstruction

Authors Haim Kaplan , Yishay Mansour , Kobbi Nissim , Uri Stemmer

PDF

File

Thumbnail PDF
PDF
LIPIcs.ITCS.2026.84.pdf
  • Filesize: 0.82 MB
  • 18 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Security and privacy → Formal security models
Keywords
  • Reconstruction
  • Memorization
  • Differential privacy

Metrics

Abstract

We introduce a new Bayesian perspective on the concept of data reconstruction, and leverage this viewpoint to propose a new security definition that, in certain settings, provably prevents reconstruction attacks. We use our paradigm to shed new light on one of the most notorious attacks in the privacy and memorization literature - fingerprinting code attacks (FPC). We argue that these attacks are really a form of membership inference attacks, rather than reconstruction attacks. Furthermore, we show that if the goal is solely to prevent reconstruction (but not membership inference), then in some cases the impossibility results derived from FPC no longer apply.

Cite As Get BibTex

Haim Kaplan, Yishay Mansour, Kobbi Nissim, and Uri Stemmer. Bayesian Perspective on Memorization and Reconstruction. In 17th Innovations in Theoretical Computer Science Conference (ITCS 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 362, pp. 84:1-84:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ITCS.2026.84

Author Details

Haim Kaplan
  • Tel Aviv University, Israel
  • Google Research, Tel Aviv, Israel
Yishay Mansour
  • Tel Aviv University, Israel
  • Google Research, Tel Aviv, Israel
Kobbi Nissim
  • Georgetown University, Washington, DC, USA
  • Google Research, USA
Uri Stemmer
  • Tel Aviv University, Israel
  • Google Research, Tel Aviv, Israel

Funding

  • Kaplan, Haim: Partially supported by the Israel Science Foundation (grant 1156/23), and the Blavatnik Research Foundation.
  • Mansour, Yishay: This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science Foundation, the Yandex Initiative for Machine Learning at Tel Aviv University and a grant from the Tel Aviv University Center for AI and Data Science (TAD).
  • Nissim, Kobbi: Partially funded by NSF grant No. 2217678 and by a gift to Georgetown University.
  • Stemmer, Uri: Partially supported by the Israel Science Foundation (grant 1419/24), and the Blavatnik Research Foundation.

Acknowledgements

We thank the anonymous reviewers for their helpful comments.

References

  1. Idan Attias, Gintare Karolina Dziugaite, Mahdi Haghifam, Roi Livni, and Daniel M. Roy. Information complexity of stochastic convex optimization: Applications to generalization, memorization, and tracing. In ICML, 2024. Google Scholar
  2. Dan Boneh and James Shaw. Collusion-secure fingerprinting for digital data. IEEE Transactions on Information Theory, 44(5):1897-1905, 1998. URL: https://doi.org/10.1109/18.705568.
  3. Gavin Brown, Mark Bun, Vitaly Feldman, Adam D. Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In STOC, 2021. Google Scholar
  4. Mark Bun, Kobbi Nissim, and Uri Stemmer. Simultaneous private learning of multiple concepts. In ITCS, pages 369-380. ACM, 2016. URL: https://doi.org/10.1145/2840728.2840747.
  5. Mark Bun, Thomas Steinke, and Jonathan Ullman. Make up your mind: The price of online queries in differential privacy. In Proceedings of the twenty-eighth annual ACM-SIAM symposium on discrete algorithms, pages 1306-1325. SIAM, 2017. URL: https://doi.org/10.1137/1.9781611974782.85.
  6. Mark Bun, Jonathan Ullman, and Salil P. Vadhan. Fingerprinting codes and the price of approximate differential privacy. In STOC, pages 1-10. ACM, May 31 - june 3 2014. URL: https://doi.org/10.1145/2591796.2591877.
  7. Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv Nikankin, and Michal Irani. Deconstructing data reconstruction: Multiclass, weight decay and general losses. In NeurIPS, 2023. Google Scholar
  8. Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In USENIX Security, 2023. Google Scholar
  9. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2023. Google Scholar
  10. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security, 2019. Google Scholar
  11. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security, 2021. Google Scholar
  12. Edith Cohen, Haim Kaplan, Yishay Mansour, Shay Moran, Kobbi Nissim, Uri Stemmer, and Eliad Tsfadia. Data reconstruction: When you see it and when you don't. In ITCS, 2025. Google Scholar
  13. Thomas M. Cover and Joy A. Thomas. Elements of information theory (2. ed.). Wiley, 2006. Google Scholar
  14. Itai Dinur, Uri Stemmer, David P Woodruff, and Samson Zhou. On differential privacy and adaptive data analysis with bounded space. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages 35-65. Springer, 2023. URL: https://doi.org/10.1007/978-3-031-30620-4_2.
  15. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, 2006. Google Scholar
  16. Cynthia Dwork, Adam D. Smith, Thomas Steinke, Jonathan R. Ullman, and Salil P. Vadhan. Robust traceability from trace amounts. In FOCS, 2015. Google Scholar
  17. Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In STOC, 2020. Google Scholar
  18. Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In NeurIPS, 2020. Google Scholar
  19. Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. In NeurIPS, 2022. Google Scholar
  20. Moritz Hardt and Jonathan Ullman. Preventing false discovery in interactive data analysis is hard. In FOCS. IEEE, october 19-21 2014. Google Scholar
  21. Palak Jain, Sofya Raskhodnikova, Satchit Sivakumar, and Adam D. Smith. The price of differential privacy under continual observation. CoRR, abs/2112.00828, 2021. URL: https://arxiv.org/abs/2112.00828.
  22. Edwin T Jaynes. Probability theory: The logic of science. Cambridge university press, 2003. Google Scholar
  23. Gautam Kamath, Argyris Mouzakis, and Vikrant Singhal. New lower bounds for private estimation and a generalized fingerprinting lemma. Advances in neural information processing systems, 35:24405-24418, 2022. Google Scholar
  24. Haim Kaplan, Yishay Mansour, Kobbi Nissim, and Uri Stemmer. Separating adaptive streaming from oblivious streaming using the bounded storage model. In CRYPTO, pages 94-121, 2021. URL: https://doi.org/10.1007/978-3-030-84252-9_4.
  25. Haim Kaplan, Yishay Mansour, Kobbi Nissim, and Uri Stemmer. Bayesian perspective on memorization and reconstruction, 2025. URL: https://doi.org/10.48550/arXiv.2505.23658.
  26. Kobbi Nissim, Adam D. Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. The limits of post-selection generalization. In NeurIPS, pages 6402-6411, 2018. URL: https://proceedings.neurips.cc/paper/2018/hash/77ee3bc58ce560b86c2b59363281e914-Abstract.html.
  27. Naty Peter, Eliad Tsfadia, and Jonathan R. Ullman. Smooth lower bounds for differentially private algorithms via padding-and-permuting fingerprinting codes. In COLT, volume 247 of Proceedings of Machine Learning Research, pages 4207-4239. PMLR, 2024. URL: https://proceedings.mlr.press/v247/peter24a.html.
  28. Menachem Sadigurschi, Moshe Shechner, and Uri Stemmer. Relaxed models for adversarial streaming: The bounded interruptions model and the advice model. In ESA, volume 274 of LIPIcs, pages 91:1-91:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2023. URL: https://doi.org/10.4230/LIPIcs.ESA.2023.91.
  29. Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948. URL: https://doi.org/10.1002/J.1538-7305.1948.TB01338.X.
  30. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017. Google Scholar
  31. Thomas Steinke and Jonathan Ullman. Between pure and approximate differential privacy. CoRR, abs/1501.06095, 2015. URL: https://arxiv.org/abs/1501.06095.
  32. Thomas Steinke and Jonathan Ullman. Tight lower bounds for differentially private selection. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pages 552-563. IEEE, 2017. URL: https://doi.org/10.1109/FOCS.2017.57.
  33. Marika Swanberg, Meenatchi Sundaram Muthu Selva Annamalai, Jamie Hayes, Borja Balle, and Adam D. Smith. Beyond the worst case: Extending differential privacy guarantees to realistic adversaries. CoRR, abs/2507.08158, 2025. URL: https://doi.org/10.48550/arXiv.2507.08158.
  34. Gábor Tardos. Optimal probabilistic fingerprint codes. J. ACM, 55(2), 2008. URL: https://doi.org/10.1145/1346330.1346335.
  35. Salil Vadhan. The complexity of differential privacy, 2016. Google Scholar
  36. Sasha Voitovych, Mahdi Haghifam, Idan Attias, Gintare Karolina Dziugaite, Roi Livni, and Daniel M. Roy. On the dichotomy between privacy and traceability in 𝓁_p stochastic convex optimization. CoRR, abs/2502.17384, 2025. URL: https://doi.org/10.48550/arXiv.2502.17384.
  37. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE computer security foundations symposium (CSF), 2018. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact