Document Open Access Logo

In-Situ Visual Programming

Authors Ulrich Brandstätter , Bernhard Schenkenfelder

PDF HTML 5

Files

Thumbnail PDF
PDF
OASIcs.Programming.2025.7.pdf
  • Filesize: 6.77 MB
  • 11 pages
HTML5 Logo HTML (experimental)
OASIcs.Programming.2025.7.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Human-centered computing → Ubiquitous and mobile computing theory, concepts and paradigms
Keywords
  • Visual programming
  • End-user programming
  • Programming paradigm

Metrics

Abstract

Most Visual Programming Environments (VPEs) available today aim to make software development more accessible for specific domains, such as automation, business intelligence, data science, education, or real-time media processing. In their niches, VPEs offer several advantages over traditional text-based programming, including shorter training times, immediate visual feedback, and lower barriers to entry. With this work, we introduce In-Situ Visual Programming (ISVP), a novel programming paradigm to enable users to create, modify, and contribute to software via visual programming in physical contexts. User-created and pre-built programs can be attached to and interlinked with physical objects - in an Augmented Reality (AR) environment. We believe that the spatial and contextual proximity of processing code and physical objects will make software development more intuitive, and we argue this position based on two model use cases.

Cite As Get BibTex

Ulrich Brandstätter and Bernhard Schenkenfelder. In-Situ Visual Programming. In Companion Proceedings of the 9th International Conference on the Art, Science, and Engineering of Programming (Programming 2025). Open Access Series in Informatics (OASIcs), Volume 134, pp. 7:1-7:11, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.Programming.2025.7

Author Details

Ulrich Brandstätter
  • Software Competence Center Hagenberg GmbH (SCCH), Austria
Bernhard Schenkenfelder
  • Software Competence Center Hagenberg GmbH (SCCH), Austria

Funding

The research reported in this paper has been funded by the Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK), the Federal Ministry for Labour and Economy (BMAW), and the State of Upper Austria in the frame of the SCCH competence center INTEGRATE (FFG grant no. 892418) in the COMET - Competence Centers for Excellent Technologies Programme managed by Austrian Research Promotion Agency FFG.

References

  1. Yuliana Apaza-Yllachura, Alfredo Paz-Valderrama, and Carlo Corrales-Delgado. Simplear: Augmented reality high-level content design framework using visual programming. In 2019 38th International Conference of the Chilean Computer Science Society (SCCC), pages 1-7, Concepcion, Chile, November 2019. IEEE. URL: https://doi.org/10.1109/SCCC49216.2019.8966427.
  2. Ryan Benac and Tauheed Khan Mohd. Recent trends in software development: Low-code solutions. In Kohei Arai, editor, Proceedings of the Future Technologies Conference (FTC) 2021, Volume 3, Lecture Notes in Networks and Systems, pages 525-533, Cham, 2022. Springer International Publishing. URL: https://doi.org/10.1007/978-3-030-89912-7_41.
  3. Sassi Bentrad and Djamel Meslati. Visual programming and program visualization – towards an ideal visual software engineering system –. ACEEE International Journal on Information Technology, 1:56-62, January 2011. Google Scholar
  4. Mary Beth Kery and Brad A. Myers. Exploring exploratory programming. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 25-29, 2017. URL: https://doi.org/10.1109/VLHCC.2017.8103446.
  5. J.A. Borges and R.E. Johnson. Multiparadigm visual programming language. In Proceedings of the 1990 IEEE Workshop on Visual Languages, pages 233-240, 1990. URL: https://doi.org/10.1109/WVL.1990.128412.
  6. Ulrich Brandstätter, Bernhard Schenkenfelder, Doris Hohensinger, and Harald Kirchtag. Design considerations for a multi-user general-purpose flow-based visual programming environment. In Proceedings of the 2024 International Conference on Advanced Visual Interfaces, pages 1-3, Arenzano, Genoa Italy, June 2024. ACM. URL: https://doi.org/10.1145/3656650.3656751.
  7. Brian Broll, Ákos Lédeczi, Hamid Zare, Dung Nguyen Do, János Sallai, Péter Völgyesi, Miklós Maróti, Lesa Brown, and Chris Vanags. A visual programming environment for introducing distributed computing to secondary education. Journal of Parallel and Distributed Computing, 118:189-200, 2018. URL: https://doi.org/10.1016/j.jpdc.2018.02.021.
  8. James C. Browne, Syed I. Hyder, Jack Dongarra, Keith Moore, and Peter Newton. Visual programming and debugging for parallel computing. IEEE Parallel Distrib. Technol., 3(1):75-83, March 1995. URL: https://doi.org/10.1109/88.384586.
  9. Alessio Bucaioni, Antonio Cicchetti, and Federico Ciccozzi. Modelling in low-code development: a multi-vocal systematic review. Software and Systems Modeling, 21:1959-1981, 2022. URL: https://doi.org/10.1007/S10270-021-00964-0.
  10. Margaret M. Burnett. Visual Programming, chapter 1, pages 275-283. John Wiley & Sons, Ltd, 1999. URL: https://doi.org/10.1002/047134608X.W1707.
  11. Margaret M. Burnett, Marla J. Baker, Carisa Bohus, Paul Carlson, Sherry Yang, and Pieter van Zee. Scaling up visual programming languages. Computer, 28:45-54, 1995. URL: https://doi.org/10.1109/2.366157.
  12. Rina Diane Caballar. Programming without code: The rise of no-code software development - ieee spectrum, March 2020. URL: https://spectrum.ieee.org/programming-without-code-no-code-software-development.
  13. Jordi Cabot. Positioning of the low-code movement within the field of model-driven engineering. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, pages 1-3, Virtual Event Canada, October 2020. ACM. URL: https://doi.org/10.1145/3417990.3420210.
  14. Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, Wang You, Ting Song, Yan Xia, Jonathan Tien, and Nan Duan. Low-code llm: Visual programming over llms, 2023. URL: https://doi.org/10.48550/arXiv.2304.08103.
  15. J.D. Campbell. Multi-user collaborative visual program development. In Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments, pages 122-130, 2002. URL: https://doi.org/10.1109/HCC.2002.1046364.
  16. Davide Di Ruscio, Dimitris Kolovos, Juan de Lara, Alfonso Pierantonio, Massimo Tisi, and Manuel Wimmer. Low-code development and model-driven engineering: Two sides of the same coin? Software and Systems Modeling, 21(2):437-446, April 2022. URL: https://doi.org/10.1007/s10270-021-00970-2.
  17. James Dominic, Brock Tubre, Charles Ritter, Jada Houser, Colton Smith, and Paige Rodeghero. Remote pair programming in virtual reality. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 406-417, Adelaide, Australia, September 2020. IEEE. URL: https://doi.org/10.1109/ICSME46990.2020.00046.
  18. Ruofei Du, Na Li, Jing Jin, Michelle Carney, Scott Miles, Maria Kleiner, Xiuxiu Yuan, Yinda Zhang, Anuva Kulkarni, Xingyu Liu, Ahmed Sabie, Sergio Orts-Escolano, Abhishek Kar, Ping Yu, Ram Iyengar, Adarsh Kowdle, and Alex Olwal. Rapsai: Accelerating machine learning prototyping of multimedia applications through visual programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3544548.3581338.
  19. Mahmoud Fayed, Muhammad AL-Qurishi, Atif Alamri, M. Hossain, and Ahmad Al-Daraiseh. Pwct: a novel general-purpose visual programming language in support of pervasive application development. CCF Transactions on Pervasive Computing and Interaction, 2, August 2020. URL: https://doi.org/10.1007/s42486-020-00038-y.
  20. Henrique Henriques, Hugo Lourenço, Vasco Amaral, and Miguel Goulão. Improving the developer experience with a low-code process modelling language. In Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pages 200-210, Copenhagen Denmark, October 2018. ACM. URL: https://doi.org/10.1145/3239372.3239387.
  21. Robert Hirschfeld, Pascal Costanza, and Michael Haupt. An Introduction to Context-Oriented Programming with ContextS, volume 5235 of Lecture Notes in Computer Science, pages 396-407. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. URL: https://doi.org/10.1007/978-3-540-88643-3_9.
  22. Robert Hirschfeld, Pascal Costanza, and Oscar Nierstrasz. Context-oriented programming. Journal of Object Technology, 7(3):125-151, 2008. URL: https://doi.org/10.5381/JOT.2008.7.3.A4.
  23. Martin Hirzel. Low-code programming models. Commun. ACM, 66(10):76-85, September 2023. URL: https://doi.org/10.1145/3587691.
  24. Michal Kapinus. End-User Cobot Programming in Augmented Reality. Ph.d. thesis, Brno University of Technology, Faculty of Information Technology, 2023. URL: https://www.fit.vut.cz/study/phd-thesis/891/.
  25. Anastasia Kovalkov, Avi Segal, and Kobi Gal. In the eye of the beholder? detecting creativity in visual programming environments. arXiv, April 2020. Google Scholar
  26. Mohammad Amin Kuhail, Shahbano Farooq, Rawad Hammad, and Mohammed Bahja. Characterizing visual programming approaches for end-user developers: A systematic review. IEEE Access, 9:14181-14202, 2021. URL: https://doi.org/10.1109/ACCESS.2021.3051043.
  27. Yustika Maharani, Cucuk Budiyanto, and Rosihan Yuana. The art of computational thinking through visual programming: A literature review. In The 3rd International Conference on Science, Mathematics, Environment, and Education, volume 2540, page 080036, January 2023. URL: https://doi.org/10.1063/5.0105766.
  28. Tim Menzies. Evaluation issues for visual programming languages. In Handbook of Software Engineering and Knowledge Engineering, May 2002. URL: https://doi.org/10.1142/9789812389701_0005.
  29. Leonel Merino, Boris Sotomayor-Gómez, Xingyao Yu, Ronie Salgado, Alexandre Bergel, Michael Sedlmair, and Daniel Weiskopf. Toward agile situated visualization: An exploratory user study. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-7, Honolulu HI USA, April 2020. ACM. URL: https://doi.org/10.1145/3334480.3383017.
  30. John T. Murray. Realityflow: Open-source multi-user immersive authoring. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 65-68, 2022. URL: https://doi.org/10.1109/VRW55335.2022.00024.
  31. B. A. Myers. Visual programming, programming by example, and program visualization: A taxonomy. SIGCHI Bull., 17(4):59-66, April 1986. URL: https://doi.org/10.1145/22339.22349.
  32. Vinh T. Nguyen, Kwanghee Jung, and Tommy Dang. Blocklyar: A visual programming interface for creating augmented reality experiences. Electronics, 9(8):1205, July 2020. URL: https://doi.org/10.3390/electronics9081205.
  33. Daniel Pinho, Ademar Aguiar, and Vasco Amaral. What about the usability in low-code platforms? a systematic literature review. Journal of Computer Languages, 74:101185, January 2023. URL: https://doi.org/10.1016/j.cola.2022.101185.
  34. José Quiroz-Fabián, Graciela Román-Alonso, Miguel Garcia, Jorge Buenabad-Chávez, Azzedine Boukerche, and Manuel Aguilar-Cornejo. Vppe: A novel visual parallel programming environment. International Journal of Parallel Programming, 47:1117-1151, December 2019. URL: https://doi.org/10.1007/s10766-019-00639-w.
  35. Elisa Rojas, Eder Ollora Zaballa, and Victoria Noci. Towards visual programming abstractions in software-defined networking. Internet Technology Letters, 5(3):e358, 2022. URL: https://doi.org/10.1002/itl2.358.
  36. Radosław Roszczyk, Marek Wdowiak, Michał Śmiałek, Kamil Rybiński, and Krzysztof Marek. Balticlsc: A low-code hpc platform for small and medium research teams. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 1-4, 2021. URL: https://doi.org/10.1109/VL/HCC51201.2021.9576305.
  37. Kamil Rybiński, Michał undefinedmiałek, Agris Sostaks, Krzysztof Marek, Radosław Roszczyk, and Marek Wdowiak. Visual low-code language for orchestrating large-scale distributed computing. J. Grid Comput., 21(3), July 2023. URL: https://doi.org/10.1007/s10723-023-09666-x.
  38. Apurvanand Sahay, Arsene Indamutsa, Davide Di Ruscio, and Alfonso Pierantonio. Supporting the understanding and comparison of low-code development platforms. In 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pages 171-178, August 2020. URL: https://doi.org/10.1109/SEAA51224.2020.00036.
  39. Bernhard Schenkenfelder, Christian Salomon, Georg Buchgeher, Robert Schossleitner, and Christian Kerl. The potential of low-code development in the manufacturing industry. In 2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA), pages 1-8, Sinaia, Romania, September 2023. IEEE. URL: https://doi.org/10.1109/ETFA54631.2023.10275503.
  40. Ben Selwyn-Smith, Craig Anslow, Michael Homer, and James R. Wallace. Co-located collaborative block-based programming. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 107-116, 2019. URL: https://doi.org/10.1109/VLHCC.2019.8818895.
  41. Karin Stolpe and Jonas Hallström. Visual Programming as a Tool for Developing Knowledge in STEM Subjects: A Literature Review, pages 130-169. Brill Academic Publishers, January 2024. URL: https://doi.org/10.1163/9789004687912_007.
  42. Srikanth G Tamilselvam, Naveen Panwar, Shreya Khare, Rahul Aralikatte, Anush Sankaran, and Senthil Mani. A visual programming paradigm for abstract deep learning model development. In Proceedings of the 10th Indian Conference on Human-Computer Interaction, IndiaHCI '19, New York, NY, USA, 2019. Association for Computing Machinery. URL: https://doi.org/10.1145/3364183.3364202.
  43. Bruce H. Thomas, Gregory F. Welch, Pierre Dragicevic, Niklas Elmqvist, Pourang Irani, Yvonne Jansen, Dieter Schmalstieg, Aurélien Tabard, Neven A. M. ElSayed, Ross T. Smith, and Wesley Willett. Situated Analytics, volume 11190 of Lecture Notes in Computer Science, pages 185-220. Springer International Publishing, Cham, 2018. URL: https://doi.org/10.1007/978-3-030-01388-2_7.
  44. Juraj Vincur, Martin Konopka, Jozef Tvarozek, Martin Hoang, and Pavol Navrat. Cubely: virtual reality block-based programming environment. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, pages 1-2, Gothenburg Sweden, November 2017. ACM. URL: https://doi.org/10.1145/3139131.3141785.
  45. Bianca Wiesmayr, Alois Zoitl, and Rick Rabiser. Assessing the usefulness of a visual programming ide for large-scale automation software. In 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS), pages 297-307, 2021. Google Scholar
  46. Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. Promptchainer: Chaining large language model prompts through visual programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA '22, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3491101.3519729.
  47. Lei Zhang and Steve Oney. Studying the benefits and challenges of immersive dataflow programming. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 223-227, Memphis, TN, USA, October 2019. IEEE. URL: https://doi.org/10.1109/VLHCC.2019.8818856.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact