Automating Mobile WEB App Prototyping from Wireframe Sketches
DOI:
https://doi.org/10.38035/dijemss.v7i1.5408Keywords:
Sketch to Code, YOLOv11, Wireframe, Deep LearningAbstract
Manual development of mobile web app prototypes is time-consuming, costly, and error-prone, particularly in arranging UI elements. This study aims to develop an automated system based on Deep Learning that can directly convert wireframe sketches into HTML and Bootstrap CSS code. The system employs the YOLOv11 algorithm to detect UI elements in the wireframe sketches, which are then translated into a hierarchical structure using a Domain-Specific Language (DSL) as an intermediary between detection results and code generation. The UI elements arranged within the DSL structure are subsequently converted into HTML and Bootstrap CSS code using a template-based approach with the Bootstrap 5 framework, ensuring layout consistency with the original design. The results show that the system is capable of generating prototypes with an element conversion success rate exceeding 85%. Evaluations conducted with several respondents indicate that the system can accelerate the early design process of web applications, reduce manual errors, and optimize resource usage in software development. These findings highlight the potential of integrating DSL and Deep Learning to automate the generation of accurate and efficient web UI prototypes from wireframe sketches.
References
Abdelhamid, A. A., Alotaibi, S. R., & Mousa, A. (2020). Deep Learning -based prototyping of android gui from hand-drawn mockups. IET Software, 14(7), 816–824. https://doi.org/10.1049/iet-sen.2019.0378
Adarsh, S., Harish, D., Balaganapathy, K., Venkatachalapaty, R., Abishiek, E., & Nagarajan, M. (2017). Improved Software Quality and Design Standards Based on Customer Preferences by Applying Evolutionary Prototyping Software Development Model. International Journal of New Technology and Research, 3(5). www.ijntr.org?
Adefris, B. B., Habtie, A. B., & Taye, Y. G. (2022). Automatic Code Generation From Low Fidelity Graphical User Interface Sketches Using Deep Learning . 2022 International Conference on Information and Communication Technology for Development for Africa, ICT4DA 2022, 47–52. https://doi.org/10.1109/ICT4DA56482.2022.9971204
Aleksi?, D., Savi?, D., Vlaji?, S., Rodrigues Da Silva, A., Stanojevi?, V., Antovi?, I., & Mili?, M. (2016). Generate User Interface Using Xtext Framework.
Beltramelli, T. (2017). pix2code: Generating Code from a Graphical User Interface Screenshot. http://arxiv.org/abs/1705.07962
Carlos, J., Ferreira, S., & Ferreira, H. S. (2019). Live Web Prototypes from Hand-Drawn Mockups.
De Macedo, G. T., Fontão, A. de L., & Gadelha, B. F. (2023). Prototyping in Software Quality Assurance: A Survey With Software Practitioners. Proceedings of the XXI Brazilian Symposium on Software Quality. https://doi.org/10.1145/3571473.3571477
Dehaerne, E., Dey, B., Halder, S., De Gendt, S., & Meert, W. (2022). Code Generation Using Machine Learning: A Systematic Review. Dalam IEEE Access (Vol. 10, hlm. 82434–82455). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ACCESS.2022.3196347
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017). Rico: A mobile app dataset for building data-driven design applications. UIST 2017 - Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 845–854. https://doi.org/10.1145/3126594.3126651
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303–338. https://doi.org/10.1007/s11263-009-0275-4
Iglesias, M. J. F. (2023). Prototyping. Dalam Design Thinking for Engineering (hlm. 91–109). https://doi.org/10.1049/PBME024E_ch6
Kolthoff, K. (2019). Automatic Generation of Graphical User Interface Prototypes from Unrestricted Natural Language Requirements. 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), 1234–1237. https://doi.org/10.1109/ASE.2019.00148
Konat, G., Steindorfer, M. J., Erdweg, S., & Visser, E. (2018). PIE: A Domain-Specific Language for Interactive Software Development Pipelines. Art, Science, and Engineering of Programming, 2(3). https://doi.org/10.22152/programming-journal.org/2018/2/9
Le, T. H. M., Chen, H., & Babar, M. A. (2020). Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges. ACM Computing Surveys, 53(3). https://doi.org/10.1145/3383458
Li, X., Liu, Z., Schäf, M., & Yin, L. (2010). AutoPA: Automatic prototyping from requirements. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 6415 LNCS(PART 1), 609–624. https://doi.org/10.1007/978-3-642-16558-0_49
Liang, J. (2024). A review of the development of YOLO object detection algorithm. Applied and Computational Engineering, 71(1), 39–46. https://doi.org/10.54254/2755-2721/71/20241642
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal Loss for Dense Object Detection . http://arxiv.org/abs/1708.02002
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2015). SSD: Single Shot MultiBox Detector. https://doi.org/10.1007/978-3-319-46448-0_2
Mohian, S., & Csallner, C. (2020). Doodle2App: Native app code by freehand UI sketching. Proceedings - 2020 IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems, MOBILESoft 2020, 81–84. https://doi.org/10.1145/3387905.3388607
Olmsted-Hawala, E. L., Romano, J. C., & Murphy, E. D. (2009). The use of paper-prototyping in a low-fidelity usability study. 2009 IEEE International Professional Communication Conference, 1–11. https://doi.org/10.1109/IPCC.2009.5208693
Rahman, M. M., Watanobe, Y., & Nakamura, K. (2021). A bidirectional LSTM language model for code evaluation and repair. Symmetry, 13(2), 1–15. https://doi.org/10.3390/sym13020247
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. http://arxiv.org/abs/1506.01497
Robinson, A. (2019). Sketch2code: Generating a website from a paper mockup. http://arxiv.org/abs/1905.13750
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., & Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), 211–252. https://doi.org/10.1007/s11263-015-0816-y
Sonje, S., Dave, H., Pardeshi, J., & Chaudhari, S. (2022). draw2code: AI based Auto Web Page Generation from Hand-drawn Page Mock-up. 2022 IEEE 7th International conference for Convergence in Technology, I2CT 2022. https://doi.org/10.1109/I2CT54291.2022.9824521
Uzayr, S. bin. (2022). Prototyping. Dalam Mastering Sketch. http://dx.doi.org/10.1201/9781003261575-6
Web UI Sketch Dataset . (t.t.). Diambil 24 April 2025, dari https://www.kaggle.com/dataset s/alaalsanea/sketch-web-ui-dataset
Yang, Y., Li, X., Ke, W., & Liu, Z. (2020). Automated Prototype Generation From Formal Requirements Model. IEEE Transactions on Reliability, 69(2), 632–656. https://doi.org/10.1109/TR.2019.2934348
Zhao, Z.-Q., Zheng, P., Xu, S., & Wu, X. (2018). Object Detection with Deep Learning : A Review. http://arxiv.org/abs/1807.05511
Zou, Z., Chen, K., Shi, Z., Guo, Y., & Ye, J. (2023). Object Detection in 20 Years: A Survey. Proceedings of the IEEE, 111(3), 257–276. https://doi.org/10.1109/JPROC.2023.3238524
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Fajrul Hidayat, Darlis Herumurti
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish their manuscripts in this journal agree to the following conditions:
- The copyright on each article belongs to the author(s).
- The author acknowledges that the Dinasti International Journal of Education Management and Social Science (DIJEMSS) has the right to be the first to publish with a Creative Commons Attribution 4.0 International license (Attribution 4.0 International (CC BY 4.0).
- Authors can submit articles separately, arrange for the non-exclusive distribution of manuscripts that have been published in this journal into other versions (e.g., sent to the author's institutional repository, publication into books, etc.), by acknowledging that the manuscript has been published for the first time in the Dinasti International Journal of Education Management and Social Science (DIJEMSS).
