Cross-Terminal Intelligent Diagnosis and Treatment System Based on Multimodal Large Language Models

Abstract

To address the prominent challenges in current medical auxiliary diagnosis—including inaccurate tongue image feature extraction, poor disease adaptability, lack of cross-terminal collaboration, and high dependence on foreign core technologies—a cross-terminal intelligent diagnosis and treatment system based on multimodal large language models was designed and implemented. The system takes traditional Chinese medicine (TCM) tongue diagnosis as the core application scenario. Using SAM-2 with LoRA lightweight fine-tuning, pixel-level precise segmentation of tongue images is achieved with 97.2% accuracy. A heterogeneous fusion feature extraction architecture combining ResNet and Vision Transformer is proposed, enabling three-layer information fusion of tongue body, tongue coating, and tongue texture, improving disease prediction accuracy to 84%. The Qwen3-VL multimodal large language model integrated with Retrieval-Augmented Generation (RAG) technology constructs an interpretable disease prediction engine with a retrieval precision rate of 61%. Full-stack deployment is completed on the domestic Kunpeng CPU and Ascend NPU hardware platform, achieving an inference speed of 20 Token/s. Experimental results demonstrate that the system achieves significant performance in accuracy, interpretability, and domestic adaptation, validating the feasibility and efficiency of domestic hardware and software systems in handling complex multimodal large model tasks.