Vol. 1 No. 4 (2026)

Articles

Hybrid-Attention Denoising Trajectories for Discrete Token Generation Under Uncertainty

Published 2026-09-22

  • Grace Ka-Man Tsang
    • ,
  • Junyi Zou

Grace Ka-Man Tsang
Department of Computer Science and Engineering, School of Engineering, Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR, China

Junyi Zou
Department of Computer Science and Engineering, School of Engineering, Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR, China

Language, Knowledge and Intelligent Systems, Volume 1, Issue 4, 2026 cover

Abstract

Diffusion models have revolutionized continuous data generation, yet their application to discrete token generation remains structurally challenging due to the inherent non-differentiability of categorical spaces. This paper introduces a comprehensive framework utilizing hybrid-attention denoising trajectories to navigate discrete token generation under severe generative uncertainty. By coupling local self-attention mechanisms with global cross-attention constraints, the proposed methodology establishes a continuous proxy space where denoising trajectories can be smoothly optimized before mapping back to discrete categorical variables. We evaluate the resilience of this architecture against various stochastic perturbations, demonstrating that hybrid-attention pathways significantly mitigate compounding errors that are typically observed in standard autoregressive decoding and foundational discrete diffusion models. Extensive empirical analyses reveal that the integration of multi-scale attention heads allows the model to maintain syntactic and semantic coherence even when token probabilities are uniformly distributed across exceptionally large vocabularies. The empirical findings suggest a critical paradigm shift in how generative uncertainty is mathematically quantified and computationally managed in non-continuous spaces, thereby paving the way for more robust natural language processing frameworks, autonomous code generation systems, and structured discrete data synthesis pipelines.

Keywords

Discrete Diffusion, Hybrid Attention, Token Generation, Generative Uncertainty

PDF

Author Biography

Grace Ka-Man Tsang

Editorial correspondence on behalf of the authors

References

  1. Liu, H., & Henao, R. (2025, April). Learning to substitute words with model-based score ranking. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 11551-11565).
  2. Pei, Z., Zhang, A., Wang, S., & Huang, Q. (2024, July). Modeling language tokens as functionals of semantic fields. In Forty-first International Conference on Machine Learning.
  3. Liu, R., McKay, D., & Buchanan, G. (2024). Sociotechnical factors affecting interoperability of digital humanities. Proceedings of the Association for Information Science and Technology, 61(1), 218-231.
  4. Sang, Y. (2026, March). Adaptive Quantization Strategies for Robust ML Inference Under Distribution Shift. In Proceedings of the 2026 5th International Conference on Cyber Security, Artificial Intelligence and Digital Economy (pp. 362-368).
  5. Li, X., Wang, P., Li, G., Ni, L., & Zhang, Y. (2023). Design of interface circuits and lightweight PUF for TMR sensors. IEEE Sensors Journal, 23(11), 11754-11761.
  6. Sun, J., Xue, Y., Li, M., Yao, Z., Li, J., Cui, Y., ... & Zhou, H. (2026). TrajWiki: Source-Grounded Memory Trajectories for Long-Horizon Dialogue Agents. arXiv preprint arXiv:2608.00967.
  7. Zou, Y., Lee, T., Fan, X., & Li, J. (2026, May). Benchmarking Large Language Models for Chinese and Japanese IMEs: Phonetic-to-Character Generation and Textual Error Correction. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) (Vol. 11, No. 16, pp. 4290-4311).
  8. Du, J., Gao, S., & Chen, C. H. (2024). A contextual ranking and selection method for personalized medicine. Manufacturing & Service Operations Management, 26(1), 167-181.
  9. Ye, Z., Ma, S., Yang, J., Yang, X., Yang, Y., Gong, Z., ... & Wang, H. (2026, April). Object fidelity diffusion for remote sensing image generation. In International Conference on Learning Representations (Vol. 2026, pp. 3258-3278).
  10. Zhang, B., Wang, S., Jiang, Y., Sui, D., Tu, Z., & Chu, D. (2025, July). Ask and Retrieve Knowledge: Towards Proactive Asking with Imperfect Information in Medical Multi-turn Dialogues. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1055-1065).
  11. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems.
  12. Li, G., Zhang, J., Tan, M. H. Y., Wang, B., & Huang, W. (2026). SeqLLM: Augmenting LLMs with Behavioral-Sequence Modeling for High-Stakes Decisions at WeChat Pay. arXiv preprint arXiv:2608.03063.
  13. Wang, Y., Fan, S., Wang, M., Gao, S., & Wang, C. (2026, April). DAMR: Efficient and adaptive context-aware knowledge graph question answering with LLM-guided MCTS. In International Conference on Learning Representations (Vol. 2026, pp. 25548-25570).
  14. Fang, S., Shen, I., Wang, Y., Tsai, Y. H., Yang, Y., Zhou, S., ... & Yang, M. H. (2025). Meshllm: Empowering large language models to progressively understand and generate 3d mesh. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 14061-14072).
  15. Hao, Y., Xu, M., Ye, C., Qin, J., Lu, S., Qin, Y., & Han, X. (2026). LoFA: Learning to Predict Personalized Prior for Fast Adaptation of Visual Generative Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21986-21996).
  16. Luong, M.-T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP.
  17. Yu, H., Zhang, J., Chen, C., Xiang, T., Fang, Y., Niebles, J. C., & Adeli, E. (2026, March). Socialgen: Modeling multi-human social interaction with language models. In 2026 International Conference on 3D Vision (3DV) (pp. 1-17). IEEE.
  18. Wang, Z., Hou, J., & Zhang, H. (2026, July). Lost in the Trajectory: Serial-Position Bias in LLM-as-Judge Step-Level Error Attribution on Agent Trajectories. In 2026 5th International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) (pp. 354-358). IEEE.
  19. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL.
  20. Hu, X. (2026). Platform Governance in Generative Search: A Theory of Authority, Relevance, and Welfare. Relevance, and Welfare (March 27, 2026).
  21. Wu, J., Chen, S., Heo, I., Gutfraind, S., Liu, S., Li, C., ... & Sharps, M. (2025). Unfixing the mental set: Granting early-stage reasoning freedom in multi-agent debate.
  22. Liu, X., Yu, Z., Yao, Y., Yang, K., Zeng, H., & Chen, C. P. (2026). A Comprehensive Survey of Cross-Modal Retrieval: Breaking Through Modal Barriers. ACM Computing Surveys.
  23. Yao, Y., Zhang, J., Miao, P., Zhang, L., Chen, G., Shu, F., & Wong, K. K. (2025). Hybrid RIS-enhanced ISAC secure systems: Joint optimization in the presence of an extended target. IEEE Transactions on Communications.
  24. Zhu, R., Ma, Z., Wu, J., Gao, J., Wang, J., Lin, D., & He, C. (2025, April). Utilize the flow before stepping into the same river twice: Certainty represented knowledge flow for refusal-aware instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 24, pp. 26157-26165).
  25. M. Emani, S. Foreman, V. Sastry, Z. Xie, S. Raskar, W. Arnold, R. Thakur, V. Vishwanath, M.E. Papka, A comprehensive performance study of large language models on novel ai accelerators. (2023). arXiv preprint arXiv:2310.04607
  26. Su, Z., Wei, H., Cen, K., Wang, Y., Chen, G., Yuan, C., & Chu, X. (2026). Generation enhances understanding in unified multimodal models via multi-representation generation. arXiv preprint arXiv:2601.21406.
  27. Qu, W., Shao, Y., Meng, L., Huang, X., & Xiao, L. (2024, June). A conditional denoising diffusion probabilistic model for point cloud upsampling. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 20786-20795). IEEE.
  28. Hu, Y., Yang, Z., Zhao, C., & Ji, W. (2026, May). Adaptive guidance semantically enhanced via multimodal llm for edge-cloud object detection. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 12962-12966). IEEE.
  29. Fensham, R., Sumner, T. D., Cutter, N., Buchanan, G., Liu, R., Munoz, J., ... & Van Der Plaat, D. (2024). Towards a National Data Architecture for Cultural Collections: Designing the Australian Cultural Data Engine. Digital Humanities Quarterly, 18(2).
  30. Lyu, S., Mao, S., & Ma, Y. (2025). Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising. arXiv preprint arXiv:2511.03272.
  31. Li, Y. (2026, July). Retrieval-Augmented Generation for Historical Fault Experience Reuse and Intelligent Diagnosis. In 2026 International Conference on Large Models and Intelligent Agent Technologies (LMIAT) (pp. 47-51). IEEE.
  32. Yao, Y., Zhu, Z., Miao, P., Cheng, X., Shu, F., & Wang, J. (2025). Optimizing hybrid RIS-aided ISAC systems in V2X networks: A deep reinforcement learning method for anti-eavesdropping techniques. IEEE Transactions on Vehicular Technology, 74 (6), 9224-9239.
  33. Yao, Z., Li, R., Chen, X., Cheng, J., Le, C., Yue, M., ... & Dong, J. (2026). PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention. arXiv preprint arXiv:2607.25157.
  34. Ding, M., Gao, S., & Zhao, Q. (2025, December). Stopping Rules for Sampling in Precision Medicine. In 2025 Winter Simulation Conference (WSC) (pp. 929-940). IEEE.
  35. Lyu, S., Wang, M., Zhang, H., Zheng, J., Lin, J., & Sun, X. (2025, October). Integrating structure-aware attention and knowledge graphs in explainable recommendation systems. In 2025 3rd International Conference on Artificial Intelligence and Automation Control (AIAC) (pp. 447-451). IEEE.
  36. Hu, X., Wan, Z., & Yin, Q. (2025). Token Design for Favor Trading with Dynamic Membership. Available at SSRN 6175079.
  37. Mo Z. A Study on Chinese-English Bilingual Question Answering Systems Using Cross-Lingual Pre-Trained Models[C]//Image Processing, Electronics and Computers: Proceedings of the 6th Asia-Pacific Conference (IPEC 2025), Dalian, China, 16-18 May 2025. 1 Oliver's Yard, 55 City Road, London, EC1Y 1SP: SAGE Publications, 2025: 1715-1724.