Publications

2026

1. 

   TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar

   Yinxi Li, Yuntian Deng, and Pengyu Nie

   In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2026

   Abs DOI arXiv Bib PDF Code

   Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently depending on superficial factors such as whitespace or identifier naming. To measure the impact of this misalignment, we introduce TokDrift, a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. Across nine code LLMs, including large ones with over 30B parameters, even minor formatting changes can cause substantial shifts in model behavior. Layer-wise analysis shows that the issue originates in early embeddings, where subword segmentation fails to capture grammar token boundaries. Our findings identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation, highlighting the need for grammar-aware tokenization for future code LLMs.

   @inproceedings{li-etal-2026-tokdrift,
     title = {{T}ok{D}rift: When {LLM} Speaks in Subwords but Code Speaks in Grammar},
     author = {Li, Yinxi and Deng, Yuntian and Nie, Pengyu},
     editor = {Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David},
     booktitle = {Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)},
     month = jul,
     year = {2026},
     address = {San Diego, California, United States},
     publisher = {Association for Computational Linguistics},
     url = {https://arxiv.org/abs/2510.14972},
     doi = {10.18653/v1/2026.acl-long.2199},
     pages = {47605--47629},
     isbn = {979-8-89176-390-6},
   }

2. Learning Multi-step Reasoning via Persistent Latent State Propagation

   Yinxi Li, Jiaao Chen, Fang Wu, and 7 more authors

   In Workshop on Latent & Implicit Thinking – Going Beyond CoT Reasoning, Jul 2026
3. Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards

   Fang Wu, Aaron Tu, Weihao Xuan, and 21 more authors

   Jul 2026
4. Proteo-R1: Reasoning Foundation Models for De Novo Protein Design

   Fang Wu, Weihao Xuan, Heli Qi, and 26 more authors

   In Forty-third International Conference on Machine Learning, Jul 2026
5. NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training

   Fang Wu, Haokai Zhao, Da Xing, and 17 more authors

   Jul 2026
6. Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

   Liliana Hotsko, Yinxi Li, Yuntian Deng, and 1 more author

   Jul 2026