Large Language Models (LLMs) undeniably belie past objections to the viability of artificial intelligence (AI) and deep learning as research programs (Bar-Hillel, 1960; Pierce et al., 1966; Minsky and Papert, 1969; Lighthill, 1973). This is the outcome, against all odds, of decades of persistent progress: the modelling of language based on neural nets (Bengio et al., 2000), the learning of contextual representations of textual data (Peters et al., 2018; Devlin et al., 2019), the development of neural architectures optimized for text generation (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017), and the training of ever-larger models on vast and heterogeneous multilingual corpora (Raffel et al., 2020; Brown et al., 2020; Touvron et al., 2023). Sign of the times: whereas the very name of "AI" had become something of a taboo at the turn of the millennium (Ford, 2018)—with the discipline retreating away from its initial ambition of reproducing all aspects of human cognition in favour of narrower specialized use cases—talk of artificial general intelligence (AGI) is du jour once more (Bubeck et al., 2023; Morris et al., 2024; Hendrycks et al., 2025; Genewein et al., 2026). The emergent capabilities of LLMs are now being probed across a wide range of tasks and domains—mathematics, programming, vision, medicine, law, psychology—as objects of study in their own right (Cobbe et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Novikov et al. 2025; Singhal et al. 2022; Jin et al., 2020; Guha et al., 2023; Katz et al., 2024; inter alia). It is in this context that another set of LLM capabilities is garnering growing interest among researchers: their metalinguistic abilities, i.e., their capacity, not merely to use language, but to treat language itself as an object of analysis—to describe its structure, judge its forms, articulate and apply its rules, and reason from an explicit linguistic description to a novel case (Beguš et al., 2025; Schneider and Anastasopoulos, 2026).
These metalinguistic capabilities have direct and pervasive applications—some still unrealized, others already widely deployed yet lacking thorough theoretical validation. In high-resource settings, the ubiquitous deployment of LLMs—and increasingly, reasoning models—as advanced writing assistants (Handa et al., 2025; Chatterji et al., 2025) relies fundamentally on their capacity to engage in metalinguistic discourse, allowing them to critique stylistic choices, articulate grammatical corrections, and refine prose (Lee et al., 2022; Bryant et al., 2023; Vipul et al., 2023; Lee et al., 2024; Shu et al., 2024). This same metalinguistic acuity underpins the growing application of LLMs in language education, touting adaptive, AI-assisted pedagogy and the generation of dynamic feedback for both L1 and L2 learners (Kasneci et al., 2023; Barrot, 2023; Dai et al., 2023; Han, 2024; Li et al., 2025). Beyond these mainstream use cases, such metalinguistic competence is also evidently critical for extremely low-resource and endangered language processing. Where massive training corpora simply do not exist, an AI's ability to acquire linguistic competence directly from explicit metalinguistic descriptions—such as reference grammars and bilingual dictionaries—would go to great lengths in reducing the digital divide and averting digital language death (Kornai, 2013), not to mention that these capabilities would also stand to facilitate critical documentation tasks for linguists through automated interlinear glossed text prediction (Gin et al., 2023; Gin et al., 2024; Elsner & Liu, 2025), for instance.
While this topic holds the promise of various concrete, ethically stimulating applications, its fundamental theoretical implications are equally compelling. In human subjects, metalinguistic abilities are cognitively more complex than mere language use, and are acquired significantly later in development (Tunmer et al., 1984). In the context of artificial intelligence, the capacity to acquire linguistic competence from the metalinguistic explanations of a grammar book, for instance, would represent a highly sophisticated manifestation of intelligence, requiring information extraction, memorization, complex chained operations of inductive and deductive reasoning, and the conversion of declarative knowledge into active skills—a task at least as challenging, to say the least, as the abstract spatial reasoning puzzles popularized by ARC-AGI (Chollet, 2019; Chollet et al., 2025), currently the most revered AGI benchmark. Indeed, as Beguš et al. (2023) argue, investigating these metalinguistic abilities might even provide a rigorous, principled testing ground for accessing the broader metacognitive capacity of these models, which is itself considered one of the most crucial dimension of intelligence and consciousness (Fleming et al., 2012).
The primary objective of this PhD project is to investigate, measure, and enhance the metalinguistic abilities of LLMs. Positioned at the intersection of linguistics, natural language processing and deep learning, the research will systematically explore how highly parameterized neural architectures internalize, process, and generate language about language.
There will be three main axes to the initial topic, with opportunities to expand beyond these directions.
The first axis concerns the characterization and measurement of metalinguistic knowledge: developing methods to determine what models actually know about linguistic phenomena—across phonology, morphology, syntax and semantics, and across typologically diverse languages. This axis also involves taking seriously the gap between metalinguistic knowledge that text-trained models actually possess, and its relationship to the implicit linguistic competence they so evidently command (Beguš et al., 2023; Weissweiler et al., 2023; Suvarna et al., 2024; Arčon et al., 2026). The two are easily confounded—a model may wield a construction fluently yet prove unable to analyse it, or recite a textbook description it cannot bring to bear—and so the axis will seek to disentangle linguistic from metalinguistic competence across the levels of linguistic description. Directions here include the design of contrastive linguistically motivated benchmarks mixing Q&A tasks and cloze exercises.
The second axis turns from prior knowledge to in-context acquisition and use. The question here is whether models can apply linguistic information—grammar rules, morphological paradigms, lexicographical descriptions—to solve tasks. A privileged testbed is the translation of under-documented languages from reference materials, in a manner closer to second-language learning than to corpus-driven training (Tanzer et al., 2024; Marmonier et al., 2025), where it has repeatedly been found that the application of grammatical description is the bottleneck (Aycock et al., 2025; Pei et al., 2025; Hu et al., 2026), and where controlled, contamination-free settings such as constructed languages allow the locus of failure to be precisely diagnosed (Marmonier et al., 2025; Liu et al., 2025). Directions here include the design of such controlled benchmarks and the systematic analysis of where, and why, the application of explicit metalinguistic knowledge breaks down.
The third axis is methodological in nature, and asks how metalinguistic competence can be cultivated through training and fine-tuning. Several difficulties make this a substantial undertaking in its own right. Descriptive resources are long: a reference grammar runs to tens of thousands of tokens, so that supplying an entire grammatical description as working context, and inducing the model to retrieve and apply the relevant rule at the relevant moment, raises questions of long-context modelling and fine-tuning that remain challenging (Liu et al., 2024; Tanzer et al., 2024). The metalinguistic setting is, on the other hand, unusually amenable to learning from outcomes: grammaticality judgments, morphological inflection, glossing and translation furnish objectively checkable signals, making reinforcement learning with verifiable rewards (RLVR) a natural avenue for converting declarative linguistic knowledge into reliable procedural skill, alongside supervised fine-tuning and the generation of explicit linguistic reasoning traces (Marmonier et al., 2025; Hu et al., 2026).
- Austin, J., Odena, A., Nye, M., Bosma, M., Henryk Michalewski, Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., & Sutton, C. (2021). Program synthesis with large language models. ArXiv, abs/2108.07732. https://api.semanticscholar.org/CorpusID:237142385
- Automatic Language Processing Advisory Committee. (1966). Language and machines: Computers in translation and linguistics (No. 1416). National Academy of Sciences, National Research Council.
- Aycock, S., Stap, D., Wu, D., Monz, C., & Khalil Sima'an. (2025). Can LLMs really learn to translate a low-resource language from one grammar book? https://arxiv.org/abs/2409.19151
- Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate.
- Barrot, J. S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57, 100745. https://doi.org/10.1016/j.asw.2023.100745
- Beguš, G., Dąbkowski, M., & Rhodes, R. (2025). Large linguistic models: Investigating LLMs’ metalinguistic abilities. IEEE Transactions on Artificial Intelligence, 6, 12. https://doi.org/10.1109/TAI.2025.3575745
- Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2000). A neural probabilistic language model. Journal of Machine Learning Research, Proceedings of the 14th International Conference on Neural Information Processing Systems.
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., & Hesse, C. (2020). Language models are few-shot learners.
- Bryant, C., Yuan, Z., Reza, Q. M., Cao, H., Ng, H. T., & Briscoe, T. (2023). Grammatical error correction: A survey of the state of the art. Computational Linguistics, 49, 3. https://doi.org/10.1162/coli%E2%82%90%E2%82%800478
- Bubeck, S., Chandrasekaran, V., Ronen Eldan, Gehrke, J. A., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y.-F., Lundberg, S. M., Nori, H., Hamid Palangi, Ribeiro, M. T., & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. ArXiv, abs/2303.12712. https://api.semanticscholar.org/CorpusID:257663729
- Chatterji, A., Cunningham, T., Deming, D. J., Hitzig, Z., Ong, C., Shan, C. Y., & Wadman, K. (2025). How people use ChatGPT (No. 34255). National Bureau of Economic Research. https://doi.org/10.3386/w34255
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Henrique, Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., & Ryder, N. (2021). Evaluating large language models trained on code. https://arxiv.org/abs/2107.03374
- Chollet, F. (2019). On the measure of intelligence. ArXiv Preprint ArXiv:1911.01547. https://arxiv.org/abs/1911.01547
- Chollet, F., Knoop, M., Kamradt, G., Landers, B., & Pinkard, H. (2025). ARC-AGI-2: A new challenge for frontier AI reasoning systems. ArXiv Preprint ArXiv:2505.11831. https://arxiv.org/abs/2505.11831
- Cobbe, K., Vineet Kosaraju, Mo Bavarian, Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training verifiers to solve math word problems. ArXiv, abs/2110.14168. https://api.semanticscholar.org/CorpusID:239998651
- Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., Gašević, D., & Chen, G. (2023). Can large language models provide feedback to students? A case study on ChatGPT. 323–325. https://doi.org/10.1109/ICALT58122.2023.00100
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding (J. Burstein, C. Doran, & T. Solorio, Eds.; pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
- Elsner, M., & Liu, D. (2025). Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages (G. Nicolai, E. Chodroff, F. Mailhot, & Ç. Çöltekin, Eds.; pp. 1–14). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.sigmorphon-main.1
- Enkelejda Kasneci, Sessler, K., Küchemann, S., Bannert, M., Daryna Dementieva, Fischer, F., Gasser, U., Groh, G., Günnemann, S., Eyke Hüllermeier, Krusche, S., Gitta Kutyniok, Michaeli, T., Nerdel, C., Pfeffer, J., Oleksandra Poquet, Sailer, M., Schmidt, A., Seidel, T., & Stadler, M. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
- Fleming, S. M., Dolan, R. J., & Frith, C. D. (2012). Metacognition: Computation, biology and function. Philosophical Transactions of the Royal Society B: Biological Sciences, 367, 1594. https://doi.org/10.1098/rstb.2012.0021
- Ford, M. (2018). Architects of intelligence: The truth about AI from the people building it. Packt Publishing.
- Genewein, T., Franklin, M., Lerchner, A., Laurent Orseau, Albanie, S., Bales, A., Wyeth, C., Chan, S. C. Y., Gabriel, I., Leibo, J. Z., Dafoe, A., Hutter, M., Thore Graepel, & Legg, S. (2026). From AGI to ASI. https://api.semanticscholar.org/CorpusID:289187195
- Ginn, M., Hulden, M., & Palmer, A. (2024). Can we teach language models to gloss endangered languages? (Y. Al-Onaizan, M. Bansal, & Y.-N. Chen, Eds.; pp. 5861–5876). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-emnlp.337
- Ginn, M., Moeller, S., Palmer, A., Stacey, A., Nicolai, G., Hulden, M., & Silfverberg, M. (2023). Findings of the SIGMORPHON 2023 shared task on interlinear glossing (G. Nicolai, E. Chodroff, F. Mailhot, & Ç. Çöltekin, Eds.; pp. 186–201). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.sigmorphon-1.20
- Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., & Wu, J. (2023). LEGALBENCH: a collaboratively built benchmark for measuring legal reasoning in large language models.
- Han, Z. (2024). Chatgpt in and for second language acquisition: A call for systematic research. Studies in Second Language Acquisition, 46, 2. https://doi.org/10.1017/S0272263124000111
- Handa, K., Tamkin, A., McCain, M., Huang, S., Durmus, E., Heck, S., Mueller, J., Hong, J., Ritchie, S., Belonax, T., Troy, K. K., Amodei, D., Kaplan, J., Clark, J., & Ganguli, D. (2025). Which economic tasks are performed with AI? Evidence from millions of claude conversations. https://arxiv.org/abs/2503.04761
- Hendrycks, D., Burns, C., Saurav Kadavath, Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. https://arxiv.org/abs/2103.03874
- Hendrycks, D., Song, D., Szegedy, C., Lee, H., Gal, Y., Brynjolfsson, E., Li, S., Zou, A., Levine, L., Han, B., Fu, J., Liu, Z., Shin, J., Lee, K., Mazeika, M., Phan, L., Ingebretsen, G., Khoja, A., Xie, C., & Olawale Salaudeen. (2025). A definition of AGI. ArXiv, abs/2510.18212. https://api.semanticscholar.org/CorpusID:282246026
- Hu, H., Zdeněk Šnajdr, Chen, P., Jannis Vamvas, & Sennrich, R. (2026a). Reinforcement learning elicits contextual learning of unseen language translation. https://arxiv.org/abs/2606.06428
- Hu, H., Zdeněk Šnajdr, Chen, P., Jannis Vamvas, & Sennrich, R. (2026b). Reinforcement learning elicits contextual learning of unseen language translation. https://arxiv.org/abs/2606.06428
- Jin, D., Pan, E., Nassim Oufattole, Weng, W.-H., Fang, H., & Szolovits, P. (2020). What disease does this patient have? A large-scale open domain question answering dataset from medical exams. ArXiv, abs/2009.13081. https://api.semanticscholar.org/CorpusID:221970190
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024). GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382, 2270. https://doi.org/10.2139/ssrn.4389233
- Lee, M., Gero, K. I., John, C., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., August, T., Bhat, A., Choksi, M. Z., Dutta, S., Guo, J. L. C., Hoque, M. N., Kim, Y., Knight, S., Parsa, N. S., & Shibani, A. (2024). A design space for intelligent and interactive writing assistants. https://doi.org/10.1145/3613904.3642697
- Lee, M., Liang, P., & Yang, Q. (2022). CoAuthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. https://doi.org/10.1145/3491102.3502030
- Li, B., Tan, Y. L., Wang, C., & Lowell, V. (2025). Two years of innovation: A systematic review of empirical generative AI research in language learning and teaching. Computers and Education: Artificial Intelligence, 9, 100445. https://doi.org/10.1016/j.caeai.2025.100445
- Lighthill, J. (1973). Artificial intelligence: A general survey (pp. 1–21). Science Research Council.
- Liu, F., Chen, Y., Liu, Y., Jin, Z., Tsai, S., & Zhong, M. (2025). The gold medals in an empty room: Diagnosing metalinguistic reasoning in LLMs with camlang. https://arxiv.org/abs/2509.00425
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl%E2%82%90%E2%82%800638
- Marmonier, M., Bawden, R., & Sagot, B. (2025). Explicit learning and the LLM in machine translation (C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng, Eds.; pp. 31372–31422). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.emnlp-main.1599
- Minsky, M., & Papert, S. (1969). Perceptrons: An introduction to computational geometry. The MIT Press.
- Morris, M. R., Sohl-Dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A., Faust, A., Farabet, C., & Legg, S. (2024). Position: levels of AGI for operationalizing progress on the path to AGI.
- Novikov, A., Ngân V˜u, Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., Sergey Shirobokov, Kozlovskii, B. M., Francisco, Mehrabian, A., Kumar, M. P., See, A., Chaudhuri, S., Holland, G., Davies, A., Nowozin, S., Kohli, P., Balog, M., & Google DeepMind. (2025). AlphaEvolve: A coding agent for scientific and algorithmic discovery. ArXiv, abs/2506.13131. https://api.semanticscholar.org/CorpusID:278658695
- Pei, R., Liu, Y., Sampo Pyysalo, Hinrich Schütze, & Ji, S. (2026). Reasoning over grammar: Can synthetic linguistic reasoning traces enhance low-resource machine translation? https://arxiv.org/abs/2606.03782
- Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations (M. Walker, H. Ji, & A. Stent, Eds.; pp. 2227–2237). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1202
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21, 1.
- Raheja, V., Kumar, D., Koo, R., & Kang, D. (2023). CoEdIT: Text editing by task-specific instruction tuning (H. Bouamor, J. Pino, & K. Bali, Eds.; pp. 5274–5291). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-emnlp.350
- Schneider, N., & Anastasopoulos, A. (2026). Speaking of language: Reflections on metalanguage research in NLP. https://arxiv.org/abs/2604.02645
- Shu, L., Luo, L., Hoskere, J., Zhu, Y., Liu, Y., Tong, S., Chen, J., & Meng, L. (2024). RewriteLM: an instruction-tuned large language model for text rewriting. https://doi.org/10.1609/aaai.v38i17.29863
- Singhal, K., Azizi, S., Tu, T., Mahdavi, S., Wei, J., Chung, H. W., Scales, N., Ajay Kumar Tanwani, Cole-Lewis, H. J., Pfohl, S. J., Payne, P. A., Seneviratne, M. G., Gamble, P., Kelly, C., Nathaneal Scharli, Aakanksha Chowdhery, Mansfield, P. A., Blaise, Webster, D. R., & Corrado, G. S. (2022). Large language models encode clinical knowledge. Nature, 620, 172–180. https://api.semanticscholar.org/CorpusID:255124952
- Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. 3104–3112.
- Suvarna, A., Khandelwal, H., & Peng, N. (2024). PhonologyBench: Evaluating phonological skills of large language models (S. Li, M. Li, M. J. Zhang, E. Choi, M. Geva, P. Hase, & H. Ji, Eds.; pp. 1–14). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.knowllm-1.1
- Tanzer, G., Suzgun, M., Visser, E., Jurafsky, D., & Melas-Kyriazi, L. (2024). A benchmark for learning to translate a new language from one grammar book. https://openreview.net/forum?id=tbVWug9f2h
- Tjaša Arčon, Matej Klemen, Marko Robnik-Šikonja, & Kaja Dobrovoljc. (2026). Evaluating metalinguistic knowledge in large language models across the world’s languages. https://arxiv.org/abs/2602.02182
- Touvron, H., Thibaut Lavril, Gautier Izacard, Martinet, X., Marie-Anne Lachaux, Lacroix, T., Baptiste Rozière, Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and efficient foundation language models. ArXiv, abs/2302.13971. https://api.semanticscholar.org/CorpusID:257219404
- Tunmer, W. E., Pratt, C., & Herriman, M. L. (Eds.). (1984). Metalinguistic awareness in children: Theory, research, and implications (Vol. 15). Springer-Verlag. https://doi.org/10.1007/978-3-642-69113-3
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. 6000–6010.
- Weissweiler, L., Hofmann, V., Kantharuban, A., Cai, A., Dutt, R., Hengle, A., Kabra, A., Kulkarni, A., Vijayakumar, A., Yu, H., Schuetze, H., Oflazer, K., & Mortensen, D. (2023). Counting the bugs in ChatGPT’s wugs: A multilingual investigation into the morphological capabilities of a large language model (H. Bouamor, J. Pino, & K. Bali, Eds.; pp. 6508–6524). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.401
- Yehoshua Bar-Hillel. (1960). The present status of automatic translation of languages. Advances in Computers, 1, 91–163. https://api.semanticscholar.org/CorpusID:18822284
