Neural language models are probabilistic models of text parameterized by neural networks. They are widely applicable to applications with outputs consisting of discrete sequences, such as document summarization, question answering, and image captioning. The minimal assumptions about data enable advancements in language modeling to drive improvements across a diverse array of applications. In natural language, structures are both pervasive and essential. For example, a book

is organized into chapters, with a logical flow connecting them; without this structure, the book would lose its coherence. Therefore, effectively understanding and modeling textual sequences requires comprehending and representing the inherent structures. This thesis focuses on structure modeling for language models.