Advancements in natural language processing (NLP) have led to the creation of sophisticated machine learning models, culminating in the development of the transformer architecture. Here three components have been critical for ensuring potent models: robust architectural design, well-curated large datasets, and judiciously selected pretraining approaches. Despite these strides in NLP, the field of property prediction for inorganic materials has yet to fully incorporate these advancements. Recent literature has proposed various methodologies to harness the power of deep learning techniques. However, most structure-agnostic approaches presented so far fall short of learning information-rich representations that can be leveraged across a variety of downstream property predictions. Such representations could play crucial roles in synthetic precursor and dopant suggestion, tasks of significant value to the community as the domain is only poorly theoretically portrayed. By harnessing the power of the three previously introduced components, this talk shows how to overcome the pervasive challenge of data fragmentation manifested in terms of disparate material properties in isolated datasets. This key challenge is addressed through the incorporation of diverse data into the property prediction models. Evaluating different strategies on the Matbench task suite unveils the capability of significantly outperforming current state-of-the-art approaches like ElemNet and CrabNet. Moreover, correlations between pretraining and final task datasets illustrate successful knowledge transfer. In addition to materials property prediction, the discussion will extend to the potential of utilizing trained models for the generation of comprehensive, structure-aware representations for inorganic materials.