Abstract: Self-supervised learning (SSL) aims to eliminate one of the major bottlenecks in representation learning - the need for human annotations. As a result, SSL holds the promise to learn representations from data in the wild, i.e., without the need for curated and static datasets. However, can current self-supervised learning approaches be effective in this setup? In this talk, I will show that the answer is no. While learning in the wild, we expect to see a continuous stream of potentially non-IID data. Yet, state-of-the-art approaches struggle to learn from such data distributions. They are inefficient (both computationally and in terms of data complexity), exhibit signs of forgetting, incapable of modeling dynamics, and result in inferior representation quality. The talk will introduce our recent efforts in tackling these issues.