This blog post looks at the growth of computation, data, deep learning researcher demographics to show that the field of deep learning could stagnate over slowing growth. We will look at recent deep learning research papers which strike up similar problems but also demonstrate how one could to solve these problems. After discussion of these papers, I conclude with promising research directions which face these challenges head on.
This blog post series discusses long-term research directions and takes a critical look at short-term thinking and its pitfalls. In this first blog post in this series, I firstly will discuss long-term trends of data and computational power by using trends in computing and hardware. Then we look at the demographics of researchers, and I show that the fraction of researchers that do not have access to powerful computational resources is increasing rapidly. Then we have a look at the core paper of this blog post “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era” which reveals that more data can improve predictive performance but it comes with a rather heavy computational burden. We will also see that compared to specialized techniques, pre-training on more data is just on-a-par with respect to predictive performance.