Camron Godbout

To begin we should clearly define the Curse of Dimensionality:

A phenomena that occurs when the dimensionality of the data increases, the sparsity of the data increases.

Data has dimensionality to it. The more dimensions that are added to the data, the more difficult it becomes to find patterns. Think about dimensionality as the range of movement of an animal you’re playing tag with. If you’re chasing an animal that can only move on the ground they can only go in 2 dimensions left or right(x), forward or backward(y). It’s harder to catch a bird because that bird can move in 3 dimensions left or right (x), forward or backward (y), up or down (z). We can imagine some mythical time traveling beast that can move in 4 dimensions left or right(x), forward or backward(y), up or down(z), past or future(t). You can see this gets more difficult as the dimensions increase.

This same problem applies with data and machine learning. As the data’s dimensionality increases the sparsity of the data increases making it harder to ascertain a pattern. There are certain ways around the curse of dimensionality in traditional ML that require certain techniques such as function smoothing and approximation. Deep Learning has overcome this “curse” by it’s inherent nature and may be one of the contributions to the increased popularity.