High-Dimensional Visualization

This refers to visualization of functions that exist in a high-dimensional space. One most important example is Loss Function, which has an enormous number of parameters. Visualization on such function can provide insight on how we should optimize most efficiently.

One possible way is to visualize along a random plane. We can first generate a random point $W$ in space and two orthogonal directions $W_1$ and $W_2$. In a Loss Function, such $W$ would correspond to a complete set of model parameters. Then we plot out different loss value corresponding to different $W + aW_1 + bW_2$ value on a plane by using different colors. By doing so we gain some intuition about how the Loss Function looks like in space.

SVM Loss Function along a plane. Left image plots loss from a single example and shows a piece-wise linear structure, whereas the right image shows average loss of a hundred examples and demonstrates a bowl-shape structure.










Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.