FC Layer

Definition

Fully Connected Layer is just a traditional linear classifier. There are N vectors(weights) and N scalars(bias) in such layer, and the dimension of the vector is determined by the input vector dimension. When an input vector is given, we calculate the dot product of it with N weights and then add N biases respectively to generate N scores. Each score corresponds to roughly how similar is the given vector to its corresponding template vector(weight).

Input:

Layer:

Output

Functionality

A single FC Layer is an universal approximator. This means FC Layer can  express any given function with proper tunning. Yet this statement carries no weight in itself because it is impractical to train such layer. Only when we stack together several such layers, the network becomes possible to optimize. That said, although a single FC layer has the same representational power as a multi-layer deeper network, empirically the multi-layer network works much better. s

 

Use Cases

FC Layer is the primary component in Neural Network. Notice that a Neural Net with 3 FC layers often outperform one with 2 FC layers, yet going deeper that 3 layers rarely helps. This stands in stark contrast to Deep Convolution Network where depth plays a much bigger role in performance.

FC Layer is used  mostly in the last several layers in ConvNet to generate scores.

 

Caveats

FC Layer is very computationally expensive. Use FC layer only if there is a necessity to connect each neuron to all input vectors.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.