Fully Connected Layer is just a traditional linear classifier. There are N vectors(weights) and N scalars(bias) in such layer, and the dimension of the vector is determined by the input vector dimension. When an input vector is given, we calculate the dot product of it with N weights and then add N biases respectively to generate N scores. Each score corresponds to roughly how similar is the given vector to its corresponding template vector(weight).
A single FC Layer is an universal approximator. This means FC Layer can express any given function with proper tunning. Yet this statement carries no weight in itself because it is impractical to train such layer. Only when we stack together several such layers, the network becomes possible to optimize. That said, although a single FC layer has the same representational power as a multi-layer deeper network, empirically the multi-layer network works much better. s
FC Layer is the primary component in Neural Network. Notice that a Neural Net with 3 FC layers often outperform one with 2 FC layers, yet going deeper that 3 layers rarely helps. This stands in stark contrast to Deep Convolution Network where depth plays a much bigger role in performance.
FC Layer is used mostly in the last several layers in ConvNet to generate scores.
FC Layer is very computationally expensive. Use FC layer only if there is a necessity to connect each neuron to all input vectors.