Understanding the computational cost of your convolutional neural networks (CNNs) is crucial for efficient model design and deployment. A key metric for this is the number of Multiply-Accumulate operations (MACs), which represents the total number of multiplications and additions performed during a forward pass. This post will guide you through calculating the number of MACs in a convolutional layer.
Understanding the Convolution Operation
Before diving into the calculations, let's briefly review the convolution operation. A convolutional layer uses filters (kernels) to slide across the input feature maps, performing element-wise multiplications and summing the results. This process generates an output feature map.
Key parameters involved in the convolution operation include:
- Input Feature Map Dimensions:
Hin x Win x Cin(Height, Width, Channels) - Filter (Kernel) Dimensions:
K x K x Cin(Kernel size, Kernel size, Input channels) Note: We assume square kernels for simplicity; the calculation can be easily adapted for rectangular kernels. - Output Feature Map Dimensions:
Hout x Wout x Cout(Height, Width, Output channels) - Stride:
S(The step size the filter moves across the input) - Padding:
P(Number of pixels added to the borders of the input)
Calculating the Number of MACs
The formula to calculate the number of MACs in a single convolutional layer is:
MACs = Hout * Wout * Cout * (K * K * Cin + 1)
Let's break down each component:
-
Hout * Wout * Cout: This represents the total number of output elements. Each element in the output feature map requires a series of multiplications and additions. -
K * K * Cin: This is the number of multiplications performed for each output element. It corresponds to the number of elements in the kernel multiplied by the number of input channels. -
+ 1: This accounts for the addition operation performed after all the multiplications for each output element. The summation of the results of the element-wise multiplications.
Example Calculation
Let's consider a convolutional layer with the following parameters:
- Input: 28 x 28 x 3 (e.g., an image)
- Filter: 3 x 3 x 3 (Kernel size 3x3, 3 input channels)
- Output: 26 x 26 x 64 (64 output channels)
- Stride: 1
- Padding: 0
Using the formula:
Hout = 26, Wout = 26, Cout = 64, K = 3, Cin = 3
MACs = 26 * 26 * 64 * (3 * 3 * 3 + 1) = 26 * 26 * 64 * (27 + 1) = 26 * 26 * 64 * 28 = 1,223,168 MACs
Therefore, this convolutional layer performs approximately 1,223,168 multiply-accumulate operations.
Considerations and Extensions
-
Multiple Layers: To determine the total MACs for an entire CNN, you need to sum the MACs for each convolutional layer.
-
Different Kernel Sizes and Stride: The formula remains the same, just adjust the values of
KandSaccordingly. Remember that stride and padding affect theHoutandWoutcalculations. Adjust these values based on your chosen stride and padding. -
Bias: Often, a bias term is added to each output element. This adds an extra
Hout * Wout * Coutadditions to the total computation. While often negligible, it might be worth accounting for in high-precision MAC calculations. -
Batch Size: The above calculation is for a single input sample. Multiply the result by the batch size to get the total MACs for a batch of inputs.
By understanding and applying this formula, you gain valuable insight into the computational complexity of your CNN models, enabling you to make informed decisions about model architecture, optimization techniques, and hardware choices. Remember to account for all layers and factors for a comprehensive analysis.
