AI Architecture 2. The Cost of Activation: Free ReLU vs. Expensive Sigmoid
In the previous post, we examined how expensive the MAC (Multiply-Accumulate) operation—the core of artificial neurons—is in terms of hardware, specifically regarding Multiplier area and Memory Bandwidth.