{"id":1248,"date":"2026-01-05T09:32:00","date_gmt":"2026-01-05T00:32:00","guid":{"rendered":"https:\/\/rtlearner.com\/?p=1248"},"modified":"2026-01-08T15:20:41","modified_gmt":"2026-01-08T06:20:41","slug":"ai-architecture-1-neuron-hardware-mac-analysis","status":"publish","type":"post","link":"https:\/\/rtlearner.com\/en\/ai-architecture-1-neuron-hardware-mac-analysis\/","title":{"rendered":"AI Architecture 1. Anatomy of an Artificial Neuron: Y=WX+B on Silicon"},"content":{"rendered":"<p class=\"wp-block-paragraph translation-block\">When starting with deep learning, the first concept we encounter is the Perceptron, or the artificial neuron. For software engineers using frameworks like PyTorch or TensorFlow, a neuron is often abstracted away as a simple matrix operation handled by a library.<\/p>\n\n\n\n<div class=\"wp-block-math\"><math display=\"block\"><semantics><mrow><mi>Y<\/mi><mo>=<\/mo><mo movablelimits=\"false\">\u2211<\/mo><mo form=\"prefix\" stretchy=\"false\">(<\/mo><mi>W<\/mi><mo>\u00d7<\/mo><mi>X<\/mi><mo form=\"postfix\" stretchy=\"false\">)<\/mo><mo>+<\/mo><mi>B<\/mi><\/mrow><annotation encoding=\"application\/x-tex\">Y = \\sum (W \\times X) + B<\/annotation><\/semantics><\/math><\/div>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">However, for Hardware Engineers (Digital Logic Designers \/ Architects) like us, this equation carries a completely different meaning. The moment this equation on a monitor translates into physical transistors and circuits on a Silicon Wafer, it directly translates into real-world constraints: 'Cost', 'Heat', and 'Area'.<\/p>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">In this article, we will dissect the physical reality of an artificial neuron, which forms the fundamental building block of NPU (Neural Processing Unit) design.<\/p>\n\n\n<style>.kb-table-of-content-nav.kb-table-of-content-id1248_d432c6-a3 .kb-table-of-content-wrap{padding-top:var(--global-kb-spacing-sm, 1.5rem);padding-right:var(--global-kb-spacing-sm, 1.5rem);padding-bottom:var(--global-kb-spacing-sm, 1.5rem);padding-left:var(--global-kb-spacing-sm, 1.5rem);box-shadow:0px 0px 14px 0px rgba(0, 0, 0, 0.2);}.kb-table-of-content-nav.kb-table-of-content-id1248_d432c6-a3 .kb-table-of-contents-title-wrap{padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;}.kb-table-of-content-nav.kb-table-of-content-id1248_d432c6-a3 .kb-table-of-contents-title{font-weight:regular;font-style:normal;}.kb-table-of-content-nav.kb-table-of-content-id1248_d432c6-a3 .kb-table-of-content-wrap .kb-table-of-content-list{font-weight:regular;font-style:normal;margin-top:var(--global-kb-spacing-sm, 1.5rem);margin-right:0px;margin-bottom:0px;margin-left:0px;}@media all and (max-width: 767px){.kb-table-of-content-nav.kb-table-of-content-id1248_d432c6-a3 .kb-table-of-contents-title{font-size:var(--global-kb-font-size-md, 1.25rem);}.kb-table-of-content-nav.kb-table-of-content-id1248_d432c6-a3 .kb-table-of-content-wrap .kb-table-of-content-list{font-size:var(--global-kb-font-size-sm, 0.9rem);}}<\/style>\n\n<style>.kadence-column1248_0701d9-5f > .kt-inside-inner-col{box-shadow:0px 0px 14px 0px rgba(0, 0, 0, 0.2);}.kadence-column1248_0701d9-5f > .kt-inside-inner-col,.kadence-column1248_0701d9-5f > .kt-inside-inner-col:before{border-top-left-radius:0px;border-top-right-radius:0px;border-bottom-right-radius:0px;border-bottom-left-radius:0px;}.kadence-column1248_0701d9-5f > .kt-inside-inner-col{column-gap:var(--global-kb-gap-sm, 1rem);}.kadence-column1248_0701d9-5f > .kt-inside-inner-col{flex-direction:column;}.kadence-column1248_0701d9-5f > .kt-inside-inner-col > .aligncenter{width:100%;}.kadence-column1248_0701d9-5f > .kt-inside-inner-col:before{opacity:0.3;}.kadence-column1248_0701d9-5f{position:relative;}@media all and (max-width: 1024px){.kadence-column1248_0701d9-5f > .kt-inside-inner-col{flex-direction:column;justify-content:center;}}@media all and (max-width: 767px){.kadence-column1248_0701d9-5f > .kt-inside-inner-col{flex-direction:column;justify-content:center;}}<\/style>\n<div class=\"wp-block-kadence-column kadence-column1248_0701d9-5f\"><div class=\"kt-inside-inner-col\">\n<p class=\"wp-block-paragraph\"><strong>Related articles<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705<a href=\"https:\/\/rtlearner.com\/en\/ai-architecture-2-activation-relu-vs-sigmoid\/\" data-type=\"post\" data-id=\"1255\">AI Architecture 2. The Cost of Activation: Free ReLU vs. Expensive Sigmoid<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705<a href=\"https:\/\/rtlearner.com\/en\/ai-architecture-3-matmul-simd-parallel-processing\/\" data-type=\"post\" data-id=\"1263\">AI Architecture 3. The Aesthetics of MatMul: Why Deep Learning Chooses GPUs\/NPUs<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705<a href=\"https:\/\/rtlearner.com\/en\/ai-architecture-4-training-vs-inference\/\" data-type=\"post\" data-id=\"1267\">AI Architecture 4. Training vs. Inference<\/a><\/p>\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>1. Not Just Logic Gates, But Massive Arithmetic Units<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In undergraduate digital logic classes, we learn about basic gates like AND, OR, and NOT. But when designing an AI accelerator, the primary building block we face is much larger.<\/p>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">The core operation of a perceptron is the Dot Product. It involves multiplying hundreds or thousands of input signals by their respective weights and summing them all up. In the hardware world, this is known as the MAC (Multiply-Accumulate) operation.<\/p>\n\n\n\n<div class=\"wp-block-math\"><math display=\"block\"><semantics><mrow><mtext>MAC&nbsp;Operation:&nbsp;<\/mtext><mspace width=\"1em\"><\/mspace><mtext>Accumulator<\/mtext><mo stretchy=\"false\">\u2190<\/mo><mtext>Accumulator<\/mtext><mo>+<\/mo><mo form=\"prefix\" stretchy=\"false\">(<\/mo><mi>A<\/mi><mo>\u00d7<\/mo><mi>B<\/mi><mo form=\"postfix\" stretchy=\"false\">)<\/mo><\/mrow><annotation encoding=\"application\/x-tex\">\\text{MAC Operation: } \\quad \\text{Accumulator} \\leftarrow \\text{Accumulator} + (A \\times B)<\/annotation><\/semantics><\/math><\/div>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">In software, it\u2019s just a simple line of code like out += a * b. But in hardware, this operation dictates the destiny of the chip. The performance metric of an NPU, TOPS (Tera Operations Per Second), is essentially synonymous with \"How many times can we cycle this MAC unit per second?\"<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>2. The Multiplier: An Area Monster<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">From a hardware architecture perspective, Adders and Multipliers are in completely different weight classes. Understanding this distinction is the first step in NPU design.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Adder:<\/strong> An N-bit addition is relatively simple. Whether using Ripple Carry or Carry Look-ahead for speed, the logic depth is shallow. It occupies a small area and consumes modest power.<\/li>\n\n\n\n<li><strong>Multiplier:<\/strong> Multiplication is a different beast entirely. Binary multiplication involves generating numerous Partial Products and then hierarchically summing them (using structures like Wallace Trees).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If the complexity of an adder is close to O(N), the complexity of a multiplier scales closer to O(N<sup>2<\/sup>). For instance, implementing a single 32-bit Floating Point (FP32) multiplier on a chip requires dozens of times more silicon area (Gate Count) than a 32-bit adder.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Architect&#8217;s Insight:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Many ask, \"Why are NPU architects so obsessed with INT8 or FP16 instead of FP32?\" The answer is simple: The size of the multiplier. Halving the data bit-width reduces the multiplier's area to roughly 1\/4. This means we can fit 4 times more processing units on the same chip area, theoretically quadrupling performance. This is the core principle of hardware acceleration.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>3. The Hardware Nightmare of Floating Point<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Artificial neuron equations typically deal with real numbers, making floating-point arithmetic the standard. However, for hardware engineers, Floating Point is a feature we try to avoid whenever possible.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Unlike simple Integer multiplication, Floating Point operations ($A \\times B$) involve a complex sequence of steps:<\/p>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li><strong>Exponent Addition:<\/strong> Adding the exponent parts of the two numbers.<\/li>\n\n\n\n<li><strong>Mantissa Multiplication:<\/strong> Multiplying the significant digits (integers).<\/li>\n\n\n\n<li><strong>Normalization:<\/strong> Shifting bits to conform the result to the standard format (1.xxx).<\/li>\n\n\n\n<li><strong>Rounding:<\/strong> Adjusting the least significant bits to match precision.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">Each of these steps requires Barrel Shifters and complex control logic. As deep learning models grow larger, we are forced to rely on Quantization techniques to replace these expensive FP32 operations with cheaper integer arithmetic like INT8.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>4. Y=WX+B: Power Consumption<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Among the three pillars of semiconductor design\u2014PPA (Power, Performance, Area)\u2014Power has become the most critical constraint in recent years. The dynamic power consumption of a CMOS circuit is described by the formula:<\/p>\n\n\n\n<div class=\"wp-block-math\"><math display=\"block\"><semantics><mrow><mi>P<\/mi><mo>=<\/mo><mi>\u03b1<\/mi><mo>\u22c5<\/mo><mi>C<\/mi><mo>\u22c5<\/mo><msup><mi>V<\/mi><mn>2<\/mn><\/msup><mo>\u22c5<\/mo><mi>f<\/mi><\/mrow><annotation encoding=\"application\/x-tex\">P = \\alpha \\cdot C \\cdot V^2 \\cdot f<\/annotation><\/semantics><\/math><\/div>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u03b1: Activity Factor (how often bits flip between 0 and 1)<\/li>\n\n\n\n<li>C: Capacitance (proportional to circuit size)<\/li>\n\n\n\n<li>V: Voltage<\/li>\n\n\n\n<li>f: Frequency<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">When a massive multiplier circuit (large C) operates at high speed (high f), P increases exponentially. The reason your smartphone heats up or data center cooling costs skyrocket is that tens of thousands of artificial neurons (MAC units) are switching simultaneously, burning energy.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>5. Data movement<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">Finally, I must share the most crucial truth. When calculating Y=WX+B, the culprit consuming the most energy is not 'Compute'. The real culprit is 'Data Movement'.<\/p>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">According to famous research presented by Professor Mark Horowitz at Stanford, in a 7nm process, while a 32-bit addition costs about 0.1 pJ (picoJoule), fetching data from DRAM costs about 640 pJ.<\/p>\n\n\n<style>.kb-image1248_d8384d-f5.kb-image-is-ratio-size, .kb-image1248_d8384d-f5 .kb-image-is-ratio-size{max-width:390px;width:100%;}.wp-block-kadence-column > .kt-inside-inner-col > .kb-image1248_d8384d-f5.kb-image-is-ratio-size, .wp-block-kadence-column > .kt-inside-inner-col > .kb-image1248_d8384d-f5 .kb-image-is-ratio-size{align-self:unset;}.kb-image1248_d8384d-f5 figure{max-width:390px;}.kb-image1248_d8384d-f5 .image-is-svg, .kb-image1248_d8384d-f5 .image-is-svg img{width:100%;}.kb-image1248_d8384d-f5 .kb-image-has-overlay:after{opacity:0.3;}<\/style>\n<div class=\"wp-block-kadence-image kb-image1248_d8384d-f5\"><figure class=\"aligncenter size-full\"><img data-dominant-color=\"bac7e4\" data-has-transparency=\"false\" style=\"--dominant-color: #bac7e4;\" loading=\"lazy\" decoding=\"async\" width=\"408\" height=\"329\" src=\"https:\/\/rtlearner.com\/wp-content\/uploads\/2026\/01\/image.jpg\" alt=\"\" class=\"kb-img wp-image-1253 not-transparent\" srcset=\"https:\/\/rtlearner.com\/wp-content\/uploads\/2026\/01\/image.jpg 408w, https:\/\/rtlearner.com\/wp-content\/uploads\/2026\/01\/image-300x242.jpg 300w, https:\/\/rtlearner.com\/wp-content\/uploads\/2026\/01\/image-15x12.jpg 15w\" sizes=\"auto, (max-width: 408px) 100vw, 408px\" \/><figcaption>Mark Horowitz energy table<\/figcaption><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">From a hardware perspective, fetching weights (W) from memory and hauling them to the MAC unit is a massive waste. This is known as the Memory Wall problem. It is no exaggeration to say that all modern AI semiconductor innovations (HBM, large caches, Dataflow architectures) exist solely to reduce this energy consumption.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>6. Conclusion: There is No 'Free Lunch' in Hardware<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph translation-block\">In that brief moment when a developer calls model.forward(x) in Python, thousands of multipliers inside the NPU are fighting for space, consuming massive power, and data is screaming through bottlenecks in the memory bus.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As AI hardware engineer, our goal is clear:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\"How can we minimize unnecessary floating-point operations (Quantization) and how can we fetch less data from memory (Data Reuse)?\"<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To find the answer, in the next post, we will analyze the \"Cost of Activation Functions.\" Why do hardware engineers loathe Sigmoid and love ReLU? We will uncover that secret.<\/p>\n\n\n<style>.kadence-column1248_e65046-3f > .kt-inside-inner-col{box-shadow:0px 0px 14px 0px rgba(0, 0, 0, 0.2);}.kadence-column1248_e65046-3f > .kt-inside-inner-col,.kadence-column1248_e65046-3f > .kt-inside-inner-col:before{border-top-left-radius:0px;border-top-right-radius:0px;border-bottom-right-radius:0px;border-bottom-left-radius:0px;}.kadence-column1248_e65046-3f > .kt-inside-inner-col{column-gap:var(--global-kb-gap-sm, 1rem);}.kadence-column1248_e65046-3f > .kt-inside-inner-col{flex-direction:column;}.kadence-column1248_e65046-3f > .kt-inside-inner-col > .aligncenter{width:100%;}.kadence-column1248_e65046-3f > .kt-inside-inner-col:before{opacity:0.3;}.kadence-column1248_e65046-3f{position:relative;}@media all and (max-width: 1024px){.kadence-column1248_e65046-3f > .kt-inside-inner-col{flex-direction:column;justify-content:center;}}@media all and (max-width: 767px){.kadence-column1248_e65046-3f > .kt-inside-inner-col{flex-direction:column;justify-content:center;}}<\/style>\n<div class=\"wp-block-kadence-column kadence-column1248_e65046-3f\"><div class=\"kt-inside-inner-col\">\n<p class=\"wp-block-paragraph\"><strong>Related articles<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705<a href=\"https:\/\/rtlearner.com\/en\/ai-architecture-2-activation-relu-vs-sigmoid\/\" data-type=\"post\" data-id=\"1255\">AI Architecture 2. The Cost of Activation: Free ReLU vs. Expensive Sigmoid<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705<a href=\"https:\/\/rtlearner.com\/en\/ai-architecture-3-matmul-simd-parallel-processing\/\" data-type=\"post\" data-id=\"1263\">AI Architecture 3. The Aesthetics of MatMul: Why Deep Learning Chooses GPUs\/NPUs<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705<a href=\"https:\/\/rtlearner.com\/en\/ai-architecture-4-training-vs-inference\/\" data-type=\"post\" data-id=\"1267\">AI Architecture 4. Training vs. Inference<\/a><\/p>\n<\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">References: <em><a href=\"https:\/\/ieeexplore.ieee.org\/document\/6757323\" target=\"_blank\" rel=\"noopener\">Computing&#8217;s Energy Problem<\/a><\/em><\/p>","protected":false},"excerpt":{"rendered":"<p>When starting with deep learning, the first concept we encounter is the Perceptron,<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_kadence_starter_templates_imported_post":false,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[116],"tags":[117,118],"class_list":["post-1248","post","type-post","status-publish","format-standard","hentry","category-ai-and-hw-fundamentals","tag-ai","tag-architecture"],"_links":{"self":[{"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/posts\/1248","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/comments?post=1248"}],"version-history":[{"count":8,"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/posts\/1248\/revisions"}],"predecessor-version":[{"id":1328,"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/posts\/1248\/revisions\/1328"}],"wp:attachment":[{"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/media?parent=1248"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/categories?post=1248"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rtlearner.com\/en\/wp-json\/wp\/v2\/tags?post=1248"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<!-- This website is optimized by Airlift. Learn more: https://airlift.net. Template:. Learn more: https://airlift.net. Template: 6a588749d36f73a90a601ae0. Config Timestamp: 2026-07-16 07:24:54 UTC, Cached Timestamp: 2026-08-15 14:15:31 UTC, Optimization Time: 2.76ms -->