When building neural networks, one of the first questions you should ask is:

How many trainable parameters does my model have?

The number of parameters determines:

• Model complexity
• Memory usage
• Training speed
• Risk of overfitting

In this article, we'll learn how to calculate the number of parameters manually and verify the results using TensorFlow 2.x (Keras).

What Are Parameters?

Parameters are the values that the neural network learns during training.

There are two types:

• Weights
• Biases

Every neuron has:

• One weight for every input it receives
• One bias

Therefore, for a layer with:

• Input features = n
• Neurons = h

Example 1: Single Hidden Layer

Suppose we have:

• Input features = 4
• Hidden neurons = 5
• Output neurons = 1

Architecture:

Input(4)
      │
Hidden(5)
      │
Output(1)

Hidden Layer

Each of the 5 neurons receives 4 inputs.

Weights:

4 × 5 = 20

Biases:

5

Total:

20 + 5 = 25

Output Layer

Input = 5

Output neurons = 1

Weights:

5 × 1 = 5

Biases:

1

Total:

5 + 1 = 6

Total Parameters

25 + 6 = 31

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(4,)),
    tf.keras.layers.Dense(5, activation="relu"),
    tf.keras.layers.Dense(1)
])

model.summary()

Output:

Layer (type)      Output Shape     Param #

dense             (None, 5)          25
dense_1           (None, 1)           6

Total params: 31

Perfect match.

Example 2: Two Hidden Layers

Architecture:

Input(8)
      │
Hidden(16)
      │
Hidden(10)
      │
Output(3)

First Hidden Layer

Input = 8

Neurons = 16

Weights = 8 × 16 = 128
Biases = 16

Total = 144

Second Hidden Layer

Input = 16

Neurons = 10

Weights = 16 × 10 = 160
Biases = 10

Total = 170

Output Layer

Input = 10

Output neurons = 3

Weights = 10 × 3 = 30
Biases = 3

Total = 33

Total Parameters

144 + 170 + 33 = 347

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(8,)),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(10, activation="relu"),
    tf.keras.layers.Dense(3, activation="softmax")
])

model.summary()

Output:

Layer (type)      Param #

dense                 144
dense_1               170
dense_2                33

Total params: 347

Again, the manual calculation matches TensorFlow exactly.

Example 3: Deep Neural Network

Architecture:

Input(20)
      │
Hidden(64)
      │
Hidden(32)
      │
Hidden(16)
      │
Output(5)

Layer 1

(20 × 64) + 64
= 1280 + 64
= 1344

Layer 2

(64 × 32) + 32
= 2048 + 32
= 2080

Layer 3

(32 × 16) + 16
= 512 + 16
= 528

Output Layer

(16 × 5) + 5
= 80 + 5
= 85

Total Parameters

1344
+2080
+528
+85
------
4037

TensorFlow Verification

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(20,)),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(5, activation="softmax")
])

model.summary()

Expected output:

Layer (type)      Param #

dense              1344
dense_1            2080
dense_2             528
dense_3              85

Total params: 4037

Why Don't Activation Functions Add Parameters?

Layers such as:

Dense(32, activation="relu")

or

Dense(10, activation="sigmoid")

have exactly the same number of parameters.

Activation functions like:

• ReLU
• Sigmoid
• Tanh
• Softmax

perform mathematical operations but do not learn any weights or biases, so they contribute zero trainable parameters.

Quick Reference

Key Takeaways

• Every Dense layer learns weights and biases.

• The parameter count for a Dense layer is:
  

  Parameters = (Input Units × Output Units) + Output Units

• Equivalently:

  Parameters = (Input Units + 1) × Output Units

• The output size of one layer becomes the input size of the next layer.

• The total number of trainable parameters is the sum of the parameters across all trainable layers.

• You can always verify your manual calculations using model.summary() in TensorFlow 2.x with Keras.