Mastering TensorFlow “Variables” in 5 Easy Steps

Learn how to use TensorFlow Variables, their differences from plain Tensor objects, and when they are preferred over these Tensor objects | Deep Learning with TensorFlow 2.x

WARNING: Do not confuse this article with “Mastering TensorFlow Tensors in 5 Easy Steps”!

If you are reading this article, I am sure that we share similar interests and are/will be in similar industries. So let’s connect via Linkedin! Please do not hesitate to send a contact request! Orhan G. Yalçın — Linkedin

 

In this tutorial, we will focus on TensorFlow Variables. After the tutorial, you will be able to create, update, and manage TensorFlow Variables effectively. As usual, our tutorial will deliver code examples with detailed explanations as well as conceptual explanations. We will master TensorFlow Variables in 5 easy steps:

• Step 1: Definition of Variables →A Brief Introduction, Comparison with Tensors
• Step 2: Creation of Variables → Instantiating tf.Variable Objects
• Step 3: Qualifications of Variables → Characteristics and Features
• Step 4: Operations with Variables → Basic Tensor Operations, Indexing, Shape Manipulation, and Broadcasting
• Step 5: Hardware Selection for Variables → GPUs, CPUs, TPUs

Fasten your belts, and let’s start!

Definition of Variables

In this step, we will briefly cover what Variables are and understand the difference between plain Tensor objects and Variable objects.

A Brief Introduction

A TensorFlow Variable is the preferred object type representing a shared and persistent state that you can manipulate with any operation, including TensorFlow models. Manipulation refers to any value or parameter update. This characteristic is the most distinguishing feature of Variables compared to tf.Tensor objects. TensorFlow Variables are recorded as tf.Variable objects. Let’s make a brief comparison between tf.Tensor and tf.Variable objects to understand their similarities and differences.

 Figure 2. Variable Values can be Updated (Figure by Author)

Comparison with Tensors

So, the most important difference between Variables and Tensors is mutability. The values in a Variable object can be updated (e.g., with the assign() function) as opposed to Tensors.

“The values of tensor objects cannot be updated, and you can only create a new Tensor object with the new values.”

Variable objects are mainly used to store model parameters, and since these values are constantly updated during training, using Variables, instead of Tensors, is a necessity rather than a choice.

The shape of a Variable object can be updated with the reshape() instance function just like the shape of a Tensor object. Since Variable objects are built on top of Tensor objects, they have common attributes such as .shape and .dtype. But, Variables also have unique attributes such as .trainable,.device, and .name attributes that the Tensors do not have.

 Figure 3. A Tensorflow Variable is actually a wrapper around a TensorFlow Tensor with additional features (Figure by Author)

Let’s see how we can create tf.Variable objects!

Creation of Variables

We can instantiate (i.e., create) tf.Variableobjects with the tf.Variable() function. The tf.Variable() function accepts different data types as parameter such as integers, floats, strings, lists, and tf.Constant objects.

Before showing different Variable object examples with these different data types, I want you to start a new Google Colab notebook and import TensorFlow library with the following code:

Now, we can start creating tf.Variable objects.

1 — We can pass a tf.constant() object as the initial_value:

2 — We can pass a single integer as the initial_value:

3 — We can pass a list of integers or floats as the initial_value:

4— We can pass a single string as the initial_value:

5— We can pass a list of strings as the initial_value:

As you can see, there are several data types that th etf.Variable() function accepts as the initial_value argument. Now let’s take a look at the characteristics and features of variables.

Qualifications of Variables

Every Variable must have some properties such as value, name, uniform data type, shape, rank, size, and more. In this section, we will see what these properties are and how we can view these properties in a Colab notebook.

Value

Every Variable must specify an initial_value. Otherwise, TensorFlow raises an error and says that Value Error: initial_value must be specified. Therefore, make sure that you pass on an initial_valueargument when creating Variable objects. To be able to view a Variable’s values, we can use the .value() function as well as the .numpy() function. See the example below:

Output:
The values stored in the variables:
tf.Tensor( [[1. 2.]  
            [1. 2.]], shape=(2, 2), dtype=float32)

The values stored in the variables:
[[1. 2.]
[1. 2.]]

Name

Name is a Variable attribute which helps developers to track the updates on a particular variable. You can pass a name argument while creating the Variable object. If you don’t specify a name, TensorFlow assigns a default name, as shown below:

Output:
The name of the variable:  Variable:0

Dtype

Each Variable must have a uniform data type that it stores. Since there is a single type of data stored for every Variable, you can also view this type with the .dtype attribute. See the example below:

Output:
The selected datatype for the variable:  <dtype: 'float32'>

Shape, Rank, and Size

The shape property shows the size of each dimension in the form of a list. We can view the shape of the Variable object with the .shape attribute. Then, we can view the number of dimensions that a Variable object has with the tf.size() function. Finally, Size corresponds to the total number of elements a Variable has. We need to use the tf.size() function to count the number of elements in a Variable. See the code below for all three properties:

Output:
The shape of the variable:  (2, 2)
The number of dimensions in the variable: 2
The number of dimensions in the variable: 4

Operations with Variables

There are several basic operations you can easily conduct with math operators and TensorFlow functions. On top of what we covered in Part 2 of this tutorial series, you may also use the following math operators for Variable operations.

Basic Tensor Operations

 Figure 4. You May Benefit from Basic Math Operators (Figure by Author)

• Addition and Subtraction: We can conduct addition and subtraction with + and — signs.

Addition by 2:
tf.Tensor( [[3. 4.]  [3. 4.]], shape=(2, 2), dtype=float32)

Substraction by 2:
tf.Tensor( [[-1.  0.]  [-1.  0.]], shape=(2, 2), dtype=float32)

• Multiplication and Division: We can conduct multiplication and division with * and / signs.

Multiplication by 2:
tf.Tensor( [[2. 4.]  [2. 4.]], shape=(2, 2), dtype=float32)

Division by 2:
tf.Tensor( [[0.5 1. ]  [0.5 1. ]], shape=(2, 2), dtype=float32)

• Matmul and Modulo Operations: Finally, you can also do matmul and modulo operations with @ and % signs:

Matmul operation with itself:
tf.Tensor( [[3. 6.]  [3. 6.]], shape=(2, 2), dtype=float32)

Modulo operation by 2:
tf.Tensor( [[1. 0.]  [1. 0.]], shape=(2, 2), dtype=float32)

These are elementary examples, but they can be extended into complex calculations, which creates the algorithms that we use for deep learning applications.

Note: These operators also work on regular Tensor objects.

Assignment, Indexing, Broadcasting, and Shape Manipulation

Assignment

With the tf.assign() function, you may assign new values to a Variable object without creating a new object. Being able to assign new values is one of the advantages of Variables, where value reassignment is required. Here is an example of reassignment of values:

Output:
...array([[  2., 100.],
          [  1.,  10.]],...

Indexing

Just as in Tensors, you may easily access particular elements using index values, as shown below:

Output:
The 1st element of the first level is: [1. 2.]
The 2nd element of the first level is: [1. 2.]
The 1st element of the second level is: 1.0
The 3rd element of the second level is: 2.0

Broadcasting

Just as with Tensor objects, when we try to do combined operations using multiple Variable objects, the smaller Variables can stretch out automatically to fit larger Variables, just as NumPy arrays can. For example, when you attempt to multiply a scalar Variable with a 2-dimensional Variable, the scalar is stretched to multiply every 2-dimensional Variable element. See the example below:

tf.Tensor([[ 5 10]
           [15 20]], shape=(2, 2), dtype=int32)

Shape Manipulation

Just as in Tensor objects, you can reshape Variable objects as well. For the reshape operation, we can use the tf.reshape() function. Let’s use the tf.reshape() function in code:

tf.Tensor( [[1.]
            [2.]
            [1.]
            [2.]], shape=(4, 1), dtype=float32)

Hardware Selection for Variables

As you will see in the upcoming Parts, we will accelerate our model training with GPUs and TPUs. To be able to see what type of device (i.e., processor) our variable is processed with, we can use .device attribute:

The device which process the variable:   /job:localhost/replica:0/task:0/device:GPU:0

We can also set which device should process a particular calculation with the tf.device() function by passing the device name as an argument. See the example below:

Output:
The device which processes the variable a: /job:localhost/replica:0/task:0/device:CPU:0

The device which processes the variable b: /job:localhost/replica:0/task:0/device:CPU:0

The device which processes the calculation: /job:localhost/replica:0/task:0/device:GPU:0

Even though you will not have to set this manually while training a model, there might be circumstances where you have to choose a device for a particular calculation or data processing work. So, beware of this option.

Congratulations

We have successfully covered the basics of TensorFlow’s Variable objects.

Give yourself a pat on the back!

This should give you a lot of confidence since you are now much more informed about the main mutable Variable object type used for all kinds of operations in TensorFlow.

If this is your first post, consider starting from Part 1 of this tutorial series:

or check out Part 2: