Learn how to use TensorFlow Variables, their differences from plain Tensor objects, and when they are preferred over these Tensor objects | Deep Learning with TensorFlow 2.x
WARNING: Do not confuse this article with “Mastering TensorFlow Tensors in 5 Easy Steps”!
If you are reading this article, I am sure that we share similar interests and are/will be in similar industries. So let’s connect via Linkedin! Please do not hesitate to send a contact request! Orhan G. Yalçın — Linkedin
In this tutorial, we will focus on TensorFlow Variables. After the tutorial, you will be able to create, update, and manage TensorFlow Variables effectively. As usual, our tutorial will deliver code examples with detailed explanations as well as conceptual explanations. We will master TensorFlow Variables in 5 easy steps:
- Step 1: Definition of Variables →A Brief Introduction, Comparison with Tensors
- Step 2: Creation of Variables → Instantiating tf.Variable Objects
- Step 3: Qualifications of Variables → Characteristics and Features
- Step 4: Operations with Variables → Basic Tensor Operations, Indexing, Shape Manipulation, and Broadcasting
- Step 5: Hardware Selection for Variables → GPUs, CPUs, TPUs
Fasten your belts, and let’s start!
Definition of Variables
In this step, we will briefly cover what Variables are and understand the difference between plain Tensor objects and Variable objects.
A Brief Introduction
A TensorFlow Variable is the preferred object type representing a shared and persistent state that you can manipulate with any operation, including TensorFlow models. Manipulation refers to any value or parameter update. This characteristic is the most distinguishing feature of Variables compared to
tf.Tensor objects. TensorFlow Variables are recorded as
tf.Variable objects. Let’s make a brief comparison between
tf.Variable objects to understand their similarities and differences.
Comparison with Tensors
So, the most important difference between Variables and Tensors is mutability. The values in a Variable object can be updated (e.g., with the
assign() function) as opposed to Tensors.
Variable objects are mainly used to store model parameters, and since these values are constantly updated during training, using Variables, instead of Tensors, is a necessity rather than a choice.
The shape of a Variable object can be updated with the
reshape() instance function just like the shape of a Tensor object. Since Variable objects are built on top of Tensor objects, they have common attributes such as
.dtype. But, Variables also have unique attributes such as
.name attributes that the Tensors do not have.
Let’s see how we can create
Creation of Variables
We can instantiate (i.e., create)
tf.Variableobjects with the
tf.Variable() function. The
tf.Variable() function accepts different data types as parameter such as integers, floats, strings, lists, and
Before showing different Variable object examples with these different data types, I want you to start a new Google Colab notebook and import TensorFlow library with the following code:
Now, we can start creating
1 — We can pass a
tf.constant() object as the
2 — We can pass a single integer as the
3 — We can pass a list of integers or floats as the
4— We can pass a single string as the
5— We can pass a list of strings as the
As you can see, there are several data types that th e
tf.Variable() function accepts as the
initial_value argument. Now let’s take a look at the characteristics and features of variables.
Qualifications of Variables
Every Variable must have some properties such as value, name, uniform data type, shape, rank, size, and more. In this section, we will see what these properties are and how we can view these properties in a Colab notebook.
Every Variable must specify an
initial_value. Otherwise, TensorFlow raises an error and says that
Value Error: initial_value must be specified. Therefore, make sure that you pass on an
initial_valueargument when creating Variable objects. To be able to view a Variable’s values, we can use the
.value() function as well as the
.numpy() function. See the example below:
Output: The values stored in the variables: tf.Tensor( [[1. 2.] [1. 2.]], shape=(2, 2), dtype=float32)
The values stored in the variables: [[1. 2.] [1. 2.]]
Name is a Variable attribute which helps developers to track the updates on a particular variable. You can pass a
name argument while creating the Variable object. If you don’t specify a name, TensorFlow assigns a default name, as shown below:
Output: The name of the variable: Variable:0
Each Variable must have a uniform data type that it stores. Since there is a single type of data stored for every Variable, you can also view this type with the
.dtype attribute. See the example below:
Output: The selected datatype for the variable: <dtype: 'float32'>
Shape, Rank, and Size
The shape property shows the size of each dimension in the form of a list. We can view the shape of the Variable object with the
.shape attribute. Then, we can view the number of dimensions that a Variable object has with the
tf.size() function. Finally, Size corresponds to the total number of elements a Variable has. We need to use the
tf.size() function to count the number of elements in a Variable. See the code below for all three properties:
Output: The shape of the variable: (2, 2) The number of dimensions in the variable: 2 The number of dimensions in the variable: 4
Operations with Variables
There are several basic operations you can easily conduct with math operators and TensorFlow functions. On top of what we covered in Part 2 of this tutorial series, you may also use the following math operators for Variable operations.
Basic Tensor Operations
- Addition and Subtraction: We can conduct addition and subtraction with
Addition by 2: tf.Tensor( [[3. 4.] [3. 4.]], shape=(2, 2), dtype=float32)
Substraction by 2: tf.Tensor( [[-1. 0.] [-1. 0.]], shape=(2, 2), dtype=float32)
- Multiplication and Division: We can conduct multiplication and division with
Multiplication by 2: tf.Tensor( [[2. 4.] [2. 4.]], shape=(2, 2), dtype=float32)
Division by 2: tf.Tensor( [[0.5 1. ] [0.5 1. ]], shape=(2, 2), dtype=float32)
- Matmul and Modulo Operations: Finally, you can also do matmul and modulo operations with
Matmul operation with itself: tf.Tensor( [[3. 6.] [3. 6.]], shape=(2, 2), dtype=float32)
Modulo operation by 2: tf.Tensor( [[1. 0.] [1. 0.]], shape=(2, 2), dtype=float32)
These are elementary examples, but they can be extended into complex calculations, which creates the algorithms that we use for deep learning applications.
Note: These operators also work on regular Tensor objects.
Assignment, Indexing, Broadcasting, and Shape Manipulation
tf.assign() function, you may assign new values to a Variable object without creating a new object. Being able to assign new values is one of the advantages of Variables, where value reassignment is required. Here is an example of reassignment of values:
Output: ...array([[ 2., 100.], [ 1., 10.]],...
Just as in Tensors, you may easily access particular elements using index values, as shown below:
Output: The 1st element of the first level is: [1. 2.] The 2nd element of the first level is: [1. 2.] The 1st element of the second level is: 1.0 The 3rd element of the second level is: 2.0
Just as with Tensor objects, when we try to do combined operations using multiple Variable objects, the smaller Variables can stretch out automatically to fit larger Variables, just as NumPy arrays can. For example, when you attempt to multiply a scalar Variable with a 2-dimensional Variable, the scalar is stretched to multiply every 2-dimensional Variable element. See the example below:
tf.Tensor([[ 5 10] [15 20]], shape=(2, 2), dtype=int32)
Just as in Tensor objects, you can reshape Variable objects as well. For the reshape operation, we can use the
tf.reshape() function. Let’s use the
tf.reshape() function in code:
tf.Tensor( [[1.] [2.] [1.] [2.]], shape=(4, 1), dtype=float32)
Hardware Selection for Variables
As you will see in the upcoming Parts, we will accelerate our model training with GPUs and TPUs. To be able to see what type of device (i.e., processor) our variable is processed with, we can use
The device which process the variable: /job:localhost/replica:0/task:0/device:GPU:0
We can also set which device should process a particular calculation with the
tf.device() function by passing the device name as an argument. See the example below:
Output: The device which processes the variable a: /job:localhost/replica:0/task:0/device:CPU:0
The device which processes the variable b: /job:localhost/replica:0/task:0/device:CPU:0
The device which processes the calculation: /job:localhost/replica:0/task:0/device:GPU:0
Even though you will not have to set this manually while training a model, there might be circumstances where you have to choose a device for a particular calculation or data processing work. So, beware of this option.
We have successfully covered the basics of TensorFlow’s Variable objects.
Give yourself a pat on the back!
This should give you a lot of confidence since you are now much more informed about the main mutable Variable object type used for all kinds of operations in TensorFlow.
If this is your first post, consider starting from Part 1 of this tutorial series:
Understanding the TensorFlow Platform and What it has to Offer to a Machine Learning Experttowardsdatascience.com
or check out Part 2: