3 Ways to Build Neural Networks in TensorFlow with the Keras API

Building Deep Learning models with Keras in TensorFlow 2.x is possible with the Sequential API, the Functional API, and Model Subclassing

  Figure 1. The Sequential API, The Functional API, Model Subclassing Methods Side-by-Side

If you are going around, checking out different tutorials, doing Google searches, spending a lot of time on Stack Overflow about TensorFlow, you might have realized that there are a ton of different ways to build neural network models. This has been an issue for TensorFlow for a long time. It is almost like TensorFlow is trying to find its path towards a bright deep learning environment. Well if you think about it, this is exactly what is happening and this is pretty normal for a library in its version 2.x. Since TensorFlow is so far the most mature deep learning library on the market, this is basically the best you can get.

Keras-TensorFlow Relationship

A Little Background

TensorFlow’s evolution into a deep learning platform did not happen overnight. Initially, TensorFlow marketed itself as a symbolic math library for dataflow programming across a range of tasks. Therefore, the value proposition that TensorFlow initially offered was not a pure machine learning library. The goal was to create an efficient math library so that custom machine learning algorithms that are built on top of this efficient structure would train in a short amount of time with high accuracy.

However, building models from scratch with low-level APIs repetitively was not very ideal. So, François Chollet, a Google engineer, developed Keras, as a separate high-level deep learning library. Although Keras has been capable of running on top of different libraries such as TensorFlow, Microsoft Cognitive Toolkit, Theano, or PlaidML, TensorFlow was and still is the most common library that people use Keras with.

Current Situation

After seeing the messiness around the model-building process, the TensorFlow team announced that Keras is going to be the central high-level API used to build and train models in TensorFlow 2.0. The alternative high-level API, the Estimator API, has started to lose its already-diminishing popularity after this announcement.

The Estimator API and The Keras API

 Figure 2. The Positions of the Keras API and the Estimator API within TensorFlow Diagram

Now, let’s go back to the problem: There are a lot of different methods that people build their models using TensorFlow. The main reason for this problem is TensorFlow’s failure to adopt a single Model API.

In version 1.x, for the production-level projects, the go-to model-building API was the Estimator API. But, with the recent changes, the Keras API has almost caught up with the Estimator API. Initially, the Estimator API was more scaleable, allowed multi-distribution, and had a convenient cross-platform functionality. Yet, most of these advantages of the Estimator API are now eliminated, and therefore, soon the Keras API will probably become the single standard API to build TensorFlow models.

So, in this post, we will only focus on the Keras API methods to build models in TensorFlow and there are three of them:

• Using the Sequential API
• Using the Functional API
• Model Subclassing

I will make their comparison directly with their corresponding model building codes so that you can actually test them yourself. Let’s dive into coding.

Initial Code to Make the Comparison

To test these three Keras methods, we need to select a deep learning problem. Image Classification with MNIST is a very straightforward task. What we are trying to achieve is training a model to recognize handwritten digits, using the famous MNIST dataset.

MNIST dataset, which stands for Modified National Institute of Standards and Technology database, is a large database of handwritten digits that is commonly used for training various image processing systems. The MNIST database contains 60,000 training images and 10,000 testing images taken from American Census Bureau employees and American high school students. You can find my separate tutorial about Image Classification if you would like to follow the full tutorial.

With the code below, we will import all layers and models so that it would not bother us in the upcoming parts. We also download the MNIST dataset and preprocess it so that it can be used in all models we will build with these three different methods. Just run the code below:

Now, this part is out of the way, let’s focus on the three methods to build TensorFlow models.

3 Ways to Build a Keras Model

There are three methods to build a Keras model in TensorFlow:

• The Sequential API: The Sequential API is the best method when you are trying to build a simple model with a single input, output, and layer branch. It is an excellent option for newcomers who would like to learn fast.
• The Functional API: The Functional API is the most popular method to build Keras models. It can do everything that the Sequential API can do. Also, it allows multiple inputs, multiple outputs, branching, and layer sharing. It is a clean and easy-to-use method, and it still allows a good level of customization flexibility.
• Model Subclassing: Model subclassing is for advanced level developers who need full control over their model, layer, and training process. You need to create a custom class defining the model, and you probably won’t need it for daily tasks. But, if you are a researcher with experimental needs, then model subclassing might be the best option for you since it would give you all the flexibility you need.

 Figure 4. Converting 2-Dim Image Array into 1 Dim Array with Flatten Layer

Let’s see how these methods are implemented. We will build a basic feedforward neural network with a single Flatten layer to convert 2-dimensional image arrays to 1-dimensional arrays and two Dense layers.

The Sequential API

In the Sequential API, we need to create a Sequential object from tf.keras.Models module. We can simply pass all the layers as a single argument in the list format as shown below. As you can see that it is very simple.

The Functional API

With Functional API, we need to define our input separately. Then, we need to create an output object by also creating all the layers which are tied to one another and to the output. Finally, we create a Model object which would accept inputs and outputs as arguments. The code is still very clean, but we have much more flexibility in the Functional API.

Model Subclassing

Let’s move on to model subclassing. In model subclassing, we start with creating a class extending tf.keras.Model class. There are two crucial functions in Model subclassing:

• __init__ function acts as a constructor. Thanks to __init__, we can initialize the attributes (e.g., layers) of our model. super is used calls the parent constructor (the constructor in tf.keras.Model) and self is used to refer to instance attributes (e.g., layers).
• call function is where the operations are defined after the layers are defined in the __init__ function.

To build the same model with Model Subclassing, we need to write much more code, as shown below:

The End Code

Now that you can create the same model with three different methods, you can choose any single one of them, build the model, and run the code below.

The lines above take care of model configuration, training, and evaluation. When we compare the performances of these three methods, we see that they are pretty close but slightly different.

 Figure 5. Photo by Jonathan Chng on Unsplash

Our more complex Model Subclassing method outperforms the Sequential API and the Functional API. This shows that there are slight differences in the design of these methods in the low-end as well. However, these differences are negligible.

Final Evaluations

By now, you have an idea about the similarities and differences between these three Keras methods. But, let’s summarize them all in a table:

In summary, if you are just starting out, stick with the Sequential API. As you dive into more complex models, try out the Functional API. If you are doing a Ph.D. or just enjoy conducting independent research, try out Model Subclassing. If you are a professional, stick with the Functional API. It will probably satisfy your needs.

Let me know your favorite method to build Keras models in the comments.