Overview of Deep learning with Gluon

Overview of Deep learning with Gluon

This chapter introduces a fundamental concept and jargon that every machine learning engineer and data scientist should know. In this chapter, we will discuss some basic concept of machine learning, deep learning, and AI. Then in subsequent chapters of this book, we will dive deeper and makes hand dirty. Deep learning(DL) has been outbreaking technology for all the industries and its booster for the AI adaptation.

Andrew Ng once said Artificial Intelligence is the new electricity!

AI and DL and ML are used interchangeably but there is a substantial difference between these three. We will start with a brief definition of each one. This chapter will cover basic of the machine learning, deep learning and AI and some foundation terminology to understand the deep learning then we will have a glance over gluon API. We will also cover some part of MXnet Deep learning framework along with Gluon API. This book is for any technical person who wants to get up to speed on machine learning and deep learning quickly. And anyone who is a novice to the technology but who is curious about how the machine thinks and act. In this book, we can dig deeper into the Deep learning neural network using Gluon API and underline deep learning framework is MxNet. Gluon is packaged along with MxNet and it is an abstraction layer over Apache MxNet Deep learning framework. Gluon name was given by subatomic particle. A gluon is an elementary particle that acts as the exchange particle. This book is for the data scientist and machine learning Engineer and aspiring data scientist.

This chapter contains below points,

  • Artificial Intelligence
  • Machine learning
  • Deep learning
  • Neural Network Architectures
  • Gluon API overview and environment setup

Artificial Intelligence(AI)

Artificial intelligence is where the machine will think, act, fail, learn and react without human intervention. Artificial intelligence is the hype now in the industry and there are tons of articles available — they teach us, dream us for future and scare us as well but above all AI the revolutionary technology. The progress which we did in the last couple of years was awesome due to the amount of innovation in computation power and a vast amount of data. At the very highest level, AI is about creating machines capable of solving problems like a human. As a human, we learn through reasoning, intuition, cognitive thinking, and creativity. There are several definitions of AI floating around, my favorite one “the science and engineering of making intelligent machines”.

The history of AI:-

During the second world war, the Germans build the Enigma machine to be used in military communications to send messages securely.
Alan Turing and team built the machine that used to decipher enigma messages.
Cracking the enigma code by a human was very challenging due to the different permutation and combination. The journey of the question of whether can machines think and act like a human or not started much earlier than that. In the early days of AI, machines were able to solve problems that were difficult for humans to solve or the mundane industry work.
There are different aspects of human intelligence and AI. We just want how to mimic human and built an intelligent machine.

In 1956, American computer scientist John McCarthy organized the Dartmouth Conference, at which the term ‘Artificial Intelligence’ was coined first. Researchers Allen Newell and Herbert Simon were instrumental in promoting AI as a field of computer science that could transform the world. The father of AI developed the LISP programming language which becomes important in AI. In 1951, a machine known as Ferranti Mark 1 successfully used an algorithm to master checkers. Subsequently, Newell and Simon developed a General Problem Solver algorithm to solve mathematical problems. It was also in the late 1960s that the first mobile decision-making robot capable of various actions was made. Its name was Shakey.
Shakey could create a map of its surroundings prior to moving. The first ‘intelligent’ humanoid robot, was built in Japan in 1972. In the early days of AI, researcher believe AI could be able to solve the problem by hard-coding a rule-based system like a decision tree.
This AI system aka Symbolic AI and it was very successfully to solve well defined logical problems but it was failed to solve complex problems such as natural language understanding, image detection, understanding scene, Object detection, time-based forecasting.
Over the decade of efforts and well funded global efforts, researchers found it incredibly difficult to create intelligent machine due to different reasons unavailable of computing power, lack of data.
In 1997, IBM’s Deep Blue defeated became the first computer to beat a supreme world chess champion, Garry Kasparov. AI technology continued its march, largely thanks to improvements in computer hardware and people used AI methods in a narrow domain instead of general intelligence that help researchers to solve some complex problem.
Exponential gains in computer processing power and storage ability allowed companies to store vast quantities of data. Today’s AI hits on almost every aspect of human life, from the military and entertainment to our cell phone and driverless cars, from real-time voice translation to a vacuum that knows where and how to clean our floor without you, from our own computer to your doctor’s office. An autonomous (driverless) car, facial recognization for authentication So what where is AI going in the future? Is it scary or not. No one can tell you for sure.

AI-powered machines are usually classified into two groups — general and narrow. The narrow AI machines can perform specific tasks very well, sometimes better than humans
The technology used for classifying images on Airbnb is an example of narrow AI.
AI, DL, and ML fit together.

Machine learning:-

Machine learning is a computer science branch that deals with methods and technique to implement an algorithm. Machine learning is inferential leaning from a descriptive data set.
This era is data mining era. Data is the fuel of the 21st century. If you have data(fuel ) then you can develop an AI system that electrifies your business. In generally while doing programming means we have data and rules and we will expect some result from this. This is one of the paradigms we follow as a programmer. We want to write down a program to convert temperature Fahrenheit to Celsius, to do this we need data values in Fahrenheit and formula for conversation, then with the help of this, we will write down code snippet that fulfills this requirement and result of this code snippet is a temperature in Celsius.
Machine learning has shifted this paradigm, Machine learning will take data and Answers as input and in a result return the rules. As we discussed above Fahrenheit to celsius programming, but we will just think this problem in ML context. We will provide both Fahrenheit and Celsius values and ask the ML program will find out the relation between this, that means find out the formula. This is just a simple example but there are many more complex problems addressed with the help of ML.

There are plenty of definitions are articles available over the internet that can explain to you what is machine learning? When I just fire a query to google wiki, this is the very simple definition of a machine learning I come across.

“Machine learning gives computers the ability to learn without being explicitly programmed (Arthur Samuel, 1959). It is a subfield of computer science. The idea came from work in artificial intelligence. Machine learning explores the study and construction of algorithms which can learn and make predictions on data.”

more engineering-oriented Definition:

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.
 — Tom Mitchell, 1997

Likewise in the above example, we discussed your program should explicitly identify the relationship between Fahrenheit and Celsius value and take action accordingly in the future instead of providing an implicite formula for conversion. Machine learning is not only a science but also its an art. In machine learning data is the challenging part, If we have a data for training algorithm to validate that ML algorithm we need testing data set as well, so in ML we need two data set one if to train Algorithm and another for testing.

Types of Machine learning:

Less is More

1. Supervised Machine learning:

Supervised machine learning is the technique of Inferring a rule from the labeled dataset. Supervised machine learning means machine learning with some amount of human supervision, means we have input data along with output label. In the supervised learning data set is available with expected output this aka label. From data, the machine learning algorithm will understand for which input what is the output. Typical supervised learning address classification, regression problem such as spam filter, prediction of house prices. To train these systems we need a huge amount of data set.

Below are some Supervised Machine learning algorithms

  • Linear Regression
  • Logistic Regression
  • Support Vector Machines (SVMs)
  • Decision Trees and Random Forests
  • k-Nearest Neighbors
  • Neural networks2

2. UnSupervised Machine learning:

UnSupervised machine learning is the technique of Inferring a rule and find a meaning full pattern from data set. In this type of machine learning, datasets consisting of input data without labeled result. UnSupervised learning with supervision or learning without a teacher. To train unsupervised algorithm the given data are not annotated that mean only input values provided. This technique is useful to group the data or do the clustering and find the common pattern from the data.

Some unsupervised Algorithm

  • Clustering:
    k-Means
    Hierarchical Cluster Analysis (HCA)
    Expectation Maximization
  • Visualization and dimensionality reduction:
    Principal Component Analysis (PCA) — Kernel PCA
    Locally-Linear Embedding (LLE)
    t-distributed Stochastic Neighbor Embedding (t-SNE)
  • Association rule learning:
    Apriori
    Eclat

3. Self-supervised learning:

Self Supervised learning is a very recent technique of machine learning. This is supervised learning but instead of providing labeled data by human as an input, the data set is auto labeled. Self-supervised learning technique as a potential to solve a problem which is not addressed by supervised learning. As I mentioned earlier in the machine learning data set is the challenging thing. To provide a huge amount of labeled data is a very crucial task.

Self-supervised learning is autonomous supervised learning. It is a representation learning approach that eliminates human supervision to label data. Self-supervised learning is very relevant to human because we learn a few things in a supervised manner and few unsupervised ways but we learn from very few examples and generalize exceptionally well.

4. ReInforcement Leaning:

ReInforcement learning is another technique in machine learning. Have you visited a circus ever, in circus ringmaster train the tiger? For tigers, positive behavior ring master can reward him and for negative behavior can be punishment. The way we learn in academia.
Reinforcement learning means the agent will learn to reinforce the way in a particular environment, can be rewarded for positive behavior and get punished for negative behavior.
Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. Reinforcement learning is useful in gaming, it’s goal-oriented leaning where an agent can learn how to behave in the environment by performing actions and accumulate maximize reward to reach to the goal.

This is a very interesting analogy used by Yann LeCun to understand this.

“ Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. “

Deep Leaning:

Deep-learning is the inferring learning approach that human beings use to gain knowledge. Over the past couple of year, deep learning revolutionized the many aspects of research and industry including things like an autonomous driving vehicle, healthcare, reinforcement learning, generative modeling, NLP, robotics, fintech. Deep-learning technique is the part of a broader family of machine learning. Deep learning technique taking inference of how the human brain works. Deep neural networks are in deep structured and hierarchical each level of a hierarchy represent a different level of abstraction. Deep-learning is now become hype because of advancement in hardware and software. Artificial Neural network is the core part of deep-learning. ANN is the inference of taken from human brain neuron. In a human brain, there are millions of neuron present and they are interconnected and there structure and hierarchy are very deep and complex. Deep-learning neural networks are taken inference from the human brain such how human understand the scene or how our cortex work to identify the object, CNN (Convolutional Neural network) is the best example for this one. Before deep-learning technique to Object detection or to detect human face is a very crucial one, you need to extract feature and create a template for the same, such as the detect nose, the left eye, right eye means you need to define every single step to reach an outcome but with the help of deep learning we can understand any scene and object detection has become very easy. The deep neural network has a deep level of neural network in a hierarchical and abstract way to understand the things and finally combine the result.
Deep-learning is a subset of machine learning which takes ML one step further to process and understand data and find meaningful insight.

Our brain consists of a large network of interconnected neurons, which act as a roadway for information to be transmitted from point A to point B. To send different kinds of information from A to B, the brain activates a different set of neurons, and so essentially uses a different route to get from A to B. Biological neurons are interconnected they understand things by an alteration of sending signals. The cell consists of a cell body, with dendrites acting as connecting wires for other neurons to connect to. In most cases, a neuron has one axon capable of transmitting electric currents actively to other connecting cells. The connections between neurons are established using synapses located at the end of the axon. These synapses are responsible for a lot of the magic of computing and memory in the nervous system. The ANN model is modeled after the biological neural network. In the above diagram Just like a biological neuron has dendrites to receive signals, a cell body to process them, and an axon to send signals out to other neurons, the artificial neuron has a number of input channels, a processing stage, and one output that can fan out to multiple other artificial neurons.

Bit history:

Deep-learning evolved all industry but in general deep learning addressed major problems such as speech recognization system, image recognization, Object detection from the image.

The word “deep learning” was first used when talking about Artificial Neural Networks (ANNs) by Igor Aizenberg and colleagues in or around 2000. In deep-learning deep refers to the number layers typically.

1960s: Shallow neural networks
1960–70s: Backpropagation emerges
1974–80: First AI Winter
1980s: Convolution emerges
1987–93: Second AI Winter
1990s: Unsupervised deep learning
1990s-2000s: Supervised deep learning back in vogue
2006s-present: Modern deep learning

Neural Network architectures:

A neural network is designed to solve a complex task, some tasks are more complex to solve but not impossible such as write down a recommendation system based on shopping history. As a programmer, we can write down some sort of hardcoded rules to fulfill this requirement but this is mundane work, so come with machine learning algorithms will help us to explore data and find meaningful insight pattern. Machine learning or AI system comes into the picture where there is more uncertainty, such as

  1. It’s hard to identify the fraudulent transaction in digital money transfer where the end user is not in front of a system its virtual one.
  2. It’s very hard for a machine to detect the pedestrian.

Artificial Neural networks are the first class model to predicate this uncertain result. ANN is the inference of the human brain. ANN is a simulation of the human brain. Neural network architecture is very complex and they are very adaptive and do parallel computation.

Neural network research is motivated by two desires,

  1. Understand the human brain better way,
  2. Mimic human activity and intelligence in computers that can deal with a complex problem.

There is a different architecture of Neural network will address domain specific problem. Human intelligence is generally intelligent. It’s very tough to develop artificial general intelligence to address almost all or some problem. Neural Network architectures are consist of three major layers: the input layer, hidden layers, and the output layer. The number of hidden layers defines the depth of the Neural network architecture.

Below we will check some brief introduction about some ANN.

  1. Perceptrons
  2. Hopfield Neural Network
  3. CNN
  4. Recurrent Neural Networks

Gluon API

Overview

Gluon API is the high-level simple, concise and efficient deep learning API. Amazon and Microsoft research group developed Gluon API specification. This is the product of joint effort taken by both leading tech companies to generalize AI for any developer. Gluon is open source deep learning interface, jointly developed by the companies to let developers “prototype, build, train and deploy sophisticated machine learning models for the cloud, devices at the edge and mobile apps.

Gluon is an API, not another deep learning framework, they provided some concise and clear API abstraction layer this helps us to improve speed, flexibility, and accessibility of deep learning technology for all developers, regardless of their deep learning framework of choice. Gluon offers an interface that allows developers to prototype, build, and train deep learning models.

Developers who are new to machine learning will find this interface more familiar to traditional code since machine learning models can be defined and manipulated just like any other data structure. Seasoned data scientists and researchers will value the ability to build prototypes quickly and utilize dynamic neural network graphs for entirely new model architectures, all without sacrificing training speed.

Gluon is imperative for developing but symbolic for deploying.

Before we dive dipper into the Gluon API, we should know at least one of the underline framework on Gluon is rely upon. Gluon is the abstraction layer for deep learning framework such as MxNet.

Distinct Advantages:

  1. Friendly API Simple, Easy-to-Understand Code
  2. Flexible, Imperative Structure
  3. Build graphs on the fly
  4. High-performance operators for training

MxNet:

MXNet is open source deep learning library by Amazon. This founded by U.Washington and Carnegie Mellon U. This is a portable, efficient and scalable deep learning framework. This will support python, javascript, Scala, Julia, and R. The best thing about MXNet is, it allows both imperative(define by run) and symbolic programming. It has a vibrant community backed by Amazon.

Installing Gluon on MacOS:

The Gluon specification has already been implemented in Apache MXNet so we need to install apache MxNet to setup environment. It’s easy to set up an environment for Gluon API using different options such as docker, pip, virtual environment. MxNet is supporting different languages along with different OS platform. I will show here installation for Mac OS.

You can refer this link to do the installation for your respective platform. (https://mxnet.incubator.apache.org/versions/master/install/index.html?platform=MacOS&language=Python&processor=CPU)

By default, MxNet gets installed with CPU but you can also do the installation for GPU enabled mode.

Pip mode

$ pip install mxnet

MXNet offers MKL pip packages that will be much faster when running on Intel hardware.

Using Docker

Docker images with MXNet are available at Docker Hub(https://hub.docker.com/).

Step 1 Install Docker on your machine. For more detail (https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-for-mac)

Step 2 Pull the MXNet docker image.

$ docker pull mxnet/python

Very your docker pull command

$ docker images

I recommend using Python version 3.3 or greater and setup environment using a Jupyter Notebook

# I used minicoda and virtual environment 
# source activate gulons
# optional: update pip to the newest version
sudo pip install --upgrade pip
# install jupyter
pip install jupyter --user
# install the nightly built mxnet
pip install mxnet --pre --user
Default MxNet is come up with CPU you can install GPU as well If you have GPU availability.
pip install mxnet-cu75 --pre --user  # for CUDA 7.5
# for CUDA 8.0 use this mxnet-cu80 --pre --user
#start notebook and enjoy coding
jupyter notebook

Validate the installation:

To validate the installation this is the simple steps

Pip installation validation, start the terminal and type

$ python
$ import mxnet as mx
$ from mxnet import gluon

The same way you can do the validation for Docker setup and starting docker image and executing a bash command.

MXNet should work on any cloud provider’s CPU-only instances. You can Also do setup Gluon and MxNet over any cloud platform. It’s easy to set up over Amazon AWS.

AWS Deep learning AMI (Amazon Machine Images) — Preinstalled Conda environments for Python 2 or 3 with MXNet and MKL-DNN.

Also, MxNet supports different edge platforms such as Raspberry Pi and NVIDIA Jetson Devices.

The architecture of MxNet:

In the above diagram, you can explore the key modules of the MxNet framework and their relation. As stated in the above diagram solid arrow indicate the concrete dependency and dotted line indicate light dependency. In the above modules, lower modules indicated by bluish color are the system modules and high-level modules indicate user-facing modules this is the actual API where the programmer will do interaction. The modules are

KVStore:- key-value store interface for parameter synchronization

Data Loading:- Efficient distributed data loading and augmentation

NDArray:- Dynamic, asynchronous n-dimensional arrays

Symbolic Execution:- Static symbolic graph executor

Symbolic Construction:- provides a way to construct a computation graph

Operators:-Operators that define static forward and gradient calculation

Storage Allocator:- Allocates and recycles memory blocks

Runtime Dependency Engine:-Schedules and executes the operations

Resource Manager:- Manages global resources

Gluon Package

Gluon package comes with four key modules.

  1. Parameter:- Parameter is a basic component. A parameter can hold the weight of blocks. There are two standard API one is Parameter and to manage a set of parameters we have ParameterDict
  2. Containers:- Containers the blocks that will help you to build neural network Containers are the blocks which hold the parameters.
  3. Trainer:- Trainer helps you to do parameter optimization. Trainer applies optimizer over parameters in the containers.
  4. Utilities:- Utilities contains small utils that help us in certain operations, such as split, and rescale dataset for data parallelism.

Gluon APIs:

Gluon API contains below APIs.

  1. Gluon Neural Network Layers API:- Gluon Neural network layer API provides you building blocks of neural network. It contains API to directly add blocks in a neural network, such as Dense layers, Convolution layers, Activation function layer and Max Pooling layer.
  2. Gluon Recurrent Neural Network API:- This API provides building blocks to define the Recurrent Neural Network. This can help us to define RNN with LSTM.
  3. Gluon Loss API:- This API contains different loss function which is required while building a different neural network. This API can help you to calculate mean squared loss or mean absolute loss.
  4. Gluon Data API:- This API is very useful API for people who want to make hands dirty but don’t have a dataset. This API contains dataset utilities and common public datasets.
  5. Gluon Model Zoo:- Gluon model zoo contains pre-trained and pre-defined models that will help us to bootstrap our development.
  6. Gluon Contrib API:- This is for the whom who had mastery in Gluon and Who want to contribute into Gluon API. This API is for the community who wanted to try out some new features and get feedback.

Deep learning Programming style:

One of my favorite things about Gluon API is that it offers multiple levels of abstraction so you can choose the right one for your project. Gluon offers two styles to create your neural network. First one is symbolic style or Declarative style and the second one is imperative style.
These are the two-deep learning programming style. Each one has there own pros and cons, that’s why almost all the deep learning framework offers both styles of programming.

Imperative Programming:

Imperative programming means define by run means dynamic programming. The part of the computation graph constructed at the run time. Imperative programming is flexible and straightforward. In this programming, we can take advantage of language native features such as iteration, condition, debugger, etc.
Imperative style is nothing new for you the way you are writing Numpy code is the imperative style of programming. Imperative style programs perform operations directly.
Most of the Python code shows an imperative form, for example, the following Numpy code. In this style of programming, the state of the program is getting changed.

import numpy as np
a = np.ones(20)
b = np.ones(20) * 2
c = b * a
d = c + 1

Here is above code snippet When we issue c = b * a command to run the program, the actual operation is getting executed.

PROS:

  1. straightforward and flexible because of execution flow with a programming language.
  2. Take advantage of native language features

Cons:

  1. Manual optimization
  2. Not efficient in terms of memory usage and speed.

Symbolic Style of programming:

Symbolic programming aka declarative programming it’s contrary to imperative programming style. In this style of programming execution performed after the computational process fully defined. In this paradigm you need to first define and then run, this is a status computation graph. This is the immutable graphs this is not changing at run time. Symbolic-style programs include compilation steps either explicitly or implicitly, this converts the graph into the function that actually getting called any time. In this style of programming, we can just define a function with a placeholder value and after this, you can compile the function and evaluate it with the actual input. Below is a code snippet, converting above imperative code to symbolic code In the symbolic programming generally requires three steps:

#Step 1:- Define the computation graph.
a = Variable('A')
b = Variable('B')
c = b * a
d = c + Constant(1)
#Step 2:- Compile the computation process into an executable program.
f = compile(d)
#Step 3:- Provide the required inputs and call on the compiled #program for execution.
g = f(a=np.ones(20), b=np.ones(20)*2)

In this code snippet, the c = b * a does not actually perform the operation, instead, this will generate the computation graph that represents this computation process.
Following computation, a graph is generated for operation d.

PROS:

  1. Infer optimization automatically from the dependency graph.
  2. Memory reuse opportunities.
  3. More efficient and easier to port.

Cons:

  1. Less flexible

Hybrid Programming style:

Gluon comes up with hybrid programming style and its the positive point for this, in the above description you can not conclude which programming style is good in deep learning.
Gluons hybrid approach give us more flexibility to harness the benefits of both imperative and symbolic programming. User should imperative programming to build and test a prototype on the fly and while deploying or serving in production, we can convert a program into symbolic programming to achieve product level computing performance.
This was possible due to gluon API hybrid programming.

In the hybrid programming, we can build models using either the HybridBlock or the HybridSequential Gluon API classes. By default, Gluon API uses the Block or Sequential Block classes same that is used in imperative programming. When we call hybridize function,
then Gluon will convert programs execution into symbolic programming style.

Let us take a small example of Hybrid programming.

#imperative
import mxnet as mx
from mxnet import nd
a = mx.nd.zeros((120,60))
b = mx.nd.zeros((120,60))
c = a + b
c += 1
print (c)
#Symbolic
improt mxnet as mx
from mxnet import nd
net = mx.sym.Vairable('data')
net = mx.sym.FullyConnected(data=net, num_hidden=10)
net = mx.sym.SoftmaxOutput(data=net)
texec = mx.module.Module(net)
texec.forword(data=c)
texec.backward()

The NDArray API:

In this section, we will introduce the NDArray API. In the MxNet NDArray API is the primary tool to store, transform and manipulate data. This is the core data structure for all computation. NDArray is the multi-dimensional array similar to a Numpy. The NDArray represent the multi-dimensional, fixed size homogenous array. Basically, NDArray provides API to imperative tensor operations. The mxnet.ndarray is similar to numpy.ndarray but not very similar there is some difference.

Array creation:-

We can create NDArray using python tuple or list with NDArray array function.

import mxnet as mx 
from mxnet import nd
# create a 1D array with a python list 
x = mx.nd.array([4,3,9]) 
# create a 2D array with a nested python list 
z = mx.nd.array([[4,3,6], [5,1,8]]) 
#display the array
{'x.shape':x.shape, 'z.shape':z.shape}

We can also create NDArray using numpy.array API.

# import numpy package
import numpy as np
from mxnet import nd
# create numpy array
d = np.arange(15).reshape(3,5)
# create a 2D array from a numpy.ndarray object
y = mx.nd.array(d)
# display array
{'y.shape':y.shape}

We can specify data which is optional dtype while creating of NDArray. By default, float32 is used. We can also create NDAaray with placeholder with the help of different function such as zeros, ones, etc. NDArray also offers generally all the API that are required to manipulate the data such as slicing, indexing, shape, basic arithmetic, copies, reduce, etc.

# basic operatiosn of NDArray
# float32 is used by default
a = mx.nd.array([1,2,3])
# create a 16-bit float array
c = mx.nd.array([1.2, 2.3], dtype=np.float16)
(a.dtype, c.dtype)
# create empty array
d = mx.nd.empty((2,3))
# create array with all zeros
e = mx.nd.zeros((2,3))
# create array with all 5
f = mx.nd.full((2,3),5)
# we can also perform some basic operations
# elementwise plus
g = a+ b
# elementwise minus
h = c-d
i = -e
# we can use sum or mean 
j = mx.nd.sum(e)
# exponential
j.exp()
# transpose matrix
nd.dot(a,c.T)
#indexing
j[1,2]
# for advanced way
j[:,1:2]

NDArray has some key advantages First, NDArrays support asynchronous mathematical computation on CPU, GPU, and distributed cloud architectures. Second, they provide support for automatic differentiation. These properties make NDArray vital choice for deep learning. As we saw we can create vector, matrix, and tensor and manipulate with the help of NDArray.

We can convert NDArray to Numpy if you have some scenarios and instead of NDArray if you want to use Numpy array we can use, the conversion is easy.

Note:- converted array does not share memory.

# convert x into numpy z array
z = x.asnumpy()
# display type of z for verification (type(z), z)
# display numpy array as a NDArray.
nd.array(z)

The Symbol API:

In the previous section, we learned about the NDArray to store and manipulate the data. In this section, we will be exploring the symbol API. Symbol API is the basic interface for symbolic programming. Symbolic API are following declarative approach, instead of executing program step by step you need to first define computation graph, computation graph contains the placeholder for inputs and desired output. Gluon API taking advantage of this approach under the hood before hybridization. Your computation graph is a composition of symbols, operators, network layers. With the symbolic API, we can optimize the computation graph. Symbolic API uses a small memory footprint because we can recycle memory from intermediate steps. NDArray allows writing a program in an imperative fashion but symbolic API allows writing a program in a declarative fashion. But most of the operators supported by NDArray also supports symbol API. A symbol means a multi-output symbolic expression

We will just build a simple example of a+b its symbolic API we need to declare placeholder for this using mx.sym.Variable, give them name as a and b respectively.

import mxnet as mx
a = mx.sym.Variable('a')
b = mx.sym.Variable('b')
c = 3 * a + b
type(c)
# output 
mxnet.symbol.symbol.Symbol

Symbol API also supports a rich set of neural network API with the help of those we can define neural networks as well.

First Gluon Example:

Create a simple neural network layer using the gluon nn package.

# import ndarray module from mxnet package
from mxnet import nd
# import gluon package
from mxnet.gluon import nn
# let us define layer Dense is a subclass of Block to define layer
layer = nn.Dense(2)
layer
# we need to initialise the weight [-0.7,0.7]
layer.initialize(ctx=mx.cpu(0))
# random (3,4) matrix range from -1 to 1
x = nd.random.uniform(-1,1,shape=(3,4))
layer(x)
#  print weight data
layer.weight.data()
# collect the parameters
layer.collect_params()
# type of params collected from layer
type(layer.collect_params())

In this example, we just saw How to define simple layer using Gluon API.

Summary:

In this chapter, we introduced some of the fundamental concepts such as Artificial intelligence, deep learning, machine learning, and Gluon API along with MxNet.

It consists of different machine learning types and deep learning techniques and most recent research in machine learning such as self-supervised learning. Deep learning is achieved by just adding more layers as a hidden layer this is possible because of the availability of huge data and advancement is computation. With the help of different deep learning framework and cloud computing now these techniques are available to any software engineer on a fingertip.

In this chapter, we begin our journey into deep learning using Gluon API. Introduction of Gluon API with different deep learning programming paradigm. This chapter ended with the installation of Gluon, environment setup and few small API examples. Let us ready with Gluon API tool to conquer the deep learning world.

Leave a Reply