Tensorflow Fundamentals¶

Contents:

• Introduction to tensors
• Getting information from tensors
• Manipulating tensors
• Tensors & Numpy
• Using @tf.function -> a way to speed up Python functions
• Using GPUs with tensorflow
• Exercises

Google Colab: Mount Drive and change to project notebook¶

In [ ]:

import os
os.chdir('/content/drive/MyDrive/projects/Tensorflow-tutorial-Daniel-Bourke/notebooks')

import os os.chdir('/content/drive/MyDrive/projects/Tensorflow-tutorial-Daniel-Bourke/notebooks')

In [ ]:

!ls ../

!ls ../

data	notebooks  references  requirements.txt  setup.py  src.egg-info
models	README.md  reports     scripts		 src	   venv

Google Colab: Install the package¶

In [ ]:

!pip install -e ../.

!pip install -e ../.

Obtaining file:///content/drive/MyDrive/projects/Tensorflow-tutorial-Daniel-Bourke
Installing collected packages: src
  Running setup.py develop for src
Successfully installed src

Introduction to tensors¶

In [ ]:

import tensorflow as tf
tf.__version__

import tensorflow as tf tf.__version__

Out[ ]:

'2.4.1'

In [ ]:

# Define a constant tensor -> 0D Tensor
scalar = tf.constant(2)
scalar

# Define a constant tensor -> 0D Tensor scalar = tf.constant(2) scalar

Out[ ]:

<tf.Tensor: shape=(), dtype=int32, numpy=2>

In [ ]:

# Check the dimensions
scalar.ndim

# Check the dimensions scalar.ndim

Out[ ]:

0

In [ ]:

# Check the shape
scalar.shape

# Check the shape scalar.shape

Out[ ]:

TensorShape([])

In [ ]:

# Let's create a vector! -> 1D tensor
row_vector = tf.constant([10, 10])
row_vector

# Let's create a vector! -> 1D tensor row_vector = tf.constant([10, 10]) row_vector

Out[ ]:

<tf.Tensor: shape=(2,), dtype=int32, numpy=array([10, 10])>

In [ ]:

row_vector.ndim

row_vector.ndim

Out[ ]:

1

In [ ]:

row_vector.shape

row_vector.shape

Out[ ]:

TensorShape([2])

Transposing a 1D tensor has no effect!

In [ ]:

tf.transpose(row_vector)

tf.transpose(row_vector)

Out[ ]:

<tf.Tensor: shape=(2,), dtype=int32, numpy=array([10, 10])>

In [ ]:

# Let's create a column vector -> 2D tensor now!
column_vector = tf.constant([[10], [10]])
column_vector

# Let's create a column vector -> 2D tensor now! column_vector = tf.constant([[10], [10]]) column_vector

Out[ ]:

<tf.Tensor: shape=(2, 1), dtype=int32, numpy=
array([[10],
       [10]])>

In [ ]:

# Another way to make a column vector
column_vector = tf.transpose(tf.constant([[10, 10]]))  # Notice double square brackets
column_vector

# Another way to make a column vector column_vector = tf.transpose(tf.constant([[10, 10]])) # Notice double square brackets column_vector

Out[ ]:

<tf.Tensor: shape=(2, 1), dtype=int32, numpy=
array([[10],
       [10]])>

In [ ]:

column_vector.ndim

column_vector.ndim

Out[ ]:

2

In [ ]:

column_vector.shape

column_vector.shape

Out[ ]:

TensorShape([2, 1])

In [ ]:

# You can squeeze out the "redundant" dimensions
tf.squeeze(column_vector)

# You can squeeze out the "redundant" dimensions tf.squeeze(column_vector)

Out[ ]:

<tf.Tensor: shape=(2,), dtype=int32, numpy=array([10, 10])>

In [ ]:

# Or you can add an extra dimension by tf.newaxis
row_vector[:, tf.newaxis]

# Or you can add an extra dimension by tf.newaxis row_vector[:, tf.newaxis]

Out[ ]:

<tf.Tensor: shape=(2, 1), dtype=int32, numpy=
array([[10],
       [10]])>

In [ ]:

# or doing this
tf.expand_dims(row_vector, axis=-1)

# or doing this tf.expand_dims(row_vector, axis=-1)

Out[ ]:

<tf.Tensor: shape=(2, 1), dtype=int32, numpy=
array([[10],
       [10]])>

In [ ]:

# Or doing this
tf.reshape(row_vector, shape=(*row_vector.shape, 1))

# Or doing this tf.reshape(row_vector, shape=(*row_vector.shape, 1))

Out[ ]:

<tf.Tensor: shape=(2, 1), dtype=int32, numpy=
array([[10],
       [10]])>

In [ ]:

# Let's create a matrix -> 2D Tensor
matrix = tf.constant([[1, 2],
                      [3, 4],
                      [5, 6]])
matrix

# Let's create a matrix -> 2D Tensor matrix = tf.constant([[1, 2], [3, 4], [5, 6]]) matrix

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 2],
       [3, 4],
       [5, 6]])>

In [ ]:

# Check the dimensions
matrix.ndim

# Check the dimensions matrix.ndim

Out[ ]:

2

In [ ]:

# Check the shape
matrix.shape

# Check the shape matrix.shape

Out[ ]:

TensorShape([3, 2])

In [ ]:

# Another matrix
another_matrix = tf.constant([[1, 2, 3],
                              [4, 5, 6]], dtype=tf.float16) # Specify the dtype
another_matrix

# Another matrix another_matrix = tf.constant([[1, 2, 3], [4, 5, 6]], dtype=tf.float16) # Specify the dtype another_matrix

Out[ ]:

<tf.Tensor: shape=(2, 3), dtype=float16, numpy=
array([[1., 2., 3.],
       [4., 5., 6.]], dtype=float16)>

In [ ]:

# Now let's create a "real" tensor aka more than 2 dimensions!
tensor = tf.constant([[[1, 2, 3],
                       [4, 5, 6],
                       [7, 8, 9]]], 
                     dtype=tf.int32)

tensor

# Now let's create a "real" tensor aka more than 2 dimensions! tensor = tf.constant([[[1, 2, 3], [4, 5, 6], [7, 8, 9]]], dtype=tf.int32) tensor

Out[ ]:

<tf.Tensor: shape=(1, 3, 3), dtype=int32, numpy=
array([[[1, 2, 3],
        [4, 5, 6],
        [7, 8, 9]]])>

In [ ]:

# Check the number of dims
tensor.ndim

# Check the number of dims tensor.ndim

Out[ ]:

3

In [ ]:

# Check the shape
tensor.shape

# Check the shape tensor.shape

Out[ ]:

TensorShape([1, 3, 3])

What we have created so far:

• Scalar -> 0D Tensor
• Vector (Row) - > 1D Tensor
• Vector (Column) -> 2D Tensor
• Matrix -> 2D Tensor
• Tensor -> A generalisation in n-dimensions -> nD Tensor

Use case of Tensor?

Series of images in a training set, trained in batch (BATCH_SIZE, WIDTH, HEIGHT, CHANNELS) = (32, 224, 224, 3)

• Batch size taken as 32 (Faster Gradient Descent, Batches fit in memory, data augmentaion in batches, inference/prediction in batches)
• 224 - Height of the image
• 224 - Width of the image
• 3 - RGB Channels

Suppose we have 6400 RGB images of arbitrary size (Varying size)

• Reshape all images to be of a constant size (224, 224)
• Now the dataset Tensor would have the dimension: (6400, 224, 224, 3)
• Divide into batches based on batch_size = 32 (power of 2 to better fit in the memory, not sure why? hmmm..)
• New dimensions -> (32, 224, 224, 3)

Creating tensors with tf.Variable¶

In [ ]:

changeable_tensor = tf.Variable([1, 2])
unchangeable_tensor = tf.constant([1, 2])
changeable_tensor, unchangeable_tensor

changeable_tensor = tf.Variable([1, 2]) unchangeable_tensor = tf.constant([1, 2]) changeable_tensor, unchangeable_tensor

Out[ ]:

(<tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([1, 2])>,
 <tf.Tensor: shape=(2,), dtype=int32, numpy=array([1, 2])>)

In [ ]:

# Let's try changing one of the element's value in changeable tensor
changeable_tensor[0] = 3

# Let's try changing one of the element's value in changeable tensor changeable_tensor[0] = 3

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-25-fccf5ec3691e> in <module>
      1 # Let's try changing one of the element's value in changeable tensor
----> 2 changeable_tensor[0] = 3

TypeError: 'ResourceVariable' object does not support item assignment

In [ ]:

# To change the value we need to use .assign()
changeable_tensor[0].assign(3)
changeable_tensor

# To change the value we need to use .assign() changeable_tensor[0].assign(3) changeable_tensor

Out[ ]:

<tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([3, 2])>

In [ ]:

# How about we try changing the value in unchangeable tensor
unchangeable_tensor[0] = 3

# How about we try changing the value in unchangeable tensor unchangeable_tensor[0] = 3

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-aa4e85b866f0> in <module>
      1 # How about we try changing the value in unchangeable tensor
----> 2 unchangeable_tensor[0] = 3

TypeError: 'tensorflow.python.framework.ops.EagerTensor' object does not support item assignment

In [ ]:

# How about we change values in unchangeable tensor?
unchangeable_tensor[0].assign(3)

# How about we change values in unchangeable tensor? unchangeable_tensor[0].assign(3)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-28-083f190121c0> in <module>
      1 # How about we change values in unchangeable tensor?
----> 2 unchangeable_tensor[0].assign(3)

AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'assign'

Because this is constant tensor!

Create random tensors¶

In [ ]:

# Create two random (but the same) tensors!
randgen1 = tf.random.Generator.from_seed(42)
randval1 = randgen1.normal(shape=(3, 2))
randgen2 = tf.random.Generator.from_seed(42)
randval2 = randgen2.normal(shape=(3, 2))

# Let's check if both are equal
randval1 == randval2

# Create two random (but the same) tensors! randgen1 = tf.random.Generator.from_seed(42) randval1 = randgen1.normal(shape=(3, 2)) randgen2 = tf.random.Generator.from_seed(42) randval2 = randgen2.normal(shape=(3, 2)) # Let's check if both are equal randval1 == randval2

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=bool, numpy=
array([[ True,  True],
       [ True,  True],
       [ True,  True]])>

In [ ]:

# Create two random (not the same) tensors!
randgen1 = tf.random.Generator.from_seed(42)
randval1 = randgen1.normal(shape=(3, 2))
randgen2 = tf.random.Generator.from_seed(32)
randval2 = randgen2.normal(shape=(3, 2))

# Let's check their equality
randval1 == randval2

# Create two random (not the same) tensors! randgen1 = tf.random.Generator.from_seed(42) randval1 = randgen1.normal(shape=(3, 2)) randgen2 = tf.random.Generator.from_seed(32) randval2 = randgen2.normal(shape=(3, 2)) # Let's check their equality randval1 == randval2

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=bool, numpy=
array([[False, False],
       [False, False],
       [False, False]])>

In [ ]:

# Normal distribution of the randomly sampled tensor
import matplotlib.pyplot as plt
norm_tensor = tf.random.normal(shape=(1000,))
plt.hist(norm_tensor);

# Normal distribution of the randomly sampled tensor import matplotlib.pyplot as plt norm_tensor = tf.random.normal(shape=(1000,)) plt.hist(norm_tensor);

Shuffle the orders of elements in a tensor¶

• If we have first 10000 examples as Ramen and next 5000 examples as Spaghetti
• Shuffling would be useful here to assure good mix of each in each mini batch
• Remember, each batch takes a single step of gradient step assuming the batch represents the parent probability distribution
• Each batch must have the same mix of classes/probability distribution
  • Same probability distribution can also be linked to clusters present in the images

In [ ]:

# Create a tensor
not_shuffled = tf.constant([[1, 2],
                            [3, 4],
                            [5, 6]])
not_shuffled

# Create a tensor not_shuffled = tf.constant([[1, 2], [3, 4], [5, 6]]) not_shuffled

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 2],
       [3, 4],
       [5, 6]])>

In [ ]:

# Shuffle our non shuffled tensor (keep running this cell to see how 'its shuffling)
tf.random.set_seed(42)  # If you remove this seed, every shuffle will be new
tf.random.shuffle(not_shuffled, seed=12)

# Shuffle our non shuffled tensor (keep running this cell to see how 'its shuffling) tf.random.set_seed(42) # If you remove this seed, every shuffle will be new tf.random.shuffle(not_shuffled, seed=12)

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[5, 6],
       [3, 4],
       [1, 2]])>

Note: Shuffling is being done accross the first dimension i.e. rows are being interchanged

Read about set_seed:

• Global seed
• Operation level seed

In [ ]:

# Only operation level seed is set
# Even with seed set, you would get different result each time
tf.random.shuffle(not_shuffled, seed=42)

# Only operation level seed is set # Even with seed set, you would get different result each time tf.random.shuffle(not_shuffled, seed=42)

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 2],
       [3, 4],
       [5, 6]])>

In [ ]:

# Only global level seed is set (Same result every time)
tf.random.set_seed(42)
tf.random.shuffle(not_shuffled)

# Only global level seed is set (Same result every time) tf.random.set_seed(42) tf.random.shuffle(not_shuffled)

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[3, 4],
       [5, 6],
       [1, 2]])>

In [ ]:

# Both (Same result everytime)
tf.random.set_seed(42) 
tf.random.shuffle(not_shuffled, seed=42)

# Both (Same result everytime) tf.random.set_seed(42) tf.random.shuffle(not_shuffled, seed=42)

Out[ ]:

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 2],
       [3, 4],
       [5, 6]])>

Other ways of making a tensor¶

In [ ]:

# Create a tensor of ones
tf.ones(shape=(10, 7))

# Create a tensor of ones tf.ones(shape=(10, 7))

Out[ ]:

<tf.Tensor: shape=(10, 7), dtype=float32, numpy=
array([[1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1.]], dtype=float32)>

In [ ]:

# Create a tensor of all zeros
tf.zeros(shape=(10, 7))

# Create a tensor of all zeros tf.zeros(shape=(10, 7))

Out[ ]:

<tf.Tensor: shape=(10, 7), dtype=float32, numpy=
array([[0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0.]], dtype=float32)>

Turn numpy arrays into tensors¶

Tensors can be operated must faster on by GPUs

In [ ]:

import numpy as np
numpy_A = np.arange(1, 25, dtype=np.int32)
numpy_A

import numpy as np numpy_A = np.arange(1, 25, dtype=np.int32) numpy_A

Out[ ]:

array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24])

In [ ]:

A = tf.constant(numpy_A, shape=(2, 3, 4))
B = tf.constant(numpy_A)

dict(zip(['A', 'B'], [A, B]))

A = tf.constant(numpy_A, shape=(2, 3, 4)) B = tf.constant(numpy_A) dict(zip(['A', 'B'], [A, B]))

Out[ ]:

{'A': <tf.Tensor: shape=(2, 3, 4), dtype=int32, numpy=
 array([[[ 1,  2,  3,  4],
         [ 5,  6,  7,  8],
         [ 9, 10, 11, 12]],
 
        [[13, 14, 15, 16],
         [17, 18, 19, 20],
         [21, 22, 23, 24]]])>,
 'B': <tf.Tensor: shape=(24,), dtype=int32, numpy=
 array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24])>}

In [ ]:

dict(zip(['A', 'B'], [A.ndim, B.ndim]))

dict(zip(['A', 'B'], [A.ndim, B.ndim]))

Out[ ]:

{'A': 3, 'B': 1}

Getting Information from the tensors¶

In [ ]:

from src.utils import describe_tensor

from src.utils import describe_tensor

In [ ]:

describe_tensor(tensor)

describe_tensor(tensor)

Datatype: <dtype: 'int32'>
Number of dimensions: 3
Shape of tensor: (1, 3, 3)
Elements along the 0 axis: 1
Elements along the last axis: 3
Total number of elements: 9

Indexing tensors¶

Tensors can be indexed just like Python lists

In [ ]:

# Set the seed
tf.random.set_seed(42)
tensor4 = tf.random.normal(shape=(32, 8, 8, 3), seed=42) # A single batch of 8x8 RGB Image (32)
tensor4[0, :, :, 1] # (1st batch 1st image and Green channel)

# Set the seed tf.random.set_seed(42) tensor4 = tf.random.normal(shape=(32, 8, 8, 3), seed=42) # A single batch of 8x8 RGB Image (32) tensor4[0, :, :, 1] # (1st batch 1st image and Green channel)

Out[ ]:

<tf.Tensor: shape=(8, 8), dtype=float32, numpy=
array([[-0.15421568, -0.10875293, -0.621129  , -1.0798668 , -0.16909476,
         0.8410001 , -1.6971248 , -1.7839274 ],
       [ 1.1488945 ,  0.95193976,  0.10488074,  0.21458012,  0.55202   ,
         0.34291965, -0.8316435 , -0.9340011 ],
       [ 1.2897398 , -0.13424467,  1.8755274 , -0.21415219, -0.36870828,
         0.67288345,  1.3695234 , -2.167099  ],
       [-1.1596808 ,  0.38832134, -0.46657825,  0.23319368, -1.4149377 ,
        -0.5536773 ,  1.2187223 , -1.1173725 ],
       [-0.7941299 ,  0.8506392 ,  0.9731572 ,  0.9786592 , -0.40033567,
        -1.2027842 ,  0.9958395 , -2.1588895 ],
       [-1.3097545 , -1.7249739 , -0.88928556, -0.8335409 ,  0.11488265,
         0.09552945, -0.30543768,  0.6083008 ],
       [-0.20530795,  1.4380887 ,  0.64798355,  0.26897132, -0.7120822 ,
        -0.92031354, -1.5436949 , -0.20956384],
       [ 0.11213502,  0.13058572,  0.46756333,  0.81544125,  0.69696605,
         0.4109041 ,  0.5073107 , -0.25794792]], dtype=float32)>

In [ ]:

tensor3 = tf.random.normal(shape=(10, 10, 3)) # A single data point with 10x10 and 3 RGB channels
tensor3

tensor3 = tf.random.normal(shape=(10, 10, 3)) # A single data point with 10x10 and 3 RGB channels tensor3

Out[ ]:

<tf.Tensor: shape=(10, 10, 3), dtype=float32, numpy=
array([[[ 3.27468514e-01, -8.42625797e-01,  3.19433689e-01],
        [-1.40755188e+00, -2.38805985e+00, -1.03924787e+00],
        [-5.57323217e-01,  5.39707005e-01,  1.69943225e+00],
        [ 2.88936555e-01, -1.50661159e+00, -2.64547408e-01],
        [-5.97224057e-01, -1.91711318e+00, -6.20441437e-01],
        [ 8.50402296e-01, -4.06047940e-01, -3.02584124e+00],
        [ 9.05846417e-01,  2.98559874e-01, -2.25615546e-01],
        [-7.61644304e-01, -1.89171410e+00, -9.38471258e-01],
        [ 7.78522134e-01, -4.73388970e-01,  9.77726936e-01],
        [ 2.46944040e-01,  2.05737472e-01, -5.25623322e-01]],

       [[ 3.24100167e-01,  2.54540909e-02, -1.06384970e-01],
        [-6.36947513e-01,  1.16031218e+00,  2.50735909e-01],
        [-4.17285025e-01,  4.01257813e-01, -1.41454434e+00],
        [-5.93185723e-01, -1.66172135e+00,  3.35671932e-01],
        [ 1.08156286e-01,  2.34796822e-01, -5.66687644e-01],
        [-3.58198434e-01,  8.86986136e-01,  5.27447641e-01],
        [ 7.04022467e-01, -3.34212482e-01,  2.16396436e-01],
        [-9.74854469e-01, -2.07576811e-01, -3.64772938e-02],
        [-1.33534443e+00,  6.88585520e-01,  1.11108327e+00],
        [ 4.01302516e-01,  6.32058620e-01, -3.90306145e-01]],

       [[ 1.46120000e+00, -6.33797646e-01, -9.99705136e-01],
        [-8.80446867e-04,  1.07158554e+00,  4.81234878e-01],
        [-1.60123384e+00, -1.22025335e+00, -2.60341227e-01],
        [-1.49422154e-01,  1.12653100e+00, -1.04361320e+00],
        [ 1.15956819e+00,  5.13993859e-01, -1.08452451e+00],
        [-3.13049525e-01,  4.25618500e-01, -1.80582374e-01],
        [-1.93230391e-01,  9.52812552e-01,  2.00572729e+00],
        [-5.95110774e-01,  2.72236496e-01,  2.42677748e-01],
        [ 1.50638473e+00, -1.16431601e-01, -1.53971851e+00],
        [ 2.13203907e+00, -4.65701789e-01, -8.20223927e-01]],

       [[-1.73188818e+00, -3.60235900e-01,  4.11543936e-01],
        [ 1.36178446e+00, -1.62113667e+00,  1.08125830e+00],
        [ 2.05486983e-01,  3.01658601e-01, -1.56354535e+00],
        [-1.56779742e+00, -1.22952834e-01, -1.40379882e+00],
        [-9.19952035e-01,  3.14764291e-01,  1.86195040e+00],
        [ 3.44231427e-01,  5.05643845e-01,  8.28737676e-01],
        [-1.61130369e-01,  3.71081114e-01,  1.29466414e-01],
        [-9.94873166e-01,  1.74755239e+00, -9.04302478e-01],
        [-3.88163298e-01,  6.33158982e-02, -6.85345709e-01],
        [-6.58035338e-01,  7.37734735e-01, -1.23796630e+00]],

       [[ 1.62680459e+00,  1.32617843e+00, -1.69563806e+00],
        [-6.77775323e-01,  8.08262169e-01,  6.01236662e-03],
        [ 1.55219166e-02, -1.02510095e+00,  3.21554810e-01],
        [ 3.35006177e-01,  1.99884188e+00, -6.39790371e-02],
        [-1.16190445e+00, -2.06357241e-01, -4.25627649e-01],
        [-1.06816816e+00,  1.03230691e+00,  7.80556276e-02],
        [-3.13318431e-01,  3.00168991e-01, -1.36430621e-01],
        [-7.64365673e-01,  1.39826238e-01,  1.70999169e+00],
        [-3.41805756e-01, -8.70469689e-01, -1.26517296e+00],
        [ 1.02162230e+00, -1.75407693e-01,  3.12556088e-01]],

       [[ 1.88517764e-01,  5.91816485e-01, -3.31230760e-01],
        [ 2.09661365e-01,  1.20092726e+00,  9.18924987e-01],
        [-6.76153719e-01,  2.41165683e-02, -8.36931944e-01],
        [ 1.70226169e+00,  7.32276738e-01,  1.09463346e+00],
        [-1.87608957e-01, -1.27563620e+00, -6.72676146e-01],
        [ 5.89388490e-01, -1.37254393e+00,  9.94733334e-01],
        [-1.80882025e+00,  8.75771716e-02, -2.24898958e+00],
        [ 2.19127044e-01,  1.31176460e+00,  9.05483484e-01],
        [-9.09993410e-01, -7.26843059e-01,  4.21907365e-01],
        [ 9.13502455e-01,  1.21766925e+00, -1.75209093e+00]],

       [[ 5.84937513e-01,  8.17803264e-01,  1.06913841e+00],
        [-1.09194946e+00,  5.02991199e-01, -5.13924718e-01],
        [ 2.96165347e-01,  1.63955712e+00,  5.22935867e-01],
        [-9.49233115e-01,  6.14328742e-01, -2.32862487e-01],
        [ 1.67191255e+00,  1.48186409e+00,  4.84806210e-01],
        [-2.24033189e+00, -2.25851297e+00, -1.31739661e-01],
        [ 1.62239477e-01, -9.51777101e-01, -7.06969798e-02],
        [ 6.34097397e-01, -2.64249146e-01, -6.25652552e-01],
        [-5.65087080e-01,  6.89760566e-01,  8.32360744e-01],
        [-1.43979168e+00,  1.61607251e-01,  4.59538937e-01]],

       [[ 5.18101096e-01, -2.85835471e-02, -8.67913723e-01],
        [ 1.16980612e+00, -1.99054360e+00,  8.92418504e-01],
        [ 1.88958645e+00,  1.25957215e+00, -2.31702164e-01],
        [ 6.22233927e-01,  7.59622931e-01, -8.03967595e-01],
        [ 1.20707381e+00,  1.27932549e-01,  6.21761382e-02],
        [ 2.73475528e-01,  2.31515765e-01,  7.18649089e-01],
        [ 1.79765952e+00,  3.97182614e-01, -9.28808212e-01],
        [ 2.62536436e-01, -3.34717870e-01, -4.77222323e-01],
        [-1.22534132e+00,  9.03180897e-01, -3.81502199e+00],
        [-1.28228271e+00, -2.47924656e-01, -1.04075015e+00]],

       [[-7.70410240e-01, -4.94718462e-01, -2.21131873e+00],
        [ 3.89290005e-01,  1.06228900e+00, -1.55220702e-01],
        [-1.17830861e+00,  2.99320787e-01,  3.05507690e-01],
        [ 1.87913805e-01,  6.70983553e-01, -7.76138484e-01],
        [-2.61710584e-01, -1.75008214e+00, -5.93035877e-01],
        [ 1.14544742e-01, -7.46733129e-01,  1.28347233e-01],
        [-5.16174585e-02, -9.19692278e-01,  1.22334182e+00],
        [ 3.37343067e-01,  6.64462686e-01,  8.24578702e-01],
        [ 4.70071197e-01,  1.55162755e-02, -1.79089531e-01],
        [-1.09421313e+00, -1.28419173e+00, -3.32507908e-01]],

       [[ 6.37807190e-01,  2.82803416e-01,  4.53875184e-01],
        [-6.23685002e-01,  8.13746095e-01,  5.20318687e-01],
        [-1.27764165e+00, -2.52391249e-01,  1.60077167e+00],
        [-8.25679183e-01, -9.36846018e-01, -1.26432431e+00],
        [-4.10377026e-01,  2.72557080e-01, -3.34020674e-01],
        [-2.53592461e-01,  2.93609679e-01, -4.02389169e-01],
        [ 1.35541812e-01, -1.27433807e-01, -4.95703012e-01],
        [-5.89106500e-01, -3.60340364e-02, -1.25153184e-01],
        [-1.21073723e+00,  4.19056676e-02,  9.10298705e-01],
        [-3.87856245e-01,  1.70907766e-01,  7.22056925e-01]]],
      dtype=float32)>

In [ ]:

# Add an extra dimension in beginning 
# Often done to reshape data point as model expects a tensor of this shape (None, 10, 10, 3)
tensor4 = tensor3[tf.newaxis, ...]
tensor4.shape

# Add an extra dimension in beginning # Often done to reshape data point as model expects a tensor of this shape (None, 10, 10, 3) tensor4 = tensor3[tf.newaxis, ...] tensor4.shape

Out[ ]:

TensorShape([1, 10, 10, 3])

In [ ]:

# The above thing works with numpy as well
shape = (10, 10, 3)
arr3 = np.random.normal(size=shape)
arr4 = arr3[np.newaxis, ...]
arr4.shape

# The above thing works with numpy as well shape = (10, 10, 3) arr3 = np.random.normal(size=shape) arr4 = arr3[np.newaxis, ...] arr4.shape

Out[ ]:

(1, 10, 10, 3)

Manipulating Tensors (tensor operations)¶

Basic operations

• Operator overloading i.e +-/*@

In [ ]:

# You can add values to a tensor using the addition operator
tensor = tf.constant([1, 2, 3, 4], shape=(2, 2))
tensor += 10
tensor

# You can add values to a tensor using the addition operator tensor = tf.constant([1, 2, 3, 4], shape=(2, 2)) tensor += 10 tensor

Out[ ]:

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[11, 12],
       [13, 14]])>

In [ ]:

tensor = tf.constant([1, 2, 3, 4], shape=(2, 2))
tensor *= 10

tensor = tf.constant([1, 2, 3, 4], shape=(2, 2)) tensor *= 10

In [ ]:

# In addition to operator overloading, we can use builtin functions
tf.multiply(tensor, 10)

# In addition to operator overloading, we can use builtin functions tf.multiply(tensor, 10)

Out[ ]:

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[100, 200],
       [300, 400]])>

Matrix Multiplication¶

Most common operation in whole of machine learning and deep learning

Think about matrix multiplication in the following ways \ Justifying following points:

• matching inner dimensions
• Output has the shape of outer dimensions

Contexts:

• In context of linear transformation (and relate this to below contexts)
• In context of linear regression
• In context of neural network layers and neurons and their associated dataset
• In context of batch training of RGB images, sentences, word embeddings etc
• As a way of contraction of certain dimensions. Psst - tensor dot product is also known as tensor contraction

Explanations:

• matrix A times column vector V
  • Linear combination of column vectors of matrix A using weights of column vector V
  • Matrix A is a datasets with each data point stacked as rows, and their corresponding features in each column. Then, the column vector provides how important, or rather the weight of each feature. (so dot product of each row with the column vector)
• matrix A times matrix B
  • aa
• aa

Two scenarios (for quick thinking remember outer dimensions are preserved):

• $WX$ - Data points as columns, neurons as rows
• $XW$ - Data points as rows, neurons as columns

In [ ]:

print(tensor)

print(tensor)

tf.Tensor(
[[10 20]
 [30 40]], shape=(2, 2), dtype=int32)

In [ ]:

# Matrix multiplication in tensorflow
tf.matmul(tensor, tensor)

# Matrix multiplication in tensorflow tf.matmul(tensor, tensor)

Out[ ]:

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[ 700, 1000],
       [1500, 2200]])>

In [ ]:

tensor @ tensor

tensor @ tensor

Out[ ]:

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[ 700, 1000],
       [1500, 2200]])>

In [ ]:

# Create a tensor (3, 2) tensor.
X = tf.constant([[1, 2],
                 [3, 4],
                 [5, 6]])

Y = tf.constant([[7, 8],
                 [9, 10],
                 [11, 12]])
X, Y

# Create a tensor (3, 2) tensor. X = tf.constant([[1, 2], [3, 4], [5, 6]]) Y = tf.constant([[7, 8], [9, 10], [11, 12]]) X, Y

Out[ ]:

(<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
 array([[1, 2],
        [3, 4],
        [5, 6]])>,
 <tf.Tensor: shape=(3, 2), dtype=int32, numpy=
 array([[ 7,  8],
        [ 9, 10],
        [11, 12]])>)

In [ ]:

tf.matmul(X, Y)

tf.matmul(X, Y)

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-63-b58f5d491930> in <module>
----> 1 tf.matmul(X, Y)

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\util\dispatch.py in wrapper(*args, **kwargs)
    199     """Call target, and fall back on dispatchers if there is a TypeError."""
    200     try:
--> 201       return target(*args, **kwargs)
    202     except (TypeError, ValueError):
    203       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\ops\math_ops.py in matmul(a, b, transpose_a, transpose_b, adjoint_a, adjoint_b, a_is_sparse, b_is_sparse, name)
   3313     else:
   3314       return gen_math_ops.mat_mul(
-> 3315           a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
   3316 
   3317 

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\ops\gen_math_ops.py in mat_mul(a, b, transpose_a, transpose_b, name)
   5529       return _result
   5530     except _core._NotOkStatusException as e:
-> 5531       _ops.raise_from_not_ok_status(e, name)
   5532     except _core._FallbackException:
   5533       pass

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\framework\ops.py in raise_from_not_ok_status(e, name)
   6860   message = e.message + (" name: " + name if name is not None else "")
   6861   # pylint: disable=protected-access
-> 6862   six.raise_from(core._status_to_exception(e.code, message), None)
   6863   # pylint: enable=protected-access
   6864 

~\anaconda3\envs\ds-py37\lib\site-packages\six.py in raise_from(value, from_value)

InvalidArgumentError: Matrix size-incompatible: In[0]: [3,2], In[1]: [3,2] [Op:MatMul]

In [ ]:

# Let's change the shape of Y
tf.reshape(Y, shape=(2, 3))

# Let's change the shape of Y tf.reshape(Y, shape=(2, 3))

Out[ ]:

<tf.Tensor: shape=(2, 3), dtype=int32, numpy=
array([[ 7,  8,  9],
       [10, 11, 12]])>

In [ ]:

# Different from tf.transpose
tf.transpose(Y)

# Different from tf.transpose tf.transpose(Y)

Out[ ]:

<tf.Tensor: shape=(2, 3), dtype=int32, numpy=
array([[ 7,  9, 11],
       [ 8, 10, 12]])>

In [ ]:

# now try to multiply X by Y
X @ tf.reshape(Y, shape=(2, 3))

# now try to multiply X by Y X @ tf.reshape(Y, shape=(2, 3))

Out[ ]:

<tf.Tensor: shape=(3, 3), dtype=int32, numpy=
array([[ 27,  30,  33],
       [ 61,  68,  75],
       [ 95, 106, 117]])>

Dot product¶

• Matrix multiplication can also be expressed as a dot product
• You can perform the matrix multiplication using:
  • tensor.matmul()
  • tensor.tensordot()

In [ ]:

tf.tensordot(tf.transpose(X), Y, axes=1)

tf.tensordot(tf.transpose(X), Y, axes=1)

Out[ ]:

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[ 89,  98],
       [116, 128]])>

Generally when performing matrix multiplication between two tensors, and one of the dimensions do not line up, you would transpose one of the tensors and NOT reshape.

In [ ]:

# Create a new tensor with default datatype (float32)
B = tf.constant([1.7, 7.4])
B.dtype

# Create a new tensor with default datatype (float32) B = tf.constant([1.7, 7.4]) B.dtype

Out[ ]:

tf.float32

In [ ]:

C = tf.constant([7, 10])
C.dtype

C = tf.constant([7, 10]) C.dtype

Out[ ]:

tf.int32

In [ ]:

# Change from float32 to float16 (reduced precision but high performance gains + less memory requirements)
D = tf.cast(B, dtype=tf.float16)
D

# Change from float32 to float16 (reduced precision but high performance gains + less memory requirements) D = tf.cast(B, dtype=tf.float16) D

Out[ ]:

<tf.Tensor: shape=(2,), dtype=float16, numpy=array([1.7, 7.4], dtype=float16)>

In [ ]:

E = tf.cast(C, dtype=tf.float32)
E

E = tf.cast(C, dtype=tf.float32) E

Out[ ]:

<tf.Tensor: shape=(2,), dtype=float32, numpy=array([ 7., 10.], dtype=float32)>

Aggregating tensors¶

Use cases:

• Implementing softmax (raise to e -> aggregated sum -> divide by sum)
• Standardizing tensors
• Removing noise (subtract mean noise from images)
• Batch normalization
• Mean prediction
• Argmax prediction
• Evaluating predictions
• All neural network operations
  • weighted linear combination
  • then summing (aggregation)
  • then non linearity (activation)

In [ ]:

# Creating a tensor
tensor = tf.constant([-1, 2])
tensor

# Creating a tensor tensor = tf.constant([-1, 2]) tensor

Out[ ]:

<tf.Tensor: shape=(2,), dtype=int32, numpy=array([-1,  2])>

In [ ]:

# Get the absolute values
tf.abs(tensor)

# Get the absolute values tf.abs(tensor)

Out[ ]:

<tf.Tensor: shape=(2,), dtype=int32, numpy=array([1, 2])>

In [ ]:

import tensorflow_probability as tfp

import tensorflow_probability as tfp

In [ ]:

E = tf.constant(np.random.randint(0, 100, size=50))
E

E = tf.constant(np.random.randint(0, 100, size=50)) E

Out[ ]:

<tf.Tensor: shape=(50,), dtype=int32, numpy=
array([75, 36, 27, 28, 30, 18, 74, 10, 85, 30, 85, 31, 32, 48, 79, 65, 34,
       45, 35, 28, 84, 40, 59, 23, 48, 35, 12, 18, 17, 31, 17, 90, 87, 38,
       20,  5, 69, 56, 54,  9, 89, 87, 59, 63, 82, 68, 90, 47, 14, 49])>

In [ ]:

def tensor_variance(tensor):
    mu = tf.reduce_mean(tensor)
    var = tf.reduce_sum((tensor - mu)**2)/tf.shape(tensor).numpy()
    return tf.squeeze(var)

def tensor_variance(tensor): mu = tf.reduce_mean(tensor) var = tf.reduce_sum((tensor - mu)**2)/tf.shape(tensor).numpy() return tf.squeeze(var)

In [ ]:

print('min:', tf.reduce_min(E))
print('max:', tf.reduce_max(E))
print('mean:', tf.reduce_mean(E))
print('sum:', tf.reduce_sum(E))
print('std:', tf.math.reduce_std(tf.cast(E, dtype=tf.float32)))
print('variance:', tf.math.reduce_variance(tf.cast(E, dtype=tf.float32)))
print('variance:', tensor_variance(E))
print('variance:', tfp.stats.variance(E))

print('min:', tf.reduce_min(E)) print('max:', tf.reduce_max(E)) print('mean:', tf.reduce_mean(E)) print('sum:', tf.reduce_sum(E)) print('std:', tf.math.reduce_std(tf.cast(E, dtype=tf.float32))) print('variance:', tf.math.reduce_variance(tf.cast(E, dtype=tf.float32))) print('variance:', tensor_variance(E)) print('variance:', tfp.stats.variance(E))

min: tf.Tensor(5, shape=(), dtype=int32)
max: tf.Tensor(90, shape=(), dtype=int32)
mean: tf.Tensor(47, shape=(), dtype=int32)
sum: tf.Tensor(2355, shape=(), dtype=int32)
std: tf.Tensor(25.859814, shape=(), dtype=float32)
variance: tf.Tensor(668.73, shape=(), dtype=float32)
variance: tf.Tensor(668.74, shape=(), dtype=float64)
variance: tf.Tensor(668, shape=(), dtype=int32)

In [ ]:

# NOTE: requires real or complex 
# Since our E tensor is int32, we would have to cast it as above
try:
    tf.math.reduce_std(E) 
except Exception as e:
    print(e)

# NOTE: requires real or complex # Since our E tensor is int32, we would have to cast it as above try: tf.math.reduce_std(E) except Exception as e: print(e)

Input must be either real or complex

Find the positional maximum and minumum

In [ ]:

# Create a new tensor 
tf.random.set_seed(42)
tensor = tf.random.uniform(shape=(50,))
tensor

# Create a new tensor tf.random.set_seed(42) tensor = tf.random.uniform(shape=(50,)) tensor

Out[ ]:

<tf.Tensor: shape=(50,), dtype=float32, numpy=
array([0.6645621 , 0.44100678, 0.3528825 , 0.46448255, 0.03366041,
       0.68467236, 0.74011743, 0.8724445 , 0.22632635, 0.22319686,
       0.3103881 , 0.7223358 , 0.13318717, 0.5480639 , 0.5746088 ,
       0.8996835 , 0.00946367, 0.5212307 , 0.6345445 , 0.1993283 ,
       0.72942245, 0.54583454, 0.10756552, 0.6767061 , 0.6602763 ,
       0.33695042, 0.60141766, 0.21062577, 0.8527372 , 0.44062173,
       0.9485276 , 0.23752594, 0.81179297, 0.5263394 , 0.494308  ,
       0.21612847, 0.8457197 , 0.8718841 , 0.3083862 , 0.6868038 ,
       0.23764038, 0.7817228 , 0.9671384 , 0.06870162, 0.79873943,
       0.66028714, 0.5871513 , 0.16461694, 0.7381023 , 0.32054043],
      dtype=float32)>

In [ ]:

# Find the positional maximum
tf.argmax(tensor)

# Find the positional maximum tf.argmax(tensor)

Out[ ]:

<tf.Tensor: shape=(), dtype=int64, numpy=42>

In [ ]:

# Index on our largest value position
tensor[tf.argmax(tensor)]

# Index on our largest value position tensor[tf.argmax(tensor)]

Out[ ]:

<tf.Tensor: shape=(), dtype=float32, numpy=0.9671384>

In [ ]:

tf.reduce_max(tensor)

tf.reduce_max(tensor)

Out[ ]:

<tf.Tensor: shape=(), dtype=float32, numpy=0.9671384>

Squeeze a tensor (remove all single dimensions)¶

In [ ]:

tf.random.set_seed(42)
tensor = tf.constant(tf.random.uniform(shape=(10, 2)), shape=(1, 1, 1, 1, 10, 1, 2))
tensor

tf.random.set_seed(42) tensor = tf.constant(tf.random.uniform(shape=(10, 2)), shape=(1, 1, 1, 1, 10, 1, 2)) tensor

Out[ ]:

<tf.Tensor: shape=(1, 1, 1, 1, 10, 1, 2), dtype=float32, numpy=
array([[[[[[[0.6645621 , 0.44100678]],

           [[0.3528825 , 0.46448255]],

           [[0.03366041, 0.68467236]],

           [[0.74011743, 0.8724445 ]],

           [[0.22632635, 0.22319686]],

           [[0.3103881 , 0.7223358 ]],

           [[0.13318717, 0.5480639 ]],

           [[0.5746088 , 0.8996835 ]],

           [[0.00946367, 0.5212307 ]],

           [[0.6345445 , 0.1993283 ]]]]]]], dtype=float32)>

In [ ]:

tf.squeeze(tensor)

tf.squeeze(tensor)

Out[ ]:

<tf.Tensor: shape=(10, 2), dtype=float32, numpy=
array([[0.6645621 , 0.44100678],
       [0.3528825 , 0.46448255],
       [0.03366041, 0.68467236],
       [0.74011743, 0.8724445 ],
       [0.22632635, 0.22319686],
       [0.3103881 , 0.7223358 ],
       [0.13318717, 0.5480639 ],
       [0.5746088 , 0.8996835 ],
       [0.00946367, 0.5212307 ],
       [0.6345445 , 0.1993283 ]], dtype=float32)>

One Hot Encoding¶

In [ ]:

# Create a list of labels
some_list = [0, 1, 2, 3]
num_cat = 4
tf.one_hot(some_list, depth=num_cat)

# Create a list of labels some_list = [0, 1, 2, 3] num_cat = 4 tf.one_hot(some_list, depth=num_cat)

Out[ ]:

<tf.Tensor: shape=(4, 4), dtype=float32, numpy=
array([[1., 0., 0., 0.],
       [0., 1., 0., 0.],
       [0., 0., 1., 0.],
       [0., 0., 0., 1.]], dtype=float32)>

In [ ]:

# Sample from a multinomial distribution
# Imagine this being the labels Y for a supervised problem
num_cats = 5
probs = [0.1, 0.2, 0.1, 0.45, 0.15]
samples = tf.squeeze(tf.random.categorical(tf.math.log([probs]), 20))
samples

# Sample from a multinomial distribution # Imagine this being the labels Y for a supervised problem num_cats = 5 probs = [0.1, 0.2, 0.1, 0.45, 0.15] samples = tf.squeeze(tf.random.categorical(tf.math.log([probs]), 20)) samples

Out[ ]:

<tf.Tensor: shape=(20,), dtype=int64, numpy=
array([2, 1, 3, 4, 3, 3, 3, 0, 4, 2, 3, 2, 4, 0, 3, 1, 1, 3, 2, 4],
      dtype=int64)>

In [ ]:

tf.one_hot(samples, depth=num_cats)

tf.one_hot(samples, depth=num_cats)

Out[ ]:

<tf.Tensor: shape=(20, 5), dtype=float32, numpy=
array([[0., 0., 1., 0., 0.],
       [0., 1., 0., 0., 0.],
       [0., 0., 0., 1., 0.],
       [0., 0., 0., 0., 1.],
       [0., 0., 0., 1., 0.],
       [0., 0., 0., 1., 0.],
       [0., 0., 0., 1., 0.],
       [1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1.],
       [0., 0., 1., 0., 0.],
       [0., 0., 0., 1., 0.],
       [0., 0., 1., 0., 0.],
       [0., 0., 0., 0., 1.],
       [1., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0.],
       [0., 1., 0., 0., 0.],
       [0., 1., 0., 0., 0.],
       [0., 0., 0., 1., 0.],
       [0., 0., 1., 0., 0.],
       [0., 0., 0., 0., 1.]], dtype=float32)>

In [ ]:

tf.one_hot(samples, depth=num_cats, on_value='yes', off_value='no')

tf.one_hot(samples, depth=num_cats, on_value='yes', off_value='no')

Out[ ]:

<tf.Tensor: shape=(20, 5), dtype=string, numpy=
array([[b'no', b'no', b'yes', b'no', b'no'],
       [b'no', b'yes', b'no', b'no', b'no'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'no', b'no', b'no', b'no', b'yes'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'yes', b'no', b'no', b'no', b'no'],
       [b'no', b'no', b'no', b'no', b'yes'],
       [b'no', b'no', b'yes', b'no', b'no'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'no', b'no', b'yes', b'no', b'no'],
       [b'no', b'no', b'no', b'no', b'yes'],
       [b'yes', b'no', b'no', b'no', b'no'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'no', b'yes', b'no', b'no', b'no'],
       [b'no', b'yes', b'no', b'no', b'no'],
       [b'no', b'no', b'no', b'yes', b'no'],
       [b'no', b'no', b'yes', b'no', b'no'],
       [b'no', b'no', b'no', b'no', b'yes']], dtype=object)>

Square, log, square root¶

In [ ]:

tensor = tf.range(1, 10)
tensor

tensor = tf.range(1, 10) tensor

Out[ ]:

<tf.Tensor: shape=(9,), dtype=int32, numpy=array([1, 2, 3, 4, 5, 6, 7, 8, 9])>

In [ ]:

tf.square(tensor)

tf.square(tensor)

Out[ ]:

<tf.Tensor: shape=(9,), dtype=int32, numpy=array([ 1,  4,  9, 16, 25, 36, 49, 64, 81])>

In [ ]:

# Square root will require a real (float) or complex -> on int it won't work
tf.sqrt(tensor)

# Square root will require a real (float) or complex -> on int it won't work tf.sqrt(tensor)

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-109-6d0ffb6c5cdf> in <module>
      1 # Square root will require a real (float) or complex -> on int it won't work
----> 2 tf.sqrt(tensor)

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\util\dispatch.py in wrapper(*args, **kwargs)
    199     """Call target, and fall back on dispatchers if there is a TypeError."""
    200     try:
--> 201       return target(*args, **kwargs)
    202     except (TypeError, ValueError):
    203       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\ops\math_ops.py in sqrt(x, name)
   4901     A `tf.Tensor` of same size, type and sparsity as `x`.
   4902   """
-> 4903   return gen_math_ops.sqrt(x, name)
   4904 
   4905 

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\ops\gen_math_ops.py in sqrt(x, name)
  10034       return _result
  10035     except _core._NotOkStatusException as e:
> 10036       _ops.raise_from_not_ok_status(e, name)
  10037     except _core._FallbackException:
  10038       pass

~\anaconda3\envs\ds-py37\lib\site-packages\tensorflow\python\framework\ops.py in raise_from_not_ok_status(e, name)
   6860   message = e.message + (" name: " + name if name is not None else "")
   6861   # pylint: disable=protected-access
-> 6862   six.raise_from(core._status_to_exception(e.code, message), None)
   6863   # pylint: enable=protected-access
   6864 

~\anaconda3\envs\ds-py37\lib\site-packages\six.py in raise_from(value, from_value)

InvalidArgumentError: Value for attr 'T' of int32 is not in the list of allowed values: bfloat16, half, float, double, complex64, complex128
	; NodeDef: {{node Sqrt}}; Op<name=Sqrt; signature=x:T -> y:T; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_COMPLEX64, DT_COMPLEX128]> [Op:Sqrt]

In [ ]:

tf.sqrt(tf.cast(tensor, dtype=tf.float16))

tf.sqrt(tf.cast(tensor, dtype=tf.float16))

Out[ ]:

<tf.Tensor: shape=(9,), dtype=float16, numpy=
array([1.   , 1.414, 1.732, 2.   , 2.236, 2.45 , 2.646, 2.828, 3.   ],
      dtype=float16)>

In [ ]:

tf.math.log(tf.cast(tensor, dtype=tf.float16))

tf.math.log(tf.cast(tensor, dtype=tf.float16))

Out[ ]:

<tf.Tensor: shape=(9,), dtype=float16, numpy=
array([0.    , 0.6934, 1.099 , 1.387 , 1.609 , 1.792 , 1.946 , 2.08  ,
       2.197 ], dtype=float16)>

Tensors and numpy¶

Tensorflow interacts beautifully with Numpy arrays (full interoperability)

In [ ]:

tensor.numpy()

tensor.numpy()

Out[ ]:

array([1, 2, 3, 4, 5, 6, 7, 8, 9])

In [ ]:

# The default types of each are slightly different
tensor_np = tf.constant(np.array([3., 7., 10.]))
tensor_tf = tf.constant([3., 7., 10.])

print('tensorflow datatype: ', tensor_tf.dtype)
print('numpy datatype: ', tensor_np.dtype)

# The default types of each are slightly different tensor_np = tf.constant(np.array([3., 7., 10.])) tensor_tf = tf.constant([3., 7., 10.]) print('tensorflow datatype: ', tensor_tf.dtype) print('numpy datatype: ', tensor_np.dtype)

tensorflow datatype:  <dtype: 'float32'>
numpy datatype:  <dtype: 'float64'>