TensorFlow for Machine Intelligence (TFFMI) ● Hands-On Machine Learning with Scikit-Learn and TensorFlow. Chapter 9: Up and running with TensorFlow ● Fundamentals of Deep Learning. Chapter 3: Implementing Neural Networks in TensorFlow (FODL) TensorFlow is being constantly updated so books might become outdated fast

An n-dimensional array 

0-d tensor: scalar (number) 

1-d tensor: vector 

2-d tensor: matrix 

and so on 

Create a session, assign it to variable sess so we can call it later 

Within the session, evaluate the graph to fetch the value of a 

import tensorflow as tf 

a = tf.add(3, 5) 

sess = tf.Session() 

print sess.run(a) 

sess.close()

with tf.device('/gpu:2'): 

    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a') 

    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b') 

    c = tf.matmul(a, b) 

sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

print sess.run(c)

$ vim a.py

import tensorflow as tf 

a = tf.constant(2) # name='a' 

b = tf.constant(3) # name='b'

x = tf.add(a, b) # name='add'

with tf.Session() as sess: 

  writer = tf.summary.FileWriter('./graphs', sess.graph) 

  print sess.run(x) # close the writer when you’re done using it writer.close()

$ python a.py
$ tensorboard --logdir="./graphs" 

tf​.​constant​(​value​,​ dtype​=​None​,​ shape​=​None​,​ name​=​'Const'​,​ verify_shape​=​False)      # b = tf.constant([[0, 1], [2, 3]], name="b")

tf.zeros([2, 3], tf.int32)     # [[0, 0, 0], [0, 0, 0]]

tf.zeros_like(input_tensor)   # [[0, 0], [0, 0], [0, 0]]

tf​.​ones​(​shape​,​ dtype​=​tf​.​float32​,​ name​=​None)

tf.ones_like(input_tensor)     # [[1, 1], [1, 1], [1, 1]]

tf​.​fill​(​dims​,​ value​,​ name​=​None​)

tf.linspace(10.0, 13.0, 4, name="linspace")    # [10.0 11.0 12.0 13.0]         a sequence of num evenly-spaced values 

tf.range(start, limit, delta)     # 'start' is 3, 'limit' is 1, 'delta' is -0.5      [3, 2.5, 2, 1.5]

unlike NumPy or Python sequences, TensorFlow sequences are not iterable. 

for _ in np.linspace(0, 10, 4): # OK 

for _ in tf.linspace(0, 10, 4): # TypeError("'Tensor' object is not iterable.") 

for _ in range(4): # OK 

for _ in tf.range(4): # TypeError("'Tensor' object is not iterable.")

Generate random constants from certain distributions.

tf.random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
tf.random_uniform(shape, minval=0, maxval=None, dtype=tf.float32, seed=None, name=None)
tf.random_shuffle(value, seed=None, name=None) tf.random_crop(value, size, seed=None, name=None)
tf.multinomial(logits, num_samples, seed=None, name=None)
tf.random_gamma(shape, alpha, beta=None, dtype=tf.float32, seed=None, name=None)

a = tf.constant([3, 6]) 

b = tf.constant([2, 2]) 

tf.add(a, b) # >> [5 8] 

tf.add_n([a, b, b]) # >> [7 10]. Equivalent to a + b + b 

tf.mul(a, b) # >> [6 12] because mul is element wise 

tf.matmul(a, b) # >> ValueError 

tf.matmul(tf.reshape(a, shape=[1, 2]), tf.reshape(b, shape=[2, 1])) # >> [[18]] 

tf.div(a, b) # >> [1 3] 

tf.mod(a, b) # >> [1 0]

x.initializer # init 

x.value() # read op 

x.assign(...) # write op 

x.assign_add(...) 

etc

You have to initialize variables before using them

Initializ all variables at once

init = tf.global_variables_initializer()

with tf.Session() as sess:

    tf.run(init)    # Note that you use tf.run() to run the initializer, not fetching any value.

Initialize only a subset of variables with a list of variables to initialize

init_ab = tf.variables_initializer([a, b], name="init_ab")

with tf.Session() as sess:

    tf.run(init_ab)

Initialize each variable separately using tf.Variable.initializer

# create variable W as 784 x 10 tensor, filled with zeros
W = tf.Variable(tf.zeros([784,10]))
with tf.Session() as sess:
    tf.run(W.initializer)

Print a variable

W = tf.Variable(tf.truncated_normal([700, 10]))
with tf.Session() as sess:
    sess.run(W.initializer)
    print W     # Tensor("Variable/read:0", shape=(700, 10), dtype=float32)
    print W.eval()    #  actual variable value

W = tf.Variable(10) 

W.assign(100) 

with tf.Session() as sess: 

    sess.run(W.initializer) 

    print W.eval()    #  10 Why 10 and not 100? W.assign(100) doesn’t assign the value 100 to W, but instead create an assign op to do that. For this op to take effect, we have to run this op in session. 

W = tf.Variable(10) 

assign_op = W.assign(100) 

with tf.Session() as sess:

    sess.run(assign_op) 

    print W.eval()    #  100 Note that we don’t have initialize W in this case, because assign() does it for us. In fact, initializer op is the assign op that assigns the variable’s initial value to the variable itself.

Interesting example: 

a = tf.Variable(2, name="scalar")    # create a variable whose original value is 2 

a_times_two = a.assign(a * 2)        # assign a * 2 to a and call that op a_times_two 

init = tf.global_variables_initializer() 

with tf.Session() as sess: 

    sess.run(init) # have to initialize a, because a_times_two op depends on the value of a 

    sess.run(a_times_two) # >> 4 

    sess.run(a_times_two) # >> 8 

    sess.run(a_times_two) # >> 16 

TensorFlow assigns a*2 to a every time a_times_two is fetched. [[At each time variable is calculated from scratch]]

Increment, Decrement --->    tf.Variable.assign_add() and tf.Variable.assign_sub() 

Because TensorFlow sessions maintain values separately, each Session can have its own current value for a variable defined in a graph.

W = tf.Variable(10)
sess1 = tf.Session()
sess2 = tf.Session()
sess1.run(W.initializer)
sess2.run(W.initializer)
print sess1.run(W.assign_add(10)) # >> 20
print sess2.run(W.assign_sub(2)) # >> 8
print sess1.run(W.assign_add(100)) # >> 120
print sess2.run(W.assign_sub(50)) # >> -42
sess1.close()
sess2.close()

 declare a variable that depends on other variables

# W is a random 700 x 100 tensor 

W = tf.Variable(tf.truncated_normal([700, 10])) 

U = tf.Variable(W * 2)

U = tf.Variable(W.intialized_value() * 2)    # use initialized_value() to make sure that W is initialized before its value is used to initialize W.

InteractiveSession

 makes itself the default session so you can call run() or eval() without explicitly call the session. This is convenient in interactive shells and IPython notebooks, as it avoids having to pass an explicit Session object to run ops

sess = tf.InteractiveSession() 

a = tf.constant(5.0) 

b = tf.constant(6.0) 

c = a * b 

print(c.eval())   # We can just use 'c.eval()' without passing 'sess' 

sess.close()

# your graph g have 5 ops: a, b, c, d, e 

with g.control_dependencies([a, b, c]): # `d` and `e` will only run after `a`, `b`, and `c` have executed. 

   d = ... 

   e = …

tf.Variable for trainable variables such as weights (W) and biases (B) for your model. tf.placeholder is used to feed actual training examples.

The difference is that with tf.Variable you have to provide an initial value when you declare it. With tf.placeholder you don't have to provide an initial value and you can specify it at run time with the feed_dict argument inside Session.run

We can and only need to save or restore the Variables to save or rebuild the graph. Placeholders are mostly holders for the different datasets

# create a placeholder of type float 32-bit, shape is a vector of 3 elements 

a = tf.placeholder(tf.float32, shape=[3]) # create a constant of type float 32-bit, shape is a vector of 3 elements 

b = tf.constant([5, 5, 5], tf.float32) 

# use the placeholder as you would a constant or a variable 

c = a + b # Short for tf.add(a, b) 

# If we try to fetch c, we will run into error. 

with tf.Session() as sess: 

    print(sess.run(c)) >> NameError

with tf.Session() as sess: 

    # feed [1, 2, 3] to placeholder a via the dict {a: [1, 2, 3]} 

    # fetch value of c 

    print(sess.run(c, {a: [1, 2, 3]}))

with tf.Session() as sess: 

    for a_value in list_of_a_values: 

        print(sess.run(c, {a: a_value}))

print sess.graph.as_graph_def()

tf.get_default_graph().as_graph_def()

x = tf.placeholder("float", [None, 784])
(Here None means that a dimension can be of any length.)  placeholder, a value that we'll input

A Variable is a modifiable tensor that lives in TensorFlow's graph of
interacting operations. model parameters be Variables.