Data Manipulation

Tensor Basics

A tensor is an n-dimensional array of numbers — the fundamental data structure for everything that follows in this book.

• Like a NumPy ndarray, but GPU-accelerated and differentiable.
• 1-D tensor → vector, 2-D → matrix, n-D → general tensor.
• All four frameworks expose nearly identical tensor APIs.

In this section: how to create, reshape, index, operate on, and share memory with tensors.

Getting Started

A single import wires up the framework’s tensor library:

import tensorflow as tf

A 1-D tensor of n evenly spaced floats — our running example:

x = tf.range(12, dtype=tf.float32)
x

<tf.Tensor: shape=(12,), dtype=float32, numpy=
array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.],
      dtype=float32)>

Shape and size

Two attributes you’ll reach for constantly:

• .numel() — the total number of elements
• .shape — the size along each axis (a tuple)

tf.size(x)

<tf.Tensor: shape=(), dtype=int32, numpy=12>

x.shape

TensorShape([12])

Reshaping

reshape rearranges the same elements into a different shape — the total numel is preserved.

X = tf.reshape(x, (3, 4))
X

<tf.Tensor: shape=(3, 4), dtype=float32, numpy=
array([[ 0.,  1.,  2.,  3.],
       [ 4.,  5.,  6.,  7.],
       [ 8.,  9., 10., 11.]], dtype=float32)>

A 12-element vector becomes a 3\times 4 matrix. No data is copied; only the stride metadata changes.

Filled and random tensors

Constant fills take a shape tuple — any rank, any size:

tf.zeros((2, 3, 4))

<tf.Tensor: shape=(2, 3, 4), dtype=float32, numpy=
array([[[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]],

       [[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]]], dtype=float32)>

For weight initialization, randn draws from \mathcal{N}(0, 1) (elements sampled independently):

tf.random.normal(shape=[3, 4])

<tf.Tensor: shape=(3, 4), dtype=float32, numpy=
array([[-0.70818615,  1.7719774 ,  1.2700077 , -0.3956194 ],
       [ 2.7438252 , -0.2637143 ,  1.3026156 , -0.7715996 ],
       [ 0.0515474 ,  0.69860435, -0.965453  ,  0.19026478]],
      dtype=float32)>

ones, full(shape, value), eye(n), empty (uninitialized, fastest), and *_like(x) round out the family.

Tensors from Python lists

For exact control, pass a (nested) list literal — same row-major convention as NumPy:

tf.constant([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])

<tf.Tensor: shape=(3, 4), dtype=int32, numpy=
array([[2, 1, 4, 3],
       [1, 2, 3, 4],
       [4, 3, 2, 1]], dtype=int32)>

Reading

Standard NumPy-style indexing:

• X[-1] — the last row
• X[1:3] — rows 1 and 2 (3 is exclusive)

X[-1], X[1:3]

(<tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 8.,  9., 10., 11.], dtype=float32)>,
 <tf.Tensor: shape=(2, 4), dtype=float32, numpy=
 array([[ 4.,  5.,  6.,  7.],
        [ 8.,  9., 10., 11.]], dtype=float32)>)

Writing

Assignment works the same way:

X_var = tf.Variable(X)
X_var[1, 2].assign(9)
X_var

<tf.Variable 'Variable:0' shape=(3, 4) dtype=float32, numpy=
array([[ 0.,  1.,  2.,  3.],
       [ 4.,  5.,  9.,  7.],
       [ 8.,  9., 10., 11.]], dtype=float32)>

A slice on the left sets multiple elements at once:

X_var = tf.Variable(X)
X_var[:2, :].assign(tf.ones(X_var[:2,:].shape, dtype=tf.float32) * 12)
X_var

<tf.Variable 'Variable:0' shape=(3, 4) dtype=float32, numpy=
array([[12., 12., 12., 12.],
       [12., 12., 12., 12.],
       [ 8.,  9., 10., 11.]], dtype=float32)>

Elementwise

Most common math is applied elementwise — same shape in, same shape out.

tf.exp(x)

<tf.Tensor: shape=(12,), dtype=float32, numpy=
array([1.0000000e+00, 2.7182817e+00, 7.3890562e+00, 2.0085537e+01,
       5.4598148e+01, 1.4841316e+02, 4.0342880e+02, 1.0966332e+03,
       2.9809580e+03, 8.1030840e+03, 2.2026467e+04, 5.9874145e+04],
      dtype=float32)>

The arithmetic operators are overloaded — +, -, *, /, ** all run elementwise:

x = tf.constant([1.0, 2, 4, 8])
y = tf.constant([2.0, 2, 2, 2])
x + y, x - y, x * y, x / y, x ** y

(<tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 3.,  4.,  6., 10.], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([-1.,  0.,  2.,  6.], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 2.,  4.,  8., 16.], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([0.5, 1. , 2. , 4. ], dtype=float32)>,
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 1.,  4., 16., 64.], dtype=float32)>)

Concatenation

cat glues tensors along an existing axis. Pick the axis with dim:

• dim=0 → stack rows (more rows out)
• dim=1 → stack columns (wider matrix out)

X = tf.reshape(tf.range(12, dtype=tf.float32), (3, 4))
Y = tf.constant([[2.0, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])
tf.concat([X, Y], axis=0), tf.concat([X, Y], axis=1)

(<tf.Tensor: shape=(6, 4), dtype=float32, numpy=
 array([[ 0.,  1.,  2.,  3.],
        [ 4.,  5.,  6.,  7.],
        [ 8.,  9., 10., 11.],
        [ 2.,  1.,  4.,  3.],
        [ 1.,  2.,  3.,  4.],
        [ 4.,  3.,  2.,  1.]], dtype=float32)>,
 <tf.Tensor: shape=(3, 8), dtype=float32, numpy=
 array([[ 0.,  1.,  2.,  3.,  2.,  1.,  4.,  3.],
        [ 4.,  5.,  6.,  7.,  1.,  2.,  3.,  4.],
        [ 8.,  9., 10., 11.,  4.,  3.,  2.,  1.]], dtype=float32)>)

Comparisons and reductions

Comparison operators broadcast and return a boolean tensor of the same shape — useful for masking entries that satisfy a condition:

X == Y

<tf.Tensor: shape=(3, 4), dtype=bool, numpy=
array([[False,  True, False,  True],
       [False, False, False, False],
       [False, False, False, False]])>

sum, mean, max, … collapse one or more axes. Without a dim= argument the whole tensor reduces to a scalar:

tf.reduce_sum(X)

<tf.Tensor: shape=(), dtype=float32, numpy=66.0>

Broadcasting

When tensors of different shapes meet, the smaller one is virtually expanded along missing dimensions — no data copy.

The rule: dimensions of size 1 stretch; everything else must match.

a = tf.reshape(tf.range(3), (3, 1))
b = tf.reshape(tf.range(2), (1, 2))
a, b

(<tf.Tensor: shape=(3, 1), dtype=int32, numpy=
 array([[0],
        [1],
        [2]], dtype=int32)>,
 <tf.Tensor: shape=(1, 2), dtype=int32, numpy=array([[0, 1]], dtype=int32)>)

a + b

<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[0, 1],
       [1, 2],
       [2, 3]], dtype=int32)>

A 3\times 1 + 1\times 2 becomes a 3\times 2 matrix.

The hidden cost of Y = Y + X

Every assignment of an arithmetic expression allocates a new tensor. Matters a lot when Y is gigabytes:

before = id(Y)
Y = Y + X
id(Y) == before

False

id(Y) == before is False: Y now points at a brand-new buffer.

In-place operations

Pre-allocate the output and write into it with Z[:] = ...:

Z = tf.Variable(tf.zeros_like(Y))
print('id(Z):', id(Z))
Z.assign(X + Y)
print('id(Z):', id(Z))

id(Z): 130085433189312
id(Z): 130085433189312

If the original value of X isn’t needed afterward, the most ergonomic forms are X[:] = X + Y or X += Y:

@tf.function
def computation(X, Y):
    Z = tf.zeros_like(Y)  # This unused value will be pruned out
    A = X + Y  # Allocations will be reused when no longer needed
    B = A + Y
    C = B + Y
    return C + Y

computation(X, Y)

<tf.Tensor: shape=(3, 4), dtype=float32, numpy=
array([[ 8.,  9., 26., 27.],
       [24., 33., 42., 51.],
       [56., 57., 58., 59.]], dtype=float32)>

NumPy round-trip

Tensors and NumPy ndarrays convert cheaply — most frameworks share storage with NumPy when possible:

A = X.numpy()
B = tf.constant(A)
type(A), type(B)

(numpy.ndarray, tensorflow.python.framework.ops.EagerTensor)

A size-1 tensor unwraps to a Python scalar with .item(), float(x), or int(x):

a = tf.constant([3.5]).numpy()
a, a.item(), float(a.item()), int(a.item())

(array([3.5], dtype=float32), 3.5, 3.5, 3)

Recap

• arange / zeros / ones / randn / tensor(list) — create.
• .shape, .numel(), reshape — inspect / reorganize.
• [i, j], [a:b, c:d] — read and write slices.
• + - * / **, cat, ==, sum — element-wise ops, joins, comparisons, reductions.
• Broadcasting stretches mismatched shapes; in-place ops avoid copying for large tensors.
• .numpy() / .item() — leave the tensor world.