Data Manipulation

Tensor Basics

A tensor is an n-dimensional array of numbers — the fundamental data structure for everything that follows in this book.

Like a NumPy ndarray, but GPU-accelerated and differentiable.
1-D tensor → vector, 2-D → matrix, n-D → general tensor.
All four frameworks expose nearly identical tensor APIs.

In this section: how to create, reshape, index, operate on, and share memory with tensors.

Getting Started

A single import wires up the framework’s tensor library:

import torch

A 1-D tensor of n evenly spaced floats — our running example:

x = torch.arange(12, dtype=torch.float32)
x

tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.])

Shape and size

Two attributes you’ll reach for constantly:

.numel() — the total number of elements
.shape — the size along each axis (a tuple)

x.numel()

x.shape

torch.Size([12])

Reshaping

reshape rearranges the same elements into a different shape — the total numel is preserved.

X = x.reshape(3, 4)
X

tensor([[ 0.,  1.,  2.,  3.],
        [ 4.,  5.,  6.,  7.],
        [ 8.,  9., 10., 11.]])

A 12-element vector becomes a 3\times 4 matrix. No data is copied; only the stride metadata changes.

Filled and random tensors

Constant fills take a shape tuple — any rank, any size:

torch.zeros((2, 3, 4))

tensor([[[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]],

        [[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]]])

For weight initialization, randn draws from \mathcal{N}(0, 1) (elements sampled independently):

torch.randn(3, 4)

tensor([[ 0.9909, -0.7055,  1.9036, -0.8750],
        [ 0.7395, -0.8414, -0.6979,  1.9743],
        [ 0.6293, -0.5748,  0.2322, -0.3098]])

ones, full(shape, value), eye(n), empty (uninitialized, fastest), and *_like(x) round out the family.

Tensors from Python lists

For exact control, pass a (nested) list literal — same row-major convention as NumPy:

torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])

tensor([[2, 1, 4, 3],
        [1, 2, 3, 4],
        [4, 3, 2, 1]])

Reading

Standard NumPy-style indexing:

X[-1] — the last row
X[1:3] — rows 1 and 2 (3 is exclusive)

X[-1], X[1:3]

(tensor([ 8.,  9., 10., 11.]),
 tensor([[ 4.,  5.,  6.,  7.],
         [ 8.,  9., 10., 11.]]))

Writing

Assignment works the same way:

X[1, 2] = 17
X

tensor([[ 0.,  1.,  2.,  3.],
        [ 4.,  5., 17.,  7.],
        [ 8.,  9., 10., 11.]])

A slice on the left sets multiple elements at once:

X[:2, :] = 12
X

tensor([[12., 12., 12., 12.],
        [12., 12., 12., 12.],
        [ 8.,  9., 10., 11.]])

Elementwise

Most common math is applied elementwise — same shape in, same shape out.

torch.exp(x)

tensor([162754.7969, 162754.7969, 162754.7969, 162754.7969, 162754.7969,
        162754.7969, 162754.7969, 162754.7969,   2980.9580,   8103.0840,
         22026.4648,  59874.1406])

The arithmetic operators are overloaded — +, -, *, /, ** all run elementwise:

x = torch.tensor([1.0, 2, 4, 8])
y = torch.tensor([2, 2, 2, 2])
x + y, x - y, x * y, x / y, x ** y

(tensor([ 3.,  4.,  6., 10.]),
 tensor([-1.,  0.,  2.,  6.]),
 tensor([ 2.,  4.,  8., 16.]),
 tensor([0.5000, 1.0000, 2.0000, 4.0000]),
 tensor([ 1.,  4., 16., 64.]))

Concatenation

cat glues tensors along an existing axis. Pick the axis with dim:

dim=0 → stack rows (more rows out)
dim=1 → stack columns (wider matrix out)

X = torch.arange(12, dtype=torch.float32).reshape((3,4))
Y = torch.tensor([[2.0, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])
torch.cat((X, Y), dim=0), torch.cat((X, Y), dim=1)

(tensor([[ 0.,  1.,  2.,  3.],
         [ 4.,  5.,  6.,  7.],
         [ 8.,  9., 10., 11.],
         [ 2.,  1.,  4.,  3.],
         [ 1.,  2.,  3.,  4.],
         [ 4.,  3.,  2.,  1.]]),
 tensor([[ 0.,  1.,  2.,  3.,  2.,  1.,  4.,  3.],
         [ 4.,  5.,  6.,  7.,  1.,  2.,  3.,  4.],
         [ 8.,  9., 10., 11.,  4.,  3.,  2.,  1.]]))

Comparisons and reductions

Comparison operators broadcast and return a boolean tensor of the same shape — useful for masking entries that satisfy a condition:

X == Y

tensor([[False,  True, False,  True],
        [False, False, False, False],
        [False, False, False, False]])

sum, mean, max, … collapse one or more axes. Without a dim= argument the whole tensor reduces to a scalar:

X.sum()

tensor(66.)

Broadcasting

When tensors of different shapes meet, the smaller one is virtually expanded along missing dimensions — no data copy.

The rule: dimensions of size 1 stretch; everything else must match.

a = torch.arange(3).reshape((3, 1))
b = torch.arange(2).reshape((1, 2))
a, b

(tensor([[0],
         [1],
         [2]]),
 tensor([[0, 1]]))

a + b

tensor([[0, 1],
        [1, 2],
        [2, 3]])

A 3\times 1 + 1\times 2 becomes a 3\times 2 matrix.

The hidden cost of `Y = Y + X`

Every assignment of an arithmetic expression allocates a new tensor. Matters a lot when Y is gigabytes:

before = id(Y)
Y = Y + X
id(Y) == before

False

id(Y) == before is False: Y now points at a brand-new buffer.

In-place operations

Pre-allocate the output and write into it with Z[:] = ...:

Z = torch.zeros_like(Y)
print('id(Z):', id(Z))
Z[:] = X + Y
print('id(Z):', id(Z))

id(Z): 138997007585184
id(Z): 138997007585184

If the original value of X isn’t needed afterward, the most ergonomic forms are X[:] = X + Y or X += Y:

before = id(X)
X += Y
id(X) == before

True

NumPy round-trip

Tensors and NumPy ndarrays convert cheaply — most frameworks share storage with NumPy when possible:

A = X.numpy()
B = torch.from_numpy(A)
type(A), type(B)

(numpy.ndarray, torch.Tensor)

A size-1 tensor unwraps to a Python scalar with .item(), float(x), or int(x):

a = torch.tensor([3.5])
a, a.item(), float(a), int(a)

(tensor([3.5000]), 3.5, 3.5, 3)

Recap

arange / zeros / ones / randn / tensor(list) — create.
.shape, .numel(), reshape — inspect / reorganize.
[i, j], [a:b, c:d] — read and write slices.
+ - * / **, cat, ==, sum — element-wise ops, joins, comparisons, reductions.
Broadcasting stretches mismatched shapes; in-place ops avoid copying for large tensors.
.numpy() / .item() — leave the tensor world.