Tensor Types

PyTorch Tensor Types – Why They Matter

When working with PyTorch, one of the first things you bump into (and often ignore at first) is tensor types. But they’re not just boring metadata—they control speed, memory, and whether your code even works.

---

Why Tensor Type Matters

Two big reasons:

1. Performance & Memory
   Different types use different amounts of memory and compute.

   • Example: A 1000 x 1000 matrix with:

     • torch.float32 → ~3.81 MB (4 bytes per value)

     • torch.float64 → ~7.62 MB (8 bytes per value)
       On GPUs, memory is precious—wasting it slows you down or crashes your training.

2. API Requirements
   Some PyTorch functions only work with specific types.

   • Example: Many classification loss functions expect torch.long for labels.
     If you pass a float, you get cryptic errors.

---

Common PyTorch Tensor Types

Data Type	dtype	CPU Tensor	GPU Tensor
32-bit float	torch.float32 / torch.float	torch.FloatTensor	torch.cuda.FloatTensor
64-bit float	torch.float64 / torch.double	torch.DoubleTensor	torch.cuda.DoubleTensor
8-bit int (signed)	torch.int8	torch.CharTensor	torch.cuda.CharTensor
16-bit int	torch.int16 / torch.short	torch.ShortTensor	torch.cuda.ShortTensor
Boolean	torch.bool	torch.BoolTensor	torch.cuda.BoolTensor

(Yes, there are also unsigned ints, 16-bit floats, and more specialized types.)

---

Specifying Tensor Types

If you don’t specify a type, PyTorch defaults to:

• Integers → torch.int64

• Floats → torch.float64

import torch

a = torch.tensor([1, 2, 3])  # default int64
b = torch.tensor([1, 2, 3], dtype=torch.float32)  # explicit float32

You can also create them directly with constructors:

torch.FloatTensor([1, 2, 3])
torch.IntTensor([1, 2, 3])
torch.LongTensor([1, 2, 3])

---

Casting Tensor Types

Use .to() to convert between types:

b = torch.tensor([1.5, 2.5, 3.5], dtype=torch.float32)
c = b.to(torch.int64)  # cast to int64

This does not modify the original tensor—it returns a new one.

---

Moving Tensors to GPU

One of PyTorch’s superpowers is running tensors on GPUs.
You can send them to a specific device:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
a = torch.tensor([1, 2, 3])
a = a.to(device)

• cuda:0, cuda:1, etc. → Different GPUs

• "cpu" → Back to CPU

---

Takeaway:
Choose the right tensor type early—wrong types waste memory, slow you down, or break APIs. Use dtype and .to() like a pro, and you’ll save yourself a lot of debugging pain.

---