Tensor Operations

Tensors are the core data structure in TinyRL, providing multi-dimensional arrays with automatic differentiation support.

Quick Reference

Operation	Syntax	Description
Create	`ag::Tensor(matrix, true)`	Create tensor with gradients
Add	`a + b`	Element-wise addition
Multiply	`a * b`	Element-wise multiplication
Matmul	`a.matmul(b)`	Matrix multiplication
Transpose	`ag::transpose(a)`	Swap dimensions
Sum	`ag::sum(a)`	Reduce to scalar
Mean	`ag::mean(a)`	Average all elements
Backward	`loss.backward()`	Compute gradients

Creating Tensors

Tensors wrap an ag::Matrix and track whether gradients should be computed.

#include "autograd.h"

// From random values (common for initialization)
ag::Tensor x(ag::Matrix::Random(2, 3), true, "x");  // Shape: {2, 3, 1, 1}

// From explicit values
ag::Tensor y(ag::Matrix({{1, 2}, {3, 4}}), true);

// Factory methods for special matrices
ag::Tensor zeros(ag::Matrix::Zeros(3, 3), false);   // No gradients needed
ag::Tensor ones(ag::Matrix::Ones(2, 2), false);

// 4D tensors for CNNs: (batch, channels, height, width)
ag::Tensor img(ag::Matrix::Random(1, 3, 28, 28), true);

Basic Operations

All operations support automatic differentiation when requires_grad=true.

Arithmetic Operations

ag::Tensor a(ag::Matrix::Random(2, 2), true);
ag::Tensor b(ag::Matrix::Random(2, 2), true);

auto sum = a + b;      // Element-wise addition
auto diff = a - b;     // Element-wise subtraction  
auto prod = a * b;     // Element-wise multiplication
auto quot = a / b;     // Element-wise division

// Scalar operations (broadcasts automatically)
auto scaled = a * 2.0;
auto shifted = a + 1.0;

Matrix Operations

ag::Tensor x(ag::Matrix::Random(2, 3), true);   // Shape: (2, 3)
ag::Tensor w(ag::Matrix::Random(3, 4), true);   // Shape: (3, 4)

auto z = x.matmul(w);      // Matrix multiply: (2,3) @ (3,4) → (2,4)
auto t = ag::transpose(x); // Transpose: (2,3) → (3,2)
auto t2 = x.transpose();   // Method form also works

Math Functions

All math functions preserve gradients through the computation:

ag::Tensor x(ag::Matrix::Random(2, 2), true);

auto y1 = ag::exp(x);        // e^x element-wise
auto y2 = ag::log(x);        // ln(x) element-wise
auto y3 = ag::pow(x, 2.0);   // x² element-wise
auto y4 = ag::sqrt(x);       // √x element-wise

Reductions

Reduction operations collapse dimensions to scalars:

ag::Tensor x(ag::Matrix::Random(3, 4), true);  // 12 elements

ag::Tensor s = ag::sum(x);    // Sum all → scalar (1x1)
ag::Tensor m = ag::mean(x);   // Mean → scalar (1x1)

// Use .item() to get the scalar value
float loss_value = ag::sum(x).item();

Broadcasting

TinyRL broadcasts tensors with compatible shapes, following NumPy-style rules:

Rule	Description
1	Dimensions are compared right-to-left
2	Dimensions match if equal or one is 1
3	Missing dimensions are treated as 1

2D Example:

ag::Tensor a(ag::Matrix::Random(2, 1), true);  // Shape: (2, 1)
ag::Tensor b(ag::Matrix::Random(1, 3), true);  // Shape: (1, 3)
auto c = a + b;                                 // Result: (2, 3)
// Each element: c[i,j] = a[i,0] + b[0,j]

4D Example (CNN bias addition):

// Add per-channel bias to feature maps
ag::Tensor features(ag::Matrix::Random(1, 64, 28, 28), true);  // (N,C,H,W)
ag::Tensor bias(ag::Matrix::Random(1, 64, 1, 1), true);        // (1,C,1,1)
auto result = features + bias;  // Broadcasts across H,W dimensions

Gradient Operations

Computing Gradients

Call backward() on a scalar loss to compute gradients for all tensors with requires_grad=true:

ag::Tensor x(ag::Matrix::Random(2, 2), true);  // requires_grad=true

// Forward pass: y = x², z = sum(y)
ag::Tensor y = ag::pow(x, 2.0);
ag::Tensor z = ag::sum(y);

// Backward pass: computes ∂z/∂x = 2x
z.backward();

// Access gradient
std::cout << "Gradient:\n" << x.grad() << std::endl;

Gradient Management

Method	Purpose
`x.grad()`	Get gradient matrix
`x.zero_grad()`	Reset gradients to zero
`x.clear_graph()`	Free computational graph memory
`x.detach()`	Create copy detached from graph

Shape Operations

ag::Tensor x(ag::Matrix::Random(4, 6), true);

// Query shape
auto shape = x.shape();  // Returns {4, 6, 1, 1}
int rows = shape[0];     // 4
int cols = shape[1];     // 6

// Reshape (creates a reshaped copy)
auto y = x.reshape({2, 12, 1, 1});  // (4,6) → (2,12)

// Flatten (flatten all elements into a single dimension)
ag::Tensor conv_out(ag::Matrix::Random(1, 32, 7, 7), true);  // (N,C,H,W)
auto flat = conv_out.flatten();  // → (1568, 1, 1, 1)
// For CNN → MLP with batch preserved, use nn::Flatten

Memory Management

Tensors use reference counting for automatic memory management:

// Automatic cleanup via RAII
{
    ag::Tensor x(ag::Matrix::Random(100, 100), true);
    {
        ag::Tensor y = x;  // Shares data (reference count +1)
    }  // y out of scope (reference count -1)
}  // x out of scope, memory freed

// Manual cleanup for long-running training
for (int epoch = 0; epoch < 1000; ++epoch) {
    auto loss = compute_loss(model, data);
    loss.backward();
    optimizer.step();
    loss.clear_graph();  // ← Important: free graph each iteration
}

Tip: Always call clear_graph() after backward() in training loops to prevent memory accumulation.

Error Handling

TinyRL validates shapes and throws descriptive errors:

try {
    ag::Tensor a(ag::Matrix::Random(2, 3), true);
    ag::Tensor b(ag::Matrix::Random(4, 5), true);
    auto c = a + b;  // Throws: shapes (2,3) and (4,5) not broadcastable
} catch (const std::runtime_error& e) {
    std::cerr << "Error: " << e.what() << std::endl;
}

try {
    ag::Tensor a(ag::Matrix::Random(2, 3), true);
    ag::Tensor b(ag::Matrix::Random(5, 4), true);
    auto c = a.matmul(b);  // Throws: inner dimensions 3 != 5
} catch (const std::runtime_error& e) {
    std::cerr << "Matmul error: " << e.what() << std::endl;
}