Training and inference are different jobs

By How things work · myAudience

myAudience original rendered teaching illustration. Image: illustration, not a documentary photograph. · Operator-authored generated illustration; scientific concept model reviewed separately.

Training and inference are different jobs

A model uses parameters to turn inputs into outputs. Training adjusts parameters using an objective and examples. Ordinary inference applies a trained model to a new input without performing that training update. Neither operation implies human understanding or guaranteed truth.

A simple training cycle computes a prediction, measures a loss, obtains gradients and updates parameters. In a basic gradient-descent rule, the update subtracts learning rate multiplied by gradient. Real training may use different optimisers, batching and numerical representations; this example introduces the relation rather than promising a universally suitable recipe.

Evaluation mode and disabling gradient recording are separate software decisions. Evaluation mode changes behaviour of certain layers, while disabling gradients avoids unnecessary derivative bookkeeping. Official PyTorch examples demonstrate both; changing one does not automatically perform the other.

An inference service must also receive requests, queue work and return results. Batching can combine compatible requests, but waiting to form a batch adds delay. Triton’s documented delayed batching illustrates a throughput-versus-latency choice; its settings are not universal recommendations.

Training and inference systems are established. Online learning and adaptation exist, so “inference never changes anything” would be too broad. State whether parameter updates are actually enabled and measure accuracy, latency, cost and failure behaviour for the intended task.

Discuss: should a service retain people’s inputs to improve a model by default? Compare potential benefit with consent, privacy and alternatives such as voluntary contribution. Describe what users would need to understand before making a meaningful choice.

Image: illustration, not a documentary photograph.

Worked example: For an invented one-parameter update, weight = 2.0, gradient = 0.5 and learning rate = 0.1. Calculate the next weight.

Update = learning rate × gradient = 0.1 × 0.5 = 0.05.

New weight = 2.0 − 0.05 = 1.95.

This single update does not establish convergence or accuracy.

Answer: New weight: 1.95.

Check your understanding: A service waits 8 ms to form a batch, then computation takes 12 ms. What is the simplified elapsed time excluding transport, and is this a training update?

Answer: 20 ms; ordinary batched inference need not update learned parameters.

Why: Queueing plus computation gives 8 + 12 = 20 ms. Batching and training are different mechanisms.