Unit step function

The unit step function (also the Heaviside function) $u(t)$ is a piecewise function that is off ($u=0$) until a certain time $t=a$ and on ($u=1$) after that time.

$$u(t)=\{\begin{array}{ll}0& 0\le t<a\\ 1& t\ge a\end{array}$$

The motivation is that in electrical engineering we frequently encounter phenomena that are either off or on, particularly until a certain time. Multiplying a signal by the step creates a signal that is right-sided. We can also construct square waves with the unit step:

$$x(t)=\sum _{k=-\infty }^{\infty }(-1{)}^{k}u(t-k)-u(t-1-k)$$

Important applications of the step function include the second translation theorem and the step response of a system.

Curiously:

$$\delta (t)=\frac{d}{dt}u(t)$$ $$u(t)={\int }_{-\infty }^t\delta (\tau )d\tau$$

where $\delta (t)$ is the impulse function.

Deep learning

The binary unit step function, as well as the sign function are primitive types of activation functions. They have the ability to divide the input space into two sides of a hyperplane. However! Both aren’t differentiable, smooth, or continuous.

These flaws motivate the use of sigmoid activation functions.