if

Given: $f(x)=x^{2}-2 x$

$f(A)=A^{2}-2 A$

Now,

$A^{2}=A A$

$\Rightarrow A^{2}=\left[\begin{array}{lll}0 & 1 & 2 \\ 4 & 5 & 0 \\ 0 & 2 & 3\end{array}\right]\left[\begin{array}{lll}0 & 1 & 2 \\ 4 & 5 & 0 \\ 0 & 2 & 3\end{array}\right]$

$\Rightarrow A^{2}=\left[\begin{array}{ccc}0+4+0 & 0+5+4 & 0+0+6 \\ 0+20+0 & 4+25+0 & 8+0+0 \\ 0+8+0 & 0+10+6 & 0+0+9\end{array}\right]$

$\Rightarrow A^{2}=\left[\begin{array}{ccc}4 & 9 & 6 \\ 20 & 29 & 8 \\ 8 & 16 & 9\end{array}\right]$

$f(A)=A^{2}-2 A$

$\Rightarrow f(A)=\left[\begin{array}{ccc}4 & 9 & 6 \\ 20 & 29 & 8 \\ 8 & 16 & 9\end{array}\right]-2\left[\begin{array}{lll}0 & 1 & 2 \\ 4 & 5 & 0 \\ 0 & 2 & 3\end{array}\right]$

$\Rightarrow f(A)=\left[\begin{array}{ccc}4 & 9 & 6 \\ 20 & 29 & 8 \\ 8 & 16 & 9\end{array}\right]-\left[\begin{array}{ccc}0 & 2 & 4 \\ 8 & 10 & 0 \\ 0 & 4 & 6\end{array}\right]$

$\Rightarrow f(A)=\left[\begin{array}{cc}4-0 & 9-2 & 6-4 \\ 20-8 & 29-10 & 8-0 \\ 8-0 & 16-4 & 9-6\end{array}\right]$

$\Rightarrow f(A)=\left[\begin{array}{ccc}4 & 7 & 2 \\ 12 & 19 & 8 \\ 8 & 12 & 3\end{array}\right]$