Solve this

Here,

$f(x)=x^{2}-2 x-3$

$\Rightarrow f(A)=A^{2}-2 A-3 I_{2}$

Now,

$A^{2}=A A$

$\Rightarrow A^{2}=\left[\begin{array}{ll}1 & 2 \\ 2 & 1\end{array}\right]\left[\begin{array}{ll}1 & 2 \\ 2 & 1\end{array}\right]$

$\Rightarrow A^{2}=\left[\begin{array}{ll}1+4 & 2+2 \\ 2+2 & 4+1\end{array}\right]$

$\Rightarrow A^{2}=\left[\begin{array}{ll}5 & 4 \\ 4 & 5\end{array}\right]$

$f(A)=A^{2}-2 A-3 I_{2}$

$\Rightarrow f(A)=\left[\begin{array}{ll}5 & 4 \\ 4 & 5\end{array}\right]-2\left[\begin{array}{ll}1 & 2 \\ 2 & 1\end{array}\right]-3\left[\begin{array}{ll}1 & 0 \\ 0 & 1\end{array}\right]$

$\Rightarrow f(A)=\left[\begin{array}{ll}5 & 4 \\ 4 & 5\end{array}\right]-\left[\begin{array}{ll}2 & 4 \\ 4 & 2\end{array}\right]-\left[\begin{array}{ll}3 & 0 \\ 0 & 3\end{array}\right]$

$\Rightarrow f(A)=\left[\begin{array}{cc}5-2-3 & 4-4-0 \\ 4-4-0 & 5-2-3\end{array}\right]$

$\Rightarrow f(A)=\left[\begin{array}{cc}5-5 & 0 \\ 0 & 5-5\end{array}\right]$

$\Rightarrow f(A)=0$