If A, B and C are angles of a triangle,

Solution:

Given: $A, B$ and $C$ are angles of a triangle

Therefore, $A+B+C=\pi \ldots$ (1)

$\left|\begin{array}{ccc}-1 & \cos C & \cos B \\ \cos C & -1 & \cos A \\ \cos B & \cos A & -1\end{array}\right|$

Expanding through $R_{1}$

$=\left\{-1\left(1-\cos ^{2} A\right)-\cos C(-\cos C-\cos A \cos B)+\cos B(\cos A \cos C+\cos B)\right\}$

$=-1+\cos ^{2} A+\cos ^{2} C+\cos A \cos B \cos C+\cos A \cos B \cos C+\cos ^{2} B$

$=-1+2 \cos A \cos B \cos C+\cos ^{2} A+\cos ^{2} B+\cos ^{2} C$

$=-1+2 \cos A \cos B \cos C+\frac{1+\cos 2 A}{2}+\frac{1+\cos 2 B}{2}+\cos ^{2} C \quad\left(\because 2 \cos ^{2} \theta=1+\cos 2 \theta\right)$

$=-1+2 \cos A \cos B \cos C+\frac{1+\cos 2 A+1+\cos 2 B}{2}+\cos ^{2} C$

$=-1+2 \cos A \cos B \cos C+\frac{2+2 \cos \left(\frac{2 \frac{2 A+2 B}{2}}{2}\right) \cos \left(\frac{2 A-2 B}{2}\right)}{2}+\cos ^{2} C \quad\left(\because \cos A+\cos B=2 \cos \left(\frac{A+B}{2}\right) \cos \left(\frac{A-B}{2}\right)\right)$

$=-1+2 \cos A \cos B \cos C+\frac{2+2 \cos (A+B) \cos (A-B)}{2}+\cos ^{2} C$

$=-1+2 \cos A \cos B \cos C+\frac{2+2 \cos (\pi-\mathrm{C}) \cos (A-B)}{2}+\cos ^{2} C$         $(\because A+B+C=\pi)$

$=-1+2 \cos A \cos B \cos C+\frac{2-2 \cos C \cos (A-B)+2 \cos ^{2} C}{2}$

$=-1+2 \cos A \cos B \cos C+1-\cos C \cos (A-B)+\cos ^{2} C$

$=-1+2 \cos A \cos B \cos C+1-\cos C(\cos (A-B)-\cos C)$

$=-1+2 \cos A \cos B \cos C+1-\cos C\left(-2 \sin \left(\frac{A-B+C}{2}\right) \sin \left(\frac{A-B-C}{2}\right)\right) \quad\left(\because \cos A-\cos B=-2 \sin \left(\frac{A+B}{2}\right) \sin \left(\frac{A-B}{2}\right)\right)$

$=-1+2 \cos A \cos B \cos C+1-\cos C\left(-2 \sin \left(\frac{A+C-B}{2}\right) \sin \left(\frac{A-(B+C)}{2}\right)\right)$

$=-1+2 \cos A \cos B \cos C+1-\cos C\left(-2 \sin \left(\frac{\pi-B-B}{2}\right) \sin \left(\frac{A-(\pi-A)}{2}\right)\right) \quad(\because A+B+C=\pi)$

$=-1+2 \cos A \cos B \cos C+1+2 \cos C \sin \left(\frac{\pi}{2}-B\right) \sin \left(A-\frac{\pi}{2}\right)$

$=-1+2 \cos A \cos B \cos C+1+2 \cos C \cos B \sin \left(A-\frac{\pi}{2}\right)$

$=-1+2 \cos A \cos B \cos C+1-2 \cos C \cos B \sin \left(\frac{\pi}{2}-A\right)$

$=-1+2 \cos A \cos B \cos C+1-2 \cos C \cos B \cos A$

$=0$

Hence, the correct option is (a).