Question:
$\tan ^{-1}\left(\frac{a \cos x-b \sin x}{b \cos x+a \sin x}\right),-\frac{\pi}{2}
Solution:
Let $y=\tan ^{-1}\left(\frac{a \cos x-b \sin x}{b \cos x+a \sin x}\right)$
$y=\tan ^{-1}\left[\frac{\frac{a \cos x}{b \cos x}-\frac{b \sin x}{b \cos x}}{\frac{b \cos x}{b \cos x}+\frac{a \sin x}{b \cos x}}\right]$
$y=\tan ^{-1}\left[\frac{\frac{a}{b}-\tan x}{1+\frac{a}{b} \tan x}\right]$
$y=\tan ^{-1} \frac{a}{b}-\tan ^{-1}(\tan x)$
$\left[\because \tan ^{-1}\left(\frac{x-y}{1+x y}\right)=\tan ^{-1} x-\tan ^{-1} y\right]$
$\Rightarrow y=\tan ^{-1} \frac{a}{b}-x$
Differentiating both sides with respect to $x$
$\frac{d y}{d x}=\frac{d}{d x}\left(\tan ^{-1} \frac{a}{b}\right)-\frac{d}{d x}(x)=0-1=-1$
Thus, $\frac{d y}{d x}=-1$.