If 2 cos θ − sin θ = x and cos θ − 3 sin θ = y.

Given that:

$2 \cos \theta-\sin \theta=x$

$\cos \theta-3 \sin \theta=y$

Then we have to prove that

First we take the LHS and put the value of x and y, we get

$2 x^{2}+y^{2}-2 x y=2(2 \cos \theta-\sin \theta)^{2}+(\cos \theta-3 \sin \theta)^{2}-2(2 \cos \theta-\sin \theta)(\cos \theta-3 \sin \theta)$

$=2\left(4 \cos ^{2} \theta+\sin ^{2} \theta-4 \sin \theta \cos \theta\right)+\left(\cos ^{2} \theta+9 \sin ^{2} \theta-6 \sin \theta \cos \theta\right)$

$-2\left(2 \cos ^{2} \theta-6 \sin \theta \cos \theta-\sin \theta \cos \theta+3 \sin ^{2} \theta\right)$

$=8 \cos ^{2} \theta+2 \sin ^{2} \theta-8 \sin \theta \cos \theta+\cos ^{2} \theta+9 \sin ^{2} \theta-6 \sin \theta \cos \theta-4 \cos ^{2} \theta$

$+12 \sin \theta \cos \theta+2 \sin \theta \cos \theta-6 \sin ^{2} \theta$

$=5 \sin ^{2} \theta+\cos ^{2} \theta$

$=5\left(\sin ^{2} \theta+\cos ^{2} \theta\right)$

$=5(1)$

$=5$

Hence $2 x^{2}+y^{2}-2 x y=5$