An inverse mix is usually used to compare two things which are thought to be similar. You subtract one from the other so only the difference remains which can now be examined.
Suppose you have A and B then D = A - B would obviously be the difference. When you apply this to audio signals, the mixing is usually an additive process ( A + B ), so we need to invert the B-Signal: A + (-B); hence the name. When A and B are identical, the inverse mixing process will result in silence.
Similar techniques are used in image processing, to remove dust and scratches from scanned pictures or to cover so called hot pixels in digital photography ("dark frame subtraction").
If you rip a Audio CD and receive an error on one track, you may want to do two subsequent rips and compare them with this technique to isolate the position of possible clicks and other flaws.
A Word of Warning
Many uninformed people use inverse mixing to rank the quality of psychoacoustic lossy audio encoders. They argue that the quieter the resulting residual (difference), the better the codec. However, this is fundamentally flawed. It is the goal of a psychoacoustic lossy encoder to remove all inaudible information from an audio signal. If you were to inverse mix the original and the encoded signal, you would see a large difference, but an ideal encode would still sound the same as the original. On the other hand, the difference could be very small and yet clearly audible.
By looking at the difference or listening to it, you can not tell if what has been removed would result in an audible degradation of the signal. Inverse mixing doesn't say anything about codec quality.