Multivariate Performance Measures
Bipartite ranking is a two-label ranking problem involving relevant and non-relevant elements.
Two Groups Instead of Every Pair
Some ranking problems do not need to distinguish every element from every other element. The important question may be simpler: does each element belong to the relevant group or the non-relevant group? Bipartite ranking describes this two-label setting.
Bipartite ranking is a two-label ranking problem involving relevant and non-relevant elements.
Reading the Feedback Vector
The feedback vector y records the two groups with numerical labels. An entry of 1 identifies a relevant element, while an entry of -1 identifies a non-relevant element. Each position in y therefore carries the feedback label for the corresponding element.
Interpreting a feedback vector
Consider the feedback vector y = [1, -1, 1]. What does each entry communicate?
- 1First position
The value 1 identifies the first element as relevant.
- 2Second position
The value -1 identifies the second element as non-relevant.
- 3Third position
The value 1 identifies the third element as relevant.
The vector describes two relevant elements and one non-relevant element.
From Scores to Binary Labels
A predicted ranking vector y' contains predicted values for the elements. Each predicted value y'i is compared with a threshold θ. The binary prediction is obtained through sign(y'i - θ). Values above the threshold produce the positive sign, corresponding to the relevant label 1; values below the threshold produce the negative sign, corresponding to the non-relevant label -1.
Converting predicted values
Use θ = 0 with the predicted ranking vector y' = [0.8, -0.4, 0.2]. Convert each value into a binary label using sign(y'i - θ).
- 1First value
The value 0.8 is above θ = 0, so its sign is positive and its binary label is 1.
- 2Second value
The value -0.4 is below θ = 0, so its sign is negative and its binary label is -1.
- 3Third value
The value 0.2 is above θ = 0, so its sign is positive and its binary label is 1.
The converted binary vector is [1, -1, 1].
Selecting the Threshold
The threshold θ is usually set to 0. With that choice, the conversion asks whether each predicted value is on the positive or negative side of zero. However, zero is not mandatory in every problem. Additional problem constraints may motivate choosing another threshold.
Mistakes in Label Conversion
Common mistake Treating bipartite ranking as a problem that must order every element distinctly.
What it looks likeAssuming the task must distinguish every element from every other element.
Why it failsBipartite ranking focuses on separating elements into the two groups relevant and non-relevant.
Do this insteadFirst identify the two-label structure: relevant elements receive 1 and non-relevant elements receive -1.
Common mistake Reversing the meanings of 1 and -1 in the feedback vector.
What it looks likeReading -1 as relevant and 1 as non-relevant.
Why it failsThe feedback-vector convention assigns 1 to relevant elements and -1 to non-relevant elements.
Do this insteadUse 1 for relevant and -1 for non-relevant.
Common mistake Applying the threshold without subtracting θ.
What it looks likeUsing sign(y'i) when the problem specifies a threshold other than 0.
Why it failsThe stated conversion is sign(y'i - θ), so the predicted value must be compared with the selected threshold.
Do this insteadSubtract θ from each predicted value before interpreting its sign.
Common mistake Assuming the threshold must always be zero.
What it looks likeRejecting another threshold even when additional problem constraints motivate it.
Why it failsZero is usual, but the threshold may be chosen using additional constraints.
Do this insteadStart with θ = 0 when appropriate, then check whether the problem requires a different value.
Check Your Understanding
A predicted ranking vector contains the values 0.6 and -0.2. Using θ = 0, determine the binary label produced for each value. Then explain what would need to be reconsidered if an additional problem constraint required a different threshold.
Hints
- Compare each predicted value with θ.
- A value above the threshold receives the positive label 1, while a value below the threshold receives the negative label -1.
- A different threshold changes the comparison used in sign(y'i - θ).
- 1Bipartite ranking handles a two-group ranking problem: relevant versus non-relevant elements. The feedback vector y uses 1 for relevant elements and -1 for non-relevant elements. A predicted ranking value y'i becomes a binary label through sign(y'i - θ). The threshold θ is usually 0, but additional problem constraints can motivate another choice.
Key Takeaways
- Bipartite ranking separates elements into relevant and non-relevant groups.
- The feedback vector uses 1 for relevant elements and -1 for non-relevant elements.
- Each predicted ranking value is converted with sign(y'i - θ).
- The threshold is usually 0, but additional problem constraints may require another value.