AA
A. Abdul Wahab
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
1 records found
1
Dense retrievers are often trained using contrastive learning to differentiate a positive document from a negative one. The harder the negative, the better the training signal and thus the better the distinction. In-batch training takes its negatives from other queries in the same batch, while cross-batch increases the negative document size by using several batches of them. These documents, however, are most of the time easy to tell apart, and larger batches come at the cost of more GPU memory. ANCE finds harder negatives by periodically re-encoding the full corpus with a new checkpoint of the model, which is expensive to do on limited hardware. This thesis investigates whether hard negatives can be found without re-encoding the whole corpus. GRASS is then proposed, which mines documents from a small cache that periodically gets updated and refreshed, and uses the model’s uncertainty in the selection of hard negatives. After fine-tuning the pretrained retriever BGE-M3, GRASS scored on par with ANCE on the 12 BRIGHT domains and higher than cross-batch and in-batch, while each mining round took about a third of the time of an ANCE round.
...
Dense retrievers are often trained using contrastive learning to differentiate a positive document from a negative one. The harder the negative, the better the training signal and thus the better the distinction. In-batch training takes its negatives from other queries in the same batch, while cross-batch increases the negative document size by using several batches of them. These documents, however, are most of the time easy to tell apart, and larger batches come at the cost of more GPU memory. ANCE finds harder negatives by periodically re-encoding the full corpus with a new checkpoint of the model, which is expensive to do on limited hardware. This thesis investigates whether hard negatives can be found without re-encoding the whole corpus. GRASS is then proposed, which mines documents from a small cache that periodically gets updated and refreshed, and uses the model’s uncertainty in the selection of hard negatives. After fine-tuning the pretrained retriever BGE-M3, GRASS scored on par with ANCE on the 12 BRIGHT domains and higher than cross-batch and in-batch, while each mining round took about a third of the time of an ANCE round.