Learning from unlabelled data with COVID-19 Open Research Dataset

栏目: IT技术 · 发布时间: 5年前

内容简介:The COVID-19 Open Research Dataset can help researchers and the health community in the fight against a global pandemic. TheReleased by theAs soon as it was released, there were a

Objective criteria for text search results and some surprising results

The COVID-19 Open Research Dataset can help researchers and the health community in the fight against a global pandemic. The Vespa team is contributing by releasing a search app based on the dataset. Since the data comes with no reliable labels to judge a good search result from a bad one, we would like to propose objective criteria to evaluate search results that do not rely on human-annotated labels. We use this criterion to run experiments and evaluate the value delivered by term-matching and semantic signals. We then show that the semantic signals deliver poor results even when considering a fine-tuned version of a model specifically designed for scientific text.

Learning from unlabelled data with COVID-19 Open Research Dataset

Photo by National Cancer Institute on Unsplash

Released by the Allen Institute for AI , the COVID-19 Open Research Dataset (CORD-19) contains over 44,000 scholarly articles, including over 29,000 with full text, about COVID-19 and the coronavirus family of viruses for use by the global research community. It was released to mobilize researchers to apply recent advances in natural language processing to generate new insights in support of the fight against this infectious disease. And it did exactly that.

As soon as it was released, there were a Kaggle challenge , a dataset explorer , fine-tuned embedding models and a run to collect labelled data:

Given my latest experience with labels containing strong term-matching bias in the MS MARCO dataset and the fact that we at vespa.ai wanted to move fast to build a search app around the CORD-19 dataset, I decided to spend some time to think how I could evaluate between different matching criteria and ranking functions without labelled data.

Objective criteria for text search

The goal was to have an objective criteria and to move away from the “it looks good enough” criteria so commonly used when reliable labels are not available. My proposal is simple, we can use the title of the article as a query and consider the associated abstract as the relevant document for the query.

Learning from unlabelled data with COVID-19 Open Research Dataset

Photo by Marc A on Unsplash

This criteria is simple, can scale to massive amounts of data since we do not rely on human annotation, and it makes sense. Think like this, if you use the title as a query and a given method is not able to retrieve the correct abstract and include it in the top 100 of the resulting list we have a very sub-optimal ranking function for the context of a CORD-19 search app.

Results

Some of the results obtained are summarized in this section. We report here three important metrics. The percentage of documents matched by the query, the recall at the top 100 positions and the mean reciprocal rank (MRR) considering the top 100 documents returned.

Term-matching

Table 1 shows results obtained by ranking documents with the term-matching signal BM25 score . The first row shows the result when we only match documents with abstracts that contains every word in the title (AND operator). This is way too restrictive, matching only a small fraction of documents (0.01%) and therefore misses many relevant abstracts leading to poor recall and MRR metrics (20% and 19% respectively).


以上所述就是小编给大家介绍的《Learning from unlabelled data with COVID-19 Open Research Dataset》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

Java Web整合开发王者归来

Java Web整合开发王者归来

刘京华 / 清华大学 / 2010-1 / 99.80元

《Java Web整合开发王者归来(JSP+Servlet+Struts+Hibernate+Spring)》全面介绍了Java Web开发中的各种相关技术及知识。全书分为9篇,内容层次清晰,难度循序渐进。第1篇为入门篇,内容包括Java Web开发概述等;第2篇为基础篇,内容包括Servlet技术、JSP技术、会话跟踪、过滤器Filter、监听器Listener等;第3篇为高级篇,内容包括JST......一起来看看 《Java Web整合开发王者归来》 这本书的介绍吧!

RGB转16进制工具
RGB转16进制工具

RGB HEX 互转工具

Markdown 在线编辑器
Markdown 在线编辑器

Markdown 在线编辑器

HEX CMYK 转换工具
HEX CMYK 转换工具

HEX CMYK 互转工具