The Fourth Medical NLP Shared Task
The One and Only Medical Language Processing Contest


Welcome to MedWeb (Medical Natural Language Processing for Web Document)

Recently, an increasing number of medical records is being stored in the form of electronic media instead of paper media -- making digital information processing in fields more and more necessary. Nowadays, this trend in information processing focuses not only on electronic health records but also on various data coming from patients. This data we call patient texts include social media texts, web blogs, and so on.

NTCIR-13 MedWeb (Medical Natural Language Processing for Web Document) task provides Twitter-like message texts (in Japanese, English, and Chinese), and then requires to classify them. In detail, MedWeb consists of Twitter task (Japanese subtask, English subtask, and Chinese subtask). Since these subtask settings can be formalized as multi-label classification of disease/symptom-related texts, the achievements of this task can almost be directly applied to a fundamental engine for actual applications.

MedWeb Task

Twitter task - ja, en, zh

This task requires participants to perform a multi-label classification that labels for 8 diseases/symptoms must be assigned to each tweet. According to the registered subtasks (Japanese subtask:ja, English subtask:en, Chinese subtask:zh), training data and test data will be distributed to task participants. Given tweets, the output are Positive:p or Negative:n labels for 8 diseases/symptoms. In this task, the target diseases/symptoms are not limited to influenza only since this also deals with other 7 diseases/symptoms including diarrhea/stomachache, hay fever, cough/sore throat, headache, fever, runny nose, and cold. These targets are designed based on the advice of a Japanese government research center (National Institute of Infectious Diseases (NIID)).

Annotation Guideline and Dataset

Annotation Guideline


Training corpus distribution is started via e-mail from NTCIR office. Test data will be distributed on July 24, 2017 (See Important Dates).
Participants will obtain the following data:

These tweets are related to 8 diseases/symptoms include influenza, diarrhea/stomachache, hay fever, cough/sore throat, headache, fever, runny nose, and cold. Note that the tweet data crawled using Twitter API is not allowed to release due to the Twitter’s developer policy concerning data redistribution. Therefore, we are planning to use quasi-tweets (in Japanese) for 8 diseases/symptoms by means of a crowdsourcing. We also generate English and Chinese corpus by translating a part of quasi-tweets from Japanese into English and Chinese.

(1)Training Data(May 1~)(Revised on July 5)

Training data corpus consists of 1,920 tweet texts (75% of the whole corpus) with labels. Each tweet is attached Positive:p or Negative:n labels for 8 diseases/symptoms, respectively.

An example of training data
ID Tweet Influenza Diarrhea Hayfever Cough Headache Fever Runnynose Cold
8888ja I’m so down with the flu. p n n n n p n n

(2) Test Data (July 24~)

Test data corpus consists of 640 tweet texts (25% of the whole corpus) without labels.

Important Dates

Aug 24, 2016
NTCIR-13 Kick-off event in Tokyo: Introduction of MedWeb (O)(P)
Mar 31, 2017
Task Registration Deadline (P) (Extended)
Apr 3, 2017
Annotation Guideline Distribution (O)
May 1, 2017
Training Corpus Distribution (O)
May 1-Jul 24, 2017
Dry Run (P)
Jul 24, 2017
Test Data Distribution (O)
Jul 24-Aug 7, 2017
Formal Run (P)
Aug 7, 2017
Run Result Submission Due Date (P)
Sep 4, 2017
Evaluation Result Release (O)
Sep 18, 2017
Early Draft Task Overview Release (O)
Sep 25, 2017
Task Participant Paper (Draft) Submission Due Date (P)
Oct 9, 2017
Paper Check and Notification (O)
Nov 1, 2017
Task Participant Paper (Camera Ready) Submission Due Date (P)
Dec 5-8, 2017
NTCIR-13 Conference @ NII, Tokyo, Japan. (O)(P)
MedWeb task session will be held on Thursday, December 7, 2017 at 11:00 a.m. to 1:00 p.m. (JST). Poster session is scheduled after the task session at 1:00 p.m. to 2:30 p.m. (JST).
*(P) and (O) indicate dates that should be done by participants and organizers, respectively.


Best System Award [Award certificate]

Hayate Iso, Camille Ruiz, Taichi Murayama, Katsuya Taguchi, Ryo Takeuchi, Hideya Yamamoto, Shoko Wakamiya and Eiji Aramaki
(NTCIR13 MedWeb Task: multi-label classification of tweets using an ensemble of neural networks)

Best Student Award [Award certificate]

Reine Asakawa and Tomoyoshi Akiba
(AKBL at the NTCIR-13 MedWeb Task)


ARAMAKI Eiji, Ph.D. (Nara Institute of Science and Technology
WAKAMIYA Shoko, Ph.D. (Nara Institute of Science and Technology
MORITA Mizuki, Ph.D. (Okayama University
KANO Yoshinobu, Ph.D. (Shizuoka University
OHKUMA Tomoko, Ph.D. (Fuji Xerox)


MASUICHI Hiroshi, Ph.D. (Fuji Xerox)


Nara Institute of Science and Technology


NII IDR (MedWeb Test Collection is downloadable)





NII (National Institute of Informatics)