Thursday, November 4, 2010

1st CFP for NTCIR-9 INTENT and 1CLICK out now!

NTCIR-9 runs seven tasks:
Intent (INTENT), Recognizing Inference in TExt (RITE), Geotemporal Information Retrieval (GeoTime), IR for Spoken Documents (SpokenDoc), Cross-Lingual Link Discovery (CrossLink), Interactive Visual Exploration (Vis-Ex) and Patent Machine Translation (PatentMT)

Hideo Joho and I are working as the evaluation co-chairs of NTCIR-9.

Also, I am one of the task organisers of
INTENT and One Click Access (1CLICK) which is officially a subtask of INTENT but actually more like a separate task.

INTENT Subtasks:
Subtopic Mining - given a query, list up possible intents;
Document Ranking - selectively diversify search results to cover different intents
1CLICK - return a fixed length text that immediately satisfies the user


We have started distributing the first call for participation for INTENT and 1CLICK.
Please visit the above pages and consider participating.

If you are interested in any of the NTCIR-9 tasks,
please note that the task participant registration closes on December 20, 2010.

The NTCIR-9 final meeting will take place December 6-9, 2011 at NII, Tokyo, Japan.

Friday, August 13, 2010

Accepted NTCIR-9 Tasks

The NTCIR-9 task selection commitee and
the evaluation chairs have accepted
the following tasks.
Updates will be announced at the
NTCIR website.

CORE TASKS

Intent (with One-Click Access subtask)

Recognizing Natural Language Inference

Geotemporal information retrieval

IR for Spoken Documents

PILOT TASKS

Cross-lingual Link Discovery

Evaluation Framework of Interactive Information Access

Patent Machine Translation

Friday, May 21, 2010

My publications coming up at EVIA 2010 / NTCIR-8

Just to remind myself...

Sakai, T., Craswell, N., Song, R., Robertson, S., Dou, Z. and Lin, C.-Y.:
Simple Evaluation Metrics for Diversified Search Results,
EVIA 2010 Proceedings, 2010.

Sakai, T. and Lin, C.-Y.:
Ranking Retrieval Systems without Relevance Assessments – Revisited,
EVIA 2010 Proceedings, 2010.

Song, R., Qi, D., Liu, H., Sakai, T., Nie, J.-Y., Hon, H.-W. and Yu, Y.:
Constructing a Test Collection with Multi-Intent Queries,
EVIA 2010 Proceedings, 2010.

Gey, F., Larson, R., Kando, N., Machado, J. and Sakai, T.:
NTCIR-GeoTime Overview: Evaluating Geographic and Temporal Search,
NTCIR-8 Proceedings, 2010.

Ishikawa, D., Sakai, T. and Kando, N.:
Overview of the NTCIR-8 Community QA Pilot Task (Part I): The Test Collection and the Task,
NTCIR-8 Proceedings, 2010.

Mitamura, T., Shima, H., Sakai, T., Kando, N., Mori, T., Takeda, K., Lin, C.-Y., Song, R., Lin, C.-J. and Lee, C.-W.:
Overview of the NTCIR-8 ACLIA Tasks: Advanced Cross-Lingual Information Access,
NTCIR-8 Proceedings, 2010.

Sakai, T., Ishikawa, D. and Kando, N.:
Overview of the NTCIR8 Community QA Pilot Task (Part II): System Evaluation,
NTCIR-8 Proceedings, 2010.

Sakai, T., Shima, H., Kando, N., Song, R., Lin, C.-J., Mitamura, T., Sugimoto, M. and Lee, C.-W.:
Overview of NTCIR-8 ACLIA IR4QA,
NTCIR-8 Proceedings, 2010.

Song, Y.-I., Liu, J., Sakai, T., Wang, X.-J., Feng, G., Cao, Y., Suzuki, H. and Lin, C.-Y.:
Microsoft Research Asia with Redmond at the NTCIR-8 Community QA Pilot Task,
NTCIR-8 Proceedings, 2010.

They will be available online at
http://research.nii.ac.jp/ntcir/ntcir-ws8/ws-en.html

Taipei pictures

担担面非常好吃!

もうイカ口調


Yasu buying fruits






故宫


Banquet food - no, not Matsuzaka beef, but "Matsuzaka pork" ...
and the English translation says "Kobe pork"...


谈水





Monday, May 10, 2010

Paris pictures






Fujita-san




This cheese was exquisite. Felt as if I was embracing a goat.











Jaana, co-inventor of nDCG at the RIAO banquet



Fujita-san at RIAO 2010, discussing curry


Too much mashed potatoes

Friday, April 23, 2010

CFP: AIRS2010

CALL FOR PAPERS (AIRS 2010; Taipei, Taiwan)

The Sixth Asia Information Retrieval Societies Conference (AIRS 2010)
December 1-3, 2010 at Conference Hall, National Taiwan University, Taipei, Taiwan
In cooperation with ACM SIGIR and ACLCLP ROCLING

http://irlab.csie.ntu.edu.tw/airs2010

-----------------------------------------------------------------------------------------
The Asia Information Retrieval Societies Conference (AIRS) aims to bring together
researchers and developers to exchange new ideas and latest achievements in the
field of information retrieval (IR). The scope of the conference covers
applications, systems, technologies and theory aspects of information retrieval
in text, audio, image, video, and multimedia data. The AIRS 2010 welcomes
submissions of original papers in the broad field of information retrieval.
Technical issues covered include, but are not limited to the following:

1. IR Models and Theories

2. User Study, IR Evaluation, and Interactive IR

3. Web IR, Scalability, and Adversarial IR

4. Multimedia IR

5. NLP for IR (eg. Cross-/Multi- Language IR, Question Answering,
Summarization, Information Extraction)

6. Machine Learning and Data Mining for IR (eg. Learning to Rank, Classification,
Clustering)

7. IR Applications (eg. Digital Libraries, Vertical Search, Mobile IR)

Accepted papers will be published as part of the LNCS series by Springer, and will be EI-indexed.

IMPORTANT DATES

June 30, 2010 Submissions due
July 31, 2010 Notification of acceptance
August 14, 2010 Camera-ready due
November 1, 2010 Registration deadline
December 1-3, 2010 AIRS 2010

SUBMISSION INSTRUCTIONS

The AIRS 2010 proceedings will be published as an LNCS volume, so please follow
the default author instructions available at

http://www.springer.com/computer/lncs?SGWID=0-164-7-72376-0

In addition, please anonymize your paper to facilitate blind reviewing,
and make sure your paper is no longer than 12 pages in the LNCS format.
Submissions that do not follow these guidelines will be rejected unconditionally.

Duplicate submissions (the same paper being submitted to AIRS 2010 and
to another conference at the same time) are strictly forbidden; if
detected, these submissions will be unconditionally rejected.

For paper submission and related information, please visit the AIRS 2010 website.

ORGANIZATION

Honorary Conference Chair:
Si-Chen Lee, President, National Taiwan University

Conference Chair:
Pu-Jen Cheng, National Taiwan University

Publication Chair:
Preslav Nakov, National University of Singapore

Finance Chair:
Lun-Wei Ku, National Taiwan University

Program Committee Co-Chairs:
Wai Lam, Chinese University of Hong Kong
Min-Yen Kan, National University of Singapore

Webmaster:
Chien-Wen Chen, National Taiwan University


AREA CHAIRS

IR Models and Applications
Min Zhang, Tsinghua University
Bin Wang, Chinese Academy of Sciences

User Study, IR Evaluation, and Interactive IR
Tetsuya Sakai, Microsoft Research Asia
William Webber, University of Melbourne

Web IR, Scalability, and Adversarial IR
Joemon Jose, University of Glasgow

Multimedia IR
Winston Hsu, National Taiwan University
Kazunari Sugiyama, National University of Singapore

NLP for IR
Timothy Baldwin, University of Melbourne
Atsushi Fujii, Tokyo Institute of Technology

Machine Learning and Data Mining for IR
Tie-Yan Liu, Microsoft Research Asia
Yi Zhang, University of California Santa Cruz

Thursday, April 22, 2010

Papers accepted for EVIA 2010

Sukomal Pal , Mandar Mitra and Samaresh Maiti. Estimating Pool-depth on Per Query Basis (SHORT)

William Webber, Alistair Moffat and Justin Zobel . The Effect of Pooling and Evaluation Depth on Metric Stability

Maristella Agosti , Giorgio Maria Di Nunzio, Marco Dussin and Nicola Ferro . 10 Years of CLEF Data in DIRECT: Achievements and Perspectives (SHORT)

Tetsuya Sakai and Chin-Yew Lin. Ranking Retrieval Systems without Relevance Assessments – Revisited

Ian Soboroff. Test Collection Diagnosis and Treatment

Tetsuya Sakai, Nick Craswell , Ruihua Song, Stephen Robertson, Zhicheng Dou and Chin-Yew Lin. Simple Evaluation Metrics for Diversified Search Results

Ruihua Song, Dongjie Qi, Hua Liu, Tetsuya Sakai, Jian-Yun Nie, Hsiao-Wuen Hon and Yong Yu . Constructing a Test Collection with Multi-Intent Queries

Katrin Lamm, Thomas Mandl , Christa Womser-Hacker and Werner Greve. The Influence of Expectation and System Performance on User Satisfaction with Retrieval Systems

Keun Chan Park , Jihee Ryu , Kyung-min Kim and Sung Hyon Myaeng . An Evaluation Method for Subjective Tasks

Tuesday, April 20, 2010

Looking for a light laptop?

My five-year-old has designed and assembled this cool laptop.
I've tried it - it's extremely light and small and perfect
for business trips. Runs Windows 7; PowerPoint preinstalled.

Tuesday, April 6, 2010

My important dates 2010 version 3

Apr 06 NTCIR-8 CQA runs due
Apr 08 NTCIR-8 ACLIA participants papers due
Apr 16 IIiX 2010 reviewers assigned
Apr 18 EVIA 2010 reviews due
Apr 20 EVIA 2010 acceptance notification
Apr 20 NTCIR-8 CQA pilot task results released
Apr 23 NTCIR-8 ACLIA feedback from organisers to participants
Apr 28-30 RIAO 2010 [Paris]
Apr 30 NTCIR-8 early registration deadline
May 02 NTCIR-8 CQA pilot task participants papers due
May 02 CLEF 2010 submissions due
May 06 NTCIR-8 CQA pilot task feedback from organisers to participants
May 10 EVIA 2010 final manuscripts due
May 15 NTCIR-8 final manuscripts due
May 16 IIiX 2010 reviews due
May 27 CIKM 2010 abstracts due
May 28 CLEF 2010 review deadline
Jun 3 CIKM 2010 papers due
Jun 15 EVIA 2010 [Tokyo]
Jun 15-18 NTCIR-8 [Tokyo]
Jun 30? AIRS 2010 submissions due
Jul 3? AIRS 2010 reviewer assignment
Jul 15 CIKM 2010 acceptance notification
Jul 18-23 SIGIR 2010 [Geneva]
Jul 25? AIRS 2010 reviews due
Jul 28? AIRS 2010 metareviews due
Jul 31 AIRS 2010 acceptance notification
Aug 4 DBS +IFAT@Aoyama Gakuin U
Aug 14 AIRS 2010 camera-ready
Aug 15 CIKM 2010 camera-ready
Oct 27-29 CIKM 2010 [Toronto]
Nov 1 AIRS 2010 registration deadline
Dec 1-3 AIRS 2010 [Taipei]

Friday, April 2, 2010

NTCIR-9 "Shoten" Task?

NTCIR-8 Community QA Pilot Taskというもののオーガナイザ兼参加者になった関係で、Yahoo!知恵袋のデータを眺めたりしているのだが、例えば

Q: 世界は僕中心に回っていますか?
Best Answer: いいえ私を中心に回っています。<以下略>

のように、Qの有用性が低かったり、質問者が選んだbest answerがゴミだったりすることが結構多い。そこで上記タスクでは、best answerによるシステム評価とは別に、4人の判定者に各回答の質をなるべく客観的に絶対評価してもらったデータによるシステム評価も行うことにしている。

ときに、NTCIRでは「メディアで話題になるような」タスクが求められていると聞く。研究のmotivationがそんなんでいいのか!?というつっこみはさておき、とりあえず「笑点タスク」なんてどうでしょう。つまり、歌丸師匠が誰に座布団をあげるかを推定するタスク。過去の座布団獲得枚数からauthorityを計算したりして。テレビ受けするんじゃないかな。でもこのタスク、外人にどう説明すればよいのだ。

でもUGCっておもしろいね。NTCIR-9ではMOATあたりと連携して、Community QA Taskをもっと盛り上げよう。

Tuesday, March 30, 2010

EVIA 2010 deadline extended

***** DEADLINE EXTENDED TO APRIL 2 (timezone: GMT-12)! *****

The Third International Workshop on Evaluating Information Access
(EVIA 2010), a Satellite Workshop of NTCIR-8
http://research.nii.ac.jp/ntcir/ntcir-ws8/EVIA-2010/

June 15, 2010 at NII, Tokyo, Japan.
*****************************************************************

Information Access technologies provide the interface between
human information needs and digital information resources. The
reliable evaluation of these technologies has been recognized for
decades as central to the advancement of the field. As information
retrieval becomes more pervasive, the forms of retrieval
more diverse, and retrieval tools richer, the importance of
effective, efficient, and innovative evaluation grows as well.
In recognition of the need for a forum dedicated to the intensive
discussion of evaluation issues, we have held two successful
International Workshops on Evaluating Information Access (EVIA
2007 and 2008). Proceedings and reports can be found below:

EVIA 2007 online proceedings:
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings6/EVIA/
EVIA 2008 online proceedings:
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings7/toc_evia.html

SIGIR Forum Report on EVIA 2007:
http://sigir.org/forum/2007D/2007d_sigirforum_sanderson.pdf
SIGIR Forum Report on EVIA 2008:
http://sigir.org/forum/2009J/2009j-sigirforum-sakai.pdf

Following these successes, we are happy to announce the Third
International Workshop on Evaluating Information Access (EVIA
2010) on June 15, 2010 (coinciding with Day 1 of the NTCIR-8
Workshop Meeting) at the National Institute of Informatics (NII),
Tokyo, Japan. We invite papers addressing one or more of the
following topics:

- Test collection formation, evaluation metrics, and evaluation
environments
- Statistical issues in retrieval evaluation
- User studies and the evaluation of human-computer interaction
in information retrieval (HCIR)
- Evaluation methods for multilingual, multimedia, or mobile
information access
- Novel information access tasks and their evaluation
- Evaluation and assessment using implicit user feedback,
crowdsourcing, living labs, or inferential methods
- Evaluation issues in industrial and enterprise retrieval
systems

*** SUBMISSION INSTRUCTIONS ***

Submissions will be subject to double-blind reviewing. Accepted
papers will be included in the EVIA 2010 online proceedings, and
summarized in a report to SIGIR Forum.

We invite submissions of regular papers (up to 9 pages) and
short papers (up to 5 pages).
Submission upload page:
http://www.easychair.org/conferences/?conf=evia2010

*** IMPORTANT DATES (NEW!) ***

- April 2, 2010 *submissions due*
- April 20, 2010 notification of acceptance

- May 10, 2010 final manuscripts due
- June 15, 2010 EVIA 2010@NII, Tokyo, Japan
- June 15-18, 2010 NTCIR-8@NII, Tokyo, Japan

*** EVIA 2010 PROGRAM COMMITTEE ***

Akiko Aizawa NII Japan
Javed Aslam Northeastern University USA
Nicholas Belkin Rutgers University USA
Ben Carterette University of Delaware USA
Charlie Clarke University of Waterloo Canada
Georges Dupret Yahoo! Research Chile
Norbert Fuhr Universitaet Duisburg-Essen Germany
Atsushi Fujii Tokyo Institute of Technology Japan
Donna Harman NIST USA
Hideo Joho University of Tsukuba Japan
Gareth Jones Dublin City University Ireland
Noriko Kando NII Japan
Evangelos Kanoulas University of Sheffield UK
Jaana Kekalainen University of Tampere Finland
Mounia Lalmas University of Glasgow UK
Gary Geunbae Lee POSTECH Korea
Mun-Kew Leong National Library Board Singapore
Jimmy Lin University of Maryland USA
Chin-Yew Lin Microsoft Research Asia China
Iadh Ounis University of Glasgow UK
Carol Peters IEI-CNR Italy
Ian Ruthven University of Strathclyde UK
Tetsuya Sakai Microsoft Research Asia China
Mark Sanderson University of Sheffield UK
Falk Scholer RMIT University Australia
Ian Soboroff NIST USA
Paul Thomas CSIRO Australia
William Webber University of Melbourne Australia
Emine Yilmaz Microsoft Research Cambridge UK

*** EVIA 2010 CO-CHAIRS ***

William Webber (University of Melbourne, Australia)
wew
at
csse.unimelb.edu.au
Mark Sanderson (University of Sheffield, UK)
m.sanderson
at
sheffield.ac.uk
Tetsuya Sakai (Microsoft Research Asia, China)
tetsuyasakai
at
acm.org

Wednesday, March 24, 2010

FINAL CFP: EVIA 2010

***** FINAL CALL FOR PAPERS *****

The Third International Workshop on Evaluating Information Access
(EVIA 2010), a Satellite Workshop of NTCIR-8
http://research.nii.ac.jp/ntcir/ntcir-ws8/EVIA-2010/

June 15, 2010 at NII, Tokyo, Japan.
***************************************

Information Access technologies provide the interface between
human information needs and digital information resources. The
reliable evaluation of these technologies has been recognized for
decades as central to the advancement of the field. As information
retrieval becomes more pervasive, the forms of retrieval
more diverse, and retrieval tools richer, the importance of
effective, efficient, and innovative evaluation grows as well.
In recognition of the need for a forum dedicated to the intensive
discussion of evaluation issues, we have held two successful
International Workshops on Evaluating Information Access (EVIA
2007 and 2008). Proceedings and reports can be found below:

EVIA 2007 online proceedings:
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings6/EVIA/
EVIA 2008 online proceedings:
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings7/toc_evia.html

SIGIR Forum Report on EVIA 2007:
http://sigir.org/forum/2007D/2007d_sigirforum_sanderson.pdf
SIGIR Forum Report on EVIA 2008:
http://sigir.org/forum/2009J/2009j-sigirforum-sakai.pdf

Following these successes, we are happy to announce the Third
International Workshop on Evaluating Information Access (EVIA
2010) on June 15, 2010 (coinciding with Day 1 of the NTCIR-8
Workshop Meeting) at the National Institute of Informatics (NII),
Tokyo, Japan. We invite papers addressing one or more of the
following topics:

- Test collection formation, evaluation metrics, and evaluation
environments
- Statistical issues in retrieval evaluation
- User studies and the evaluation of human-computer interaction
in information retrieval (HCIR)
- Evaluation methods for multilingual, multimedia, or mobile
information access
- Novel information access tasks and their evaluation
- Evaluation and assessment using implicit user feedback,
crowdsourcing, living labs, or inferential methods
- Evaluation issues in industrial and enterprise retrieval
systems

*** SUBMISSION INSTRUCTIONS ***

Submissions will be subject to double-blind reviewing. Accepted
papers will be included in the EVIA 2010 online proceedings, and
summarized in a report to SIGIR Forum.

We invite submissions of regular papers (up to 9 pages) and
short papers (up to 5 pages).
Submission upload page:
http://www.easychair.org/conferences/?conf=evia2010

*** IMPORTANT DATES ***

- March 31, 2010 *submissions due*
- April 18, 2010 notification of acceptance
- May 10, 2010 final manuscripts due
- June 15, 2010 EVIA 2010@NII, Tokyo, Japan
- June 15-18, 2010 NTCIR-8@NII, Tokyo, Japan

*** EVIA 2010 PROGRAM COMMITTEE ***

Akiko Aizawa NII Japan
Javed Aslam Northeastern University USA
Nicholas Belkin Rutgers University USA
Ben Carterette University of Delaware USA
Charlie Clarke University of Waterloo Canada
Georges Dupret Yahoo! Research Chile
Norbert Fuhr Universitaet Duisburg-Essen Germany
Atsushi Fujii Tokyo Institute of Technology Japan
Donna Harman NIST USA
Hideo Joho University of Tsukuba Japan
Gareth Jones Dublin City University Ireland
Noriko Kando NII Japan
Evangelos Kanoulas University of Sheffield UK
Jaana Kekalainen University of Tampere Finland
Mounia Lalmas University of Glasgow UK
Gary Geunbae Lee POSTECH Korea
Mun-Kew Leong National Library Board Singapore
Jimmy Lin University of Maryland USA
Chin-Yew Lin Microsoft Research Asia China
Iadh Ounis University of Glasgow UK
Carol Peters IEI-CNR Italy
Ian Ruthven University of Strathclyde UK
Tetsuya Sakai Microsoft Research Asia China
Mark Sanderson University of Sheffield UK
Falk Scholer RMIT University Australia
Ian Soboroff NIST USA
Paul Thomas CSIRO Australia
William Webber University of Melbourne Australia
Emine Yilmaz Microsoft Research Cambridge UK

*** EVIA 2010 CO-CHAIRS ***

William Webber (University of Melbourne, Australia)
wew
at
csse.unimelb.edu.au
Mark Sanderson (University of Sheffield, UK)
m.sanderson
at
sheffield.ac.uk
Tetsuya Sakai (Microsoft Research Asia, China)
tetsuyasakai
at
acm.org

Monday, March 22, 2010

EVIA 2010 submission site is ready

I've set up a submission upload page for EVIA 2010.
Please read the instructions on
http://research.nii.ac.jp/ntcir/ntcir-ws8/EVIA-2010/
first and then upload via
https://www.easychair.org/login.cgi?conf=evia2010
(We will soon provide a link from the EVIA main page to the submission page.)

Winter's gone, and now this...

Monday, March 8, 2010

Where is Spring?

This is not smog - it's snowing again...

Future of Flight Aviation Center & Boeing Tour

飛行機を作るところを見てきました。スケールは大きいのですが、おもいっきり手作り感があってびっくりしました。
飛行機は、機械ではなく人間が作っているのだと感じました。
(肝心の工場内は撮影禁止でした。)


Monday, March 1, 2010

Wikipedia for Breakfast?

Wekipedia actually.

Paul will never die

No more war, please...

夜中の1時をすぎてもどんどんぱちぱち。
ベランダに出ると火傷します。
もう花火と爆竹はうんざり。
来年のこの時期は国外に避難しようっと