Lydia/TextMap Publications

Websites presenting Lydia analysis include:

• TextMap identifies temporal and geographic trends in entities such as people, places, and things by analyzing roughly 500 daily English language newspapers.
• TextMed identifies relationships between medical and biological entities through analysis of PubMed/Medline abstracts.
• TextBlog identifies trends and entity relationships through large-scale analysis of blogs.
• TextBiz uses random-walk models to generate a probability distribution on the future prices for all NASDAQ, NYSE, and AMEX stocks. We soon should add analysis of business news sources.

See our October 2009 overview presentation on the Lydia project. Research papers describing our work are:

• Expanding Network Communities from Representative Examples (with A. Mehler). ACM Trans. Knowledge Discovery from Data (TKDD), Special Issue on Social Computing, Behavioral Modeling, and Prediction.
• Name-Ethnicity Classification from Open Sources (with A. Ambekar, C. Ward, J. Mohammed, and S. Male) 15th ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD 2009), Paris France, June 28-July 1, 2009.
• Improving Movie Gross Prediction Through News Analysis (with W. Zhang) IEEE/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology (WI 2009), Milan Italy, September 15-18, 2009.
• Identifying Differences in News Coverage Between Cultural/Ethnic Groups (with C. Ward and M. Bautin) News Analysis Workshop of IEEE/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology (WI 2009), Milan Italy, September 15-18, 2009.
• International Sentiment Analysis for News and Blogs (with M. Bautin and L. Vijayarenu). Second Int. Conf. on Weblogs and Social Media (ICWSM 2008), Seattle WA, March 26-28, 2008.
• Large-Scale Sentiment Analysis for News and Blogs (with N. Godbole and M. Srinivasaiah). Int. Conf. on Weblogs and Social Media (ICWSM 2007), Denver CO, March 26-28, 2007. Also see our system demonstration description.
• Concordance-Based Entity-Oriented Search (with M. Bautin) IEEE/ACM Web Intelligence (WI-07), Silicon Valley CA, November 2-5, 2007. Full version to appear in Web Intelligence and Agent Systems: An International Journal
• Newspapers vs. Blogs: Who Gets the Scoop? by L. Lloyd, P. Kaulgud, and S. Skiena, AAAI Symp. Computational Approaches to Analysing Weblogs (AAAI-CAAW 2006), Stanford University, March 27-29, 2006 provides an comparison of entity frequencies between blogs and more formal news sources.
• Identifying co-referential Names Across Large Corpora by L. Lloyd, A. Mehler, and S. Skiena, Proc. Combinatorial Pattern Matching (CPM 2006) discusses our method for identifying synonym sets of entities.
• Spatial Analysis of News Sources, by A. Mehler, Y. Bao, X. Li, Y. Wang, and S. Skiena, IEEE Trans. Visualization and Computer Graphics 12 (2006) 765-772 discusses our ``heatmap'' analysis.
• Lydia: A System for Large-Scale News Analysis by L. Lloyd, D. Kechagias, and S. Skiena, 12th Symp. of String Processing and Information Retrieval, (SPIRE '05), Lecture Notes in Computer Science, 3772 (2005) 161-166 provides an overview of the architecture of the Lydia system as of May 2005.
• Question Answering with Lydia by J. Kil, L. Lloyd, and S. Skiena, 14th Text REtrieval Conference (TREC 2005), NIST Gaithersburg MD, November 15-18, 2005 describes an extension to Lydia for answering factoid, list, and open-ended English-language questions.