The Java Source

Real-Time Topic Modeling of Microblogs

March 28, 2013 By: Janice Heiss

A new article published on the front page of otn/java, by Yogesh Tewari and Rajesh Kawad, of Infosys Limited Labs in Bangalore, India, titled “Real-Time Topic Modeling of Microblogs,” explores “the challenge of real-time extraction of topics from a continuous stream of incoming microblogs or tweets that are particular to an application” that they created. From a simple tweet text, the application is designed to accurately suggest relevant topics discussed in the tweet, and provide real-time timelines of topics generated from the tweet streams.

They explain that this is no simple tasks since a tweet, “considered as a text corpus, contains only 140 characters and second, given their brevity, tweets may not provide useful information and may contain different forms of text such as ‘smileys’ and short-form URLs. Finally, tweets are generated in real time.”

Yogesh and Rajesh apply LDA (latent Dirichlet allocation) to topic model tweets and make use of the Machine Learning for Language Toolkit (MALLET) API as the implementation for LDA – all performed in a Java environment. The LDA implementation is in turn encapsulated within the MALLET API, which here functions as a command line–based Java tool.

As they state: “Our targets are the actual Java classes that perform the LDA logic whose methods we invoke with required input in real-time. Storm is our choice of a free and open source distributed real time computation engine implemented in Java and running in a distributed mode. Storm is highly scalable and easily capable of handling incoming tweet streams. We use Twitter4J to stream tweets, which require valid Twitter authentication. So our task is to design a topology that will consume tweet streams and output a timeline of topics.”
Check out the article here.

Join the discussion

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha

Related

More on Java

Integrated Cloud Applications & Platform Services