Implementation of chat log browser – GSoC Week 2

Last week’s task was to implement chat log browser. It basically required working on three things, namely search API, UI and their integration.

Search API

The first thing I had to do was discuss with mentor and finalize API endpoints. We ended up with only one endpoint.

 /api/search/{server-name}/{channel-name-without-#}/?{get-params}

  • {server-name} is eg. Freenode and it is case-sensitive in search query.
  • {channel-name-without-#} is eg. fedora-admin and it is case-sensitive too in search query.
  • {get-params} All of the below get parameters are optional.
    • message: search term eg. ‘trouble installing Fedora’
    • from: message sent by eg. dne0
    • to: message sent from: rtnpro
    • dateFrom: message was sent after date eg. 1 May 2014
    • dateTo: message was sent before date eg. 10  May 2014
    • page: for pagination

Obviously, all these parameters are first validated before querying from elasticsearch. And then results are returned in JSON format as shown below:

{
  status: <bool>,
  errors: <array>,
  results: {
    totalCount: <int>,
    perPage: <int>,
    took: <int>, // time taken in milliseconds
    data: <array of objects>
  }
}

For routing we are using Iron Router, but there are some issues with its server side router. I hope they get resolved quickly so that I can refactor API code and make it look more cleaner 🙂

Search UI

Being a novice in designing and HTML/CSS, I tried not to waste much time on fonts/color scheme and instead focused more on adding possible search fields which user might want to use to narrow down search results. Currently, UI contains option to select channel, search by message and some advance search options like date, from, to etc.

UI & API Integration

Meteor provides a http package for sending asynchronous requests. I have used this package in my code to call API. As written earlier, the results we get are in JSON format, this JSON is rendered on client side using UI.toHTML api of meteor.

callAPI: function (serverName, channelName, getParams) {
  var API_URL = this.getAPIEnpoint(serverName, channelName, getParams);
  $('.chatlogs-loader-msg').show();
  Meteor.http.get(API_URL, function (err, body) {
    if (!err) {
      waartaa.search.helpers.renderResponse(body);
    } else {
      alert('OOPS! An error occured while fetching data.');
    }
    $('.chatlogs-loader-msg').fadeOut(500);
  });
},

Conclusion

With this, the first version of chat log browser is ready. Here is the PR:  https://github.com/waartaa/waartaa/pull/113

Screenshot from 2014-06-04 15:09:15

Screenshot from 2014-06-04 12:45:09

Please note search terms are highlighted. Thanks to elasticsearch’s highlight search terms feature which made it easy to implement.

PS: Any suggestions for improving the search interface or anything else are welcome 🙂

What’s left?

  • Refactoring API code.
  • Authentication mechanism in API.
  • Improving UI.
  • Implementing chat log permalink and bookmarking feature.

<script>JS I love you</script>

Basic Waartaa-ElasticSearch Integration – GSoC Week 1

ElasticSearch is a flexible and powerful open source, distributed, real-time search and analytics engine. Waartaa-ElasticSearch integration is the first sub-task which begins the implementation of browse/search channel chat logs. After the GSoC bonding period was over I started working on it. Here is what I did in last week.

Creating mapping for Channel Logs index

Last week I spent most of my time reading ElasticSearch documentation and tutorials. I have got good understanding of what ElasticSearch is capable of. Mapping is telling ElasticSearch which field in document is searchable, defining their type and few more things. Here is the first version of mapping that I created. It will change over time as new fields may get added in document and we may also change the analyzer used for ‘message’ field to account for misspelled search terms like Google does it.

Indexing Channel Logs

If you have read my last blog post, you will know I already completed this task during GSoC community bonding period. In last week I just refactored it a bit, made it configurable in settings file and tested the code on my system. My mentor told me he is facing some issue with ElasticSearch secure installation. He will merge my PR as soon as installation is done.

Oh wait I forgot to tell you how I will be transferring older chat logs from MongoDB to ElasticSearch. One way is to write a script but there is always a better way to do such things. There is an open source river plugin available which does exactly what I want.

Generating permalinks for incoming chat messages

Each chat log/message will have it’s own unique permalink that users can send to each other to refer to some old conversation. There are many other use cases of it which I will not be discussing here now. There wasn’t much to do in this task because these are generated automatically in form of  ‘_id’ field of document that is indexed in ElasticSearch.

Conclusion

In terms of code, basic Waartaa-ElasticSearch integration is complete. I hope the problem with ElasticSearch installation gets resolved asap so that I can see my code in action in production 🙂

What’s next?

Next, I will be working on building an API to search/browse channel logs. Hopefully I will complete it by end of this week and will write a blog post about it too 🙂

<script>JS I love you</script>