Bookmarking chat logs in waartaa – GSoC post-midterm

Post-midterm phase of GSoC has already begun and there is still lot of work to be done(mostly, UI improvement and deploying all my previous work on server).

Lately, I wasn’t getting much time but somehow I have managed to add bookmarking feature in waartaa. I have added support for both single as well as multiple bookmarking in both live chat page and search page.

Single Bookmark

Beside every chat message, there appears a bookmark icon on hover. When user clicks on it, it gets bookmarked(in front-end only) and a popup appears on top of chat window which has field ‘Label’ with default value equal to chat message’s date-time, a ‘Done’ button to save data in db and ‘Cancel’ button for obvious reason.

Multiple Bookmarks

It happens many times when user wants to bookmark multiple chat messages under one label, for instance, he wants to save a conversation happened in some random IRC channel. Its easy to bookmark multiple messages in waartaa. You just have to choose two endpoints of a conversation and long click(atleast one second) one of them and normal click the other one. This will bookmark all messages in between along with the endpoints.

Bookmarks model

/*                                              
Bookmarks {                                     
  label: String, // Bookmark label                                
  roomType: String ('channel'/'pm'/'server'),   
  logIds: List, // chat messages id               
  user: String, // username of user for whom bookmark is created                                
  userId: String,  // user id                             
  created: Datetime,                   
  lastUpdated: Datetime,                        
  creator: String, // username of user who created bookmark                   
  creatorId: String                             
}                                               
*/                                            

Screenshots

I know there isn’t much you can infer from below screenshots but this is all I have right now to share with you.

single-bookmark

Single bookmarking

multiple-bookmark

Multiple bookmarking

Conclusion

With this, bookmarking feature is complete and here is the PR 129.

<script>JS I love you.</script>

QA Testing Video Chat App – GSoC Week 4

I would recommend you to read my last post first before reading this one. This should help you understand the content of this post better🙂

Last week, I did both dev testing and load testing of video chat app. In this post, I am gonna show you the results I got.

Bugs & Fixes

Bugs I found during testing:-

  • Re-rendering video object: When a user moves from video chat app to an irc channel/server and then back to video chat again, video stream(both local and remote) doesn’t get re-render. Actually,  this was a browser specific issue and I found it’s fix by chance. Have a look at the fix.
  • Click event on ‘Accept/Reject’ button gets triggered more than once: This one was simple to fix. I just had to use jQuery off method to remove click event attached to element before attaching new click event. Fixed.
  • Option to enable/disable video chat: I know this isn’t a bug. My mentor told me to add this feature.

Load Testing

Framework

In my last post, I mentioned I used EasyRTC framework to build p2p(peer to peer) video chat. Although, it’s a p2p app, it still requires server for signaling(read more). For signaling, EasyRTC uses socket.io built on node.js. So, basically I load tested EasyRTC server’s socket.io implementation.

I used socket.io-client module to interact/connect with EasyRTC server by writing against their server side API, as documented here. Below is the snippet of simple client I created.

var Client = function () {
  /* Some code omitted */
  this.connectToServer = function (callback) {
    var client = io.connect(
        'http://'+SERVER_HOST+':'+SERVER_PORT ,
        {'force new connection': true}
    );
    if (!client) {
      throw "Couldn't connect to socket server";
    }

    var msg = {
      msgType: 'authenticate',
      msgData: {
        apiVersion: "1.0.11",
        applicationName: "waartaa"
      }
    };

    var easyrtcAuthCB = function (msg) {
      if (msg.msgType == 'error') {
        callback('easyrtc authentication falied');
      } else {
        // code omitted
      }
    };

    client.json.emit('easyrtcAuth', msg, easyrtcAuthCB);
  };
}

Script & Parameters

I made a command line tool in nodejs to ran tests.

node run.js --no-of-connection=700 --no-of-concurrent-connections=[1,5,10]

In above mentioned command you should notice two things, namely, no. of connections and concurrent connection. My system(Intel(R) Core(TM) i5 CPU M 460 @2.53GHz, 4 GB RAM) couldn’t stand more than 700 connections. Server process started consuming 1 GB RAM for more than 700 connections. I will tell you the reason for this later. For now, 700 was suffice for initial testing. My mentor told me to keep concurrent connections very low because we won’t be expecting too high concurrency in production initially.

Data collected

After the script was ran several times, I collected the following data each time for different concurrency against no. of connections:-

  • Roundtrip time(ms): Time taken for client to get connected to server. This includes latency(which is negligible because client and server are on same machine) + client-to-server message time + server-to-client message time.
  • Avg. CPU load(%) & Memory (MB): I used this node module to collect these data.

Below you can see the data plotted to graph and their analysis:-

Roundtrip time analysis roundtrip_time

  • As no. of connections increases, round-trip time increases. This is because the default behavior of EasyRTC is to send every client in a room a separate update whenever a new client leaves a room, enters a room, or changes its status. And this leads to increase in round-trip time.
  • Increase in concurrency too increases round-trip time. I guess this is because nodejs being a single threaded server processes one request at a time. Please correct me if I am wrong.
  • Roundtrip time < 500 ms is a good no. For concurrency = 1 and any no. of connections in above graph, round-trip time < 500 ms.

CPU analysis cpu

  • Nothing too unnatural about cpu load increasing with no. of connections and concurrency.
  • Good thing is the rise is not too steep for lower concurrency.

Memory analysis memory

  • As I mentioned earlier, 700 connections consumed 1 GB memory and you can clearly see that in above graph. Well, this is because as I said earlier, the default behavior of EasyRTC is to send every client in a room a separate update whenever a new client leaves a room, enters a room, or changes its status. That means for 700 connections (700*701)/2 = 245350 messages were sent by server. You can do the math now🙂
  • For any concurrency, memory rises at the same rate with increase in no. of connections. Thus, memory consumption is independent of concurrency.

Conclusion

  • More bugs will pop up when my mentors and other people start using the app until then a developer can only believe his/her app works perfectly😉
  • The default behavior of EasyRTC server can cause scaling issues. We might have to change it in future by adding custom event listeners on server side.

PS: If you have made this far and found anything wrong in this post, please do comment.🙂

<script>JS I love you</script>

p2p Video Chat – GSoC Week 3

After finished implementing chat log search last week, I moved on to my next major task – implementing p2p video chat in waartaa.

Introduction

A p2p(peer to peer) plugin-free video/audio Real Time Communication(short for RTC) had always been a nightmare for developers to implement. This was changed few years back with the introduction of WebRTC – an open source, plugin-free, built into browsers technology. Today, there are many web services out there which uses webRTC eg. Bistri.

Implementation

A WebRTC application needs to do several things:

  • Get streaming audio, video or other data.
  • Get network information such as IP addresses and ports, and exchange this with other WebRTC clients (known as peers) to enable connection, even through NATs and firewalls.
  • Coordinate signaling communication to report errors and initiate or close sessions.
  • Exchange information about media and client capability, such as resolution and codecs.
  • Communicate streaming audio, video or data.

There are many libraries/frameworks available which does that for you. I am using EasyRTC because it is open source, maintains an active repo on GitHub and uses a node server for handling signaling.

Our initial goal was to use irc nicks for communication between peers but we had to settle using waartaa’s username for now because of trust/authentication issues with irc nicks. Imagine user has to first register his irc nick before starting video chat, that would have consumed alot time of user.

Screenshots

I have tried to tightlty integrate video chat UI with current UI so that user can do both video chat and normal chat(in an IRC channel) in same browser window simultaneously.

Client A
Client A1
Client A2
Client B1
Client B2

Conclusion

I am not sure if this is production ready yet because I haven’t done much research on how much load can easyrtc server handle. Also, by default easyrtc uses a public stun server of google to handle NAT and firewall between peers and that doesn’t sound good to me. It is quite possible that we might have to setup our own stun server.

PR: https://github.com/waartaa/waartaa/pull/119

<script>JS I love you</script>

Implementation of chat log browser – GSoC Week 2

Last week’s task was to implement chat log browser. It basically required working on three things, namely search API, UI and their integration.

Search API

The first thing I had to do was discuss with mentor and finalize API endpoints. We ended up with only one endpoint.

 /api/search/{server-name}/{channel-name-without-#}/?{get-params}

  • {server-name} is eg. Freenode and it is case-sensitive in search query.
  • {channel-name-without-#} is eg. fedora-admin and it is case-sensitive too in search query.
  • {get-params} All of the below get parameters are optional.
    • message: search term eg. ‘trouble installing Fedora’
    • from: message sent by eg. dne0
    • to: message sent from: rtnpro
    • dateFrom: message was sent after date eg. 1 May 2014
    • dateTo: message was sent before date eg. 10  May 2014
    • page: for pagination

Obviously, all these parameters are first validated before querying from elasticsearch. And then results are returned in JSON format as shown below:

{
  status: <bool>,
  errors: <array>,
  results: {
    totalCount: <int>,
    perPage: <int>,
    took: <int>, // time taken in milliseconds
    data: <array of objects>
  }
}

For routing we are using Iron Router, but there are some issues with its server side router. I hope they get resolved quickly so that I can refactor API code and make it look more cleaner🙂

Search UI

Being a novice in designing and HTML/CSS, I tried not to waste much time on fonts/color scheme and instead focused more on adding possible search fields which user might want to use to narrow down search results. Currently, UI contains option to select channel, search by message and some advance search options like date, from, to etc.

UI & API Integration

Meteor provides a http package for sending asynchronous requests. I have used this package in my code to call API. As written earlier, the results we get are in JSON format, this JSON is rendered on client side using UI.toHTML api of meteor.

callAPI: function (serverName, channelName, getParams) {
  var API_URL = this.getAPIEnpoint(serverName, channelName, getParams);
  $('.chatlogs-loader-msg').show();
  Meteor.http.get(API_URL, function (err, body) {
    if (!err) {
      waartaa.search.helpers.renderResponse(body);
    } else {
      alert('OOPS! An error occured while fetching data.');
    }
    $('.chatlogs-loader-msg').fadeOut(500);
  });
},

Conclusion

With this, the first version of chat log browser is ready. Here is the PR:  https://github.com/waartaa/waartaa/pull/113

Screenshot from 2014-06-04 15:09:15

Screenshot from 2014-06-04 12:45:09

Please note search terms are highlighted. Thanks to elasticsearch’s highlight search terms feature which made it easy to implement.

PS: Any suggestions for improving the search interface or anything else are welcome🙂

What’s left?

  • Refactoring API code.
  • Authentication mechanism in API.
  • Improving UI.
  • Implementing chat log permalink and bookmarking feature.

<script>JS I love you</script>

Basic Waartaa-ElasticSearch Integration – GSoC Week 1

ElasticSearch is a flexible and powerful open source, distributed, real-time search and analytics engine. Waartaa-ElasticSearch integration is the first sub-task which begins the implementation of browse/search channel chat logs. After the GSoC bonding period was over I started working on it. Here is what I did in last week.

Creating mapping for Channel Logs index

Last week I spent most of my time reading ElasticSearch documentation and tutorials. I have got good understanding of what ElasticSearch is capable of. Mapping is telling ElasticSearch which field in document is searchable, defining their type and few more things. Here is the first version of mapping that I created. It will change over time as new fields may get added in document and we may also change the analyzer used for ‘message’ field to account for misspelled search terms like Google does it.

Indexing Channel Logs

If you have read my last blog post, you will know I already completed this task during GSoC community bonding period. In last week I just refactored it a bit, made it configurable in settings file and tested the code on my system. My mentor told me he is facing some issue with ElasticSearch secure installation. He will merge my PR as soon as installation is done.

Oh wait I forgot to tell you how I will be transferring older chat logs from MongoDB to ElasticSearch. One way is to write a script but there is always a better way to do such things. There is an open source river plugin available which does exactly what I want.

Generating permalinks for incoming chat messages

Each chat log/message will have it’s own unique permalink that users can send to each other to refer to some old conversation. There are many other use cases of it which I will not be discussing here now. There wasn’t much to do in this task because these are generated automatically in form of  ‘_id’ field of document that is indexed in ElasticSearch.

Conclusion

In terms of code, basic Waartaa-ElasticSearch integration is complete. I hope the problem with ElasticSearch installation gets resolved asap so that I can see my code in action in production🙂

What’s next?

Next, I will be working on building an API to search/browse channel logs. Hopefully I will complete it by end of this week and will write a blog post about it too🙂

<script>JS I love you</script>

Hacking on Waartaa – GSoC Community Bonding Period

Bit of a late, but yes, I have been selected in Google Summer of Code(GSoC) with The Fedora Project to hack on Waartaa. Ratnadeep Debnath(rtnpro) and Sayan Chowdhury are my mentors.

Waartaa is an open source, modern IRC client for the web. It supports centralized logging, 24×7 idling, notifications and unique identity to a user on IRC across multiple clients/devices, and also a rich UI for awesome user experience. During GSoC tenure, I plan to implement:

  • A central hub for searching/reading channel logs.
  • Bookmarking channel logs.
  • Video/audio conferencing facility.
  • Admin console panel in Waartaa.

Check out my detailed GSoC proposal for more info.

It has been a pretty exciting last three weeks. I simply love Javascript and the thought that Waartaa is a Meteor based project excites me to spend even more time on this project. Below are the things I have done/coded so far.

Setting up Waartaa

Before I was selected for GSoC, I had installed Waartaa on Ubuntu. But later I was told to set up Waartaa on Fedora for future development work. And so I did. Unfortunately, it took more time than I anticipated because of some partition issues in my laptop. I am glad that it has been set up properly now.

Indexing Channel Logs in Elastic Search

To implement a central hub for searching/reading channel logs, it is obviously necessary to store logs somewhere first. Waartaa’s current implementation saves channel logs in MongoDB but it is highly unoptimized for full text search. My mentors decided to use Elastic Search considering its full text search capability. Here is the basic code I have written to save channel logs in elastic search: https://github.com/waartaa/waartaa/pull/89/files

Fixing few minor issues

I fixed few minor issues as well. Both of them were client side related issues. Below are the PRs I sent:

Unit testing

I was told to get familiar with the code base by unit testing it. There were no previous unit tests in Waartaa’s code base so I had to start everything from scratch. There aren’t many unit testing frameworks available for Meteor. RTD test runner was our best option. RTD uses Jasmine as it’s default unit testing framework and it has a good documentation available. So writing unit tests was not as hard as I thought it would be. Till now I have written tests for client side template methods and events only. Here is the PR which I sent recently: https://github.com/waartaa/waartaa/pull/102

Please subscribe to get notifications of future posts on my GSoC work. Till then happy summer and happy hacking!🙂

<script>JS I love you</script>