QA Testing Video Chat App – GSoC Week 4

I would recommend you to read my last post first before reading this one. This should help you understand the content of this post better 🙂

Last week, I did both dev testing and load testing of video chat app. In this post, I am gonna show you the results I got.

Bugs & Fixes

Bugs I found during testing:-

  • Re-rendering video object: When a user moves from video chat app to an irc channel/server and then back to video chat again, video stream(both local and remote) doesn’t get re-render. Actually,  this was a browser specific issue and I found it’s fix by chance. Have a look at the fix.
  • Click event on ‘Accept/Reject’ button gets triggered more than once: This one was simple to fix. I just had to use jQuery off method to remove click event attached to element before attaching new click event. Fixed.
  • Option to enable/disable video chat: I know this isn’t a bug. My mentor told me to add this feature.

Load Testing

Framework

In my last post, I mentioned I used EasyRTC framework to build p2p(peer to peer) video chat. Although, it’s a p2p app, it still requires server for signaling(read more). For signaling, EasyRTC uses socket.io built on node.js. So, basically I load tested EasyRTC server’s socket.io implementation.

I used socket.io-client module to interact/connect with EasyRTC server by writing against their server side API, as documented here. Below is the snippet of simple client I created.

var Client = function () {
  /* Some code omitted */
  this.connectToServer = function (callback) {
    var client = io.connect(
        'http://'+SERVER_HOST+':'+SERVER_PORT ,
        {'force new connection': true}
    );
    if (!client) {
      throw "Couldn't connect to socket server";
    }

    var msg = {
      msgType: 'authenticate',
      msgData: {
        apiVersion: "1.0.11",
        applicationName: "waartaa"
      }
    };

    var easyrtcAuthCB = function (msg) {
      if (msg.msgType == 'error') {
        callback('easyrtc authentication falied');
      } else {
        // code omitted
      }
    };

    client.json.emit('easyrtcAuth', msg, easyrtcAuthCB);
  };
}

Script & Parameters

I made a command line tool in nodejs to ran tests.

node run.js --no-of-connection=700 --no-of-concurrent-connections=[1,5,10]

In above mentioned command you should notice two things, namely, no. of connections and concurrent connection. My system(Intel(R) Core(TM) i5 CPU M 460 @2.53GHz, 4 GB RAM) couldn’t stand more than 700 connections. Server process started consuming 1 GB RAM for more than 700 connections. I will tell you the reason for this later. For now, 700 was suffice for initial testing. My mentor told me to keep concurrent connections very low because we won’t be expecting too high concurrency in production initially.

Data collected

After the script was ran several times, I collected the following data each time for different concurrency against no. of connections:-

  • Roundtrip time(ms): Time taken for client to get connected to server. This includes latency(which is negligible because client and server are on same machine) + client-to-server message time + server-to-client message time.
  • Avg. CPU load(%) & Memory (MB): I used this node module to collect these data.

Below you can see the data plotted to graph and their analysis:-

Roundtrip time analysis roundtrip_time

  • As no. of connections increases, round-trip time increases. This is because the default behavior of EasyRTC is to send every client in a room a separate update whenever a new client leaves a room, enters a room, or changes its status. And this leads to increase in round-trip time.
  • Increase in concurrency too increases round-trip time. I guess this is because nodejs being a single threaded server processes one request at a time. Please correct me if I am wrong.
  • Roundtrip time < 500 ms is a good no. For concurrency = 1 and any no. of connections in above graph, round-trip time < 500 ms.

CPU analysis cpu

  • Nothing too unnatural about cpu load increasing with no. of connections and concurrency.
  • Good thing is the rise is not too steep for lower concurrency.

Memory analysis memory

  • As I mentioned earlier, 700 connections consumed 1 GB memory and you can clearly see that in above graph. Well, this is because as I said earlier, the default behavior of EasyRTC is to send every client in a room a separate update whenever a new client leaves a room, enters a room, or changes its status. That means for 700 connections (700*701)/2 = 245350 messages were sent by server. You can do the math now 🙂
  • For any concurrency, memory rises at the same rate with increase in no. of connections. Thus, memory consumption is independent of concurrency.

Conclusion

  • More bugs will pop up when my mentors and other people start using the app until then a developer can only believe his/her app works perfectly 😉
  • The default behavior of EasyRTC server can cause scaling issues. We might have to change it in future by adding custom event listeners on server side.

PS: If you have made this far and found anything wrong in this post, please do comment. 🙂

<script>JS I love you</script>