Node.js and Asynchronicity dictature

I’ve made recently some experiment with Node.js, the new kid on the block. Coming from the Ruby and Event Machine world, the evented approach is not new, but some aspect of JavaScript make the approach quite fun.

First examples are fine, and you enjoy it. But as soon as you try to do more complex things, you are facing the pyramid of hell: the callback nightmare.

So what’s wrong with it?

The selling point of Node.js is the evented approach. The problem is that too many event kill the evented approach. Everything want to be an event, and most of the code you are writing is then to achieve synchronicity .

Let’s take a first example, with a small “pyramid of doom”: open a file, write a line, and close it (I know that there is specific function to do it once, but the objective here is to show the issue with “all async”)

var fs = require('fs');

fs.open('toto', 'a', 666, function( e, id ) {
  fs.write( id, "Test", function(){
    fs.close(id, function(){
      console.log('file closed');
    });
  });
});

The Node.js community is aware of the issue, and says tha promise will change all, and the promise of promise, is to make async things happenning sequentially.
Basically, instead of having a pyramid, you chain events.
So same exemple , using ‘Q’, a promise library:

var fs = require('fs');
var Q=require('q');

Q.nfcall(fs.open,'toto', 'a', 666)
.then(function(data){
	fs.write( data, "Test", null, 'utf8');
})
.then(function(data){
	fs.close(id);
});

Ok, no more pyramid of doom, but not totally sure that’s is much better than before. A lot of code just to ensure sequentiality.

Async vs Sync

Let’s compare to the “classical” approach

id=fs.open('toto','a',666);
id.write("Test",null,'utf8');
id.close();

(this won’t work on node.js currently!)

Or even better in a chainable approach:

fs.open('toto','a',666).write("TEST").close();

The code is 10x time easier to read than the previous one.

Yes, but evented io is faster, it’s the future!

Evented io is great, no doubt. The point is: do we really need to raise this to the level of the developer? 95% of the tasks are sequential, even those who require IO. So instead of exposing this to the developer, the language/framework should be able to hide this to the developer, using the ability to do other things during these events.

This is the idea behind fibers, or more generally behind cooperative multitasking. This won’t make your program slower but will just hide some of the complexity for you. You still can have concurrency, joins, etc…

Event is good when :

  • You really have ‘unexpected event’ or this part of your application is push based, like somebody pressing a button on a UI, a web request, a Tweet coming in a stream, etc…

These are really event.

When event is not really needed :

  • When you want to read/write file, access to the database, etc… This does not means that you must be blocked, this just mean that in that case programmer wants to be sequential

Golang and others choose this path, and this make the code much simpler to read.

I predict than in a couple of year, all the node.js community will suddenly discover that “sequenciality is not so bad” and will introduce fiber or ways to make synch.

arduinopi

Raspberry or Arduino: how to choose?

arduinopi
When you look at a platform to play with connected objects, these are the two stars. But which one to choose? Especially when you look at the spec, there is one much more powerful than the other, so why we should bother of the Arduino ?

So we will try to understand which one is suited for what kind of project, in order to help you to do the right choice.

The raw specs are the following:

Arduino Raspberry
RAM 2 kb 515 Mb
Program size 32 kb Up to 32 G, depending of SD card
IO output 14 Digitals, (6PWM), 6 analog 8 digital 17 GPIO
CPU 16 Mhz 700Mhz
Price 25 euros 30 euros
Connectivity None by default, Ethernet/Wifi using shield or more powerful Arduino cards Ethernet, Wifi using an USB key

So definitvely raw spec of Raspberry are better than Arduino’s. But CPU and memory in IOT are not the only criteria to choose.
In fact, it depends totally from your type of project. If you want something close to the hardware, simple, cheap (a single arduino chip is around 3 euros).
To help you, here is a more detailed tab of various area of interest of each:

So, if your project is ….

Arduino Raspberry
Close to the hardware +++ +
Require analog input/output +++ -
Require CPU power - +++
Require Internet connectivity + +++
Require Screen output + +++
Needs to use USB devices - +++
Needs to store a lot of data + +++
Need to work on battery +++ +
Need to work alone +++ ++
Need to boot quickly +++ +
Need cheap device +++ ++
Require high level language
(python, javascript, ruby,…)
- +++

Raspberry is not expensive, but much more than an Arduino when it comes to industrialization. You can have an Arduino up an running for less than 3 or 4 euros, and you can freely build your own layout.

So for exemple :

Good Arduino usages
- autonomous gps tracking device
- small connected devices
- house sensor (gaz, electricity, intrusion, etc…)

Good Raspberry projects :
- home central automation
- media center
- ip camera recorder

Project that could be both :
- autonomous robot
- “complex” connected object , .i.e. nest like

Some project could need both. A good example is home automation, you can do simple capture device using arduino, and collect data in a master system based on Raspberry

Both are great device to play with, so I strongly encourage you to test both and to do your choice, depending on your project.

Who needs an operating system for the Internet Of Things?

Recently, there was a rise of so called « operating systems for the internet of things ». To name a few, there is Arm, Contikit, Riot. Even Google and Microsoft wants to be in the race. This is probably the worst thing that could happen to the IOT.

Internet of things is about simple interactions, but also about communication.

As a user of IOT, you don’t care about the OS of your ” things ” but should you care as a developer ?
Connected devices are for most of them simple devices, that does one thing that does it well. Price is a key factor, and Arduino as the major prototype board for the IOT shows us the path. Simple, no OS, just a few functions to interact with the board and the external world, and no real OS.

The biggest issue is about communication and that’s the today weakness of IOT. To be able to do communication you need something much more powerful that was is needed to interact with the user. In Arduino for instance, you embedded a full Linux based system just to be able to do Wifi with the rest of the world (Arduino YUM).
Of course, some connected devices will require some complex and powerful OS, typically those with strong interaction with users but that’s only a fraction of the IOT
Others, like Spark.IO or Electric Imp try to solve the system by integrating at the chipset level the communication stack.

So we don’t need another OS, we need communication stack and protocol, to be able to discover, interact, commands objects. This is the « real » OS for the IOT, and this have yet to come.
The basic list of features should be:

  • simple
  • secure
  • light
  • provide a discovery and registration protocol
  • ability to send and receive information
  • pull AND push! (push is really key here)

There are some candidate for this, but no real winner so far, unless I’ve missed something!

Power monitoring using Arduino

Another small but interesting project is power monitoring usage using Arduino.

The idea was to be able to get instant feedback on my home power consumption.

So I bought some CT sensor from EBay (9 euros/each) . The only problem is that I have a three phased installation. So instead of two phases as usual, (three wires) there are three phases and four wires.

The installation was made in two steps: first a breadbord connected to the CT sensor and the arduino, sending the information every ten seconds on OpenSensorCloud using an Eternet Shield.

EMon
As this worked fine, I’ve bought an a shield instead of my breadboard. This cost little bit but the result is much better than a custom one.

The advantage of having a three phase installation is that you can display each phase individually, so it gives you much more indication on what is currently hapenning. For instance, I no that on Phase 3 there is the heat pump installation, on phase 2 the kitchen, etc…

The result is visible here, you have a dashboard showing the instant consumption on each phase, as well as the daily total :
Emon

The sketch is here:
ArduinoCode
As a bonusI wanted something to display these data in the living room, so I used the spark core that I bought as an #IoT geek: the core is connected to the server and receive the data every ten seconds. This is probably not the optimum configuration (Ethernet -> Server -> Wifi -> Spark core) and I plan to do an updated version if a different connectivity. In the meantime, the result is quite nice: the core display the total consumption and the phase consumtion ona small olde display, as well as the usage graph of the last ten minutes.

The RGB led of the core is also used to provides feedback on power usage: green -> low usage, red, high usage

Bo_RURHIUAEGdMI_002

First SigFox program with Arduino (Akeru)

Following my previous post, I’ve started to explore Sigfox capacities with a small first project. The idea was to make a low powered GPS tracker. So I’ve just added a 10€ GPS adapter from China and wrote my first sketch.
The idea is to make a simple GPS Traker, ideally not using SMS but being able to send his position from anywhere, with of course very low energy consumption. This first version is just a proof of concept so we are not yet focused on the consumption. The data are then send to the OpenSensorCloud platform (feel free to register! )

photo(2)

The programe is quite simple:
1. initialize everything
2. wait for the GPS to get fix position
3. send it to server through Sigfox network
4. wait 10 min
5. start at 2)

We need to wait 10 minutes before sending another 22 bytes messages, it’s a Sigfox limitation and the tradeoff between low consumption, high coverage, and size. I don’t see this as a big issue for many usage, but for sure, it will add some interesting challenges

In short:

// include the library code:
#include <Akeru.h>
#include <TinyGPS.h>
#include <SoftwareSerial.h>

// initialize the library with the numbers of the interface pins
TinyGPS gps;
SoftwareSerial ss(8, 9);  // GPS RX is set on pin 8

void setup() {

  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }
  Akeru.begin();  // Not sure it's neded, as I need to call it again later
  delay(1000);
  ss.begin(9600);
  // Init modem
  delay(1000);
}

// int is 16 bits, float is 32 bits. All little endian
typedef struct {
  float lat;
  float lon;
  float alt;
} Payload;


void loop()
{
  Payload p;
  float flat, flon;

  gps.f_get_position(&flat, &flon, &age);
  p.lat=flat;
  p.lon=flon;
  p.alt=gps.f_altitude();
  
  if(flat==TinyGPS::GPS_INVALID_F_ANGLE){
    delay(1000);
  }else{
    // Note Seems that we need to call Akeru.begin() again or
    // further call to isReady or send will be blocked...
    delay(1000);
    Akeru.begin();
    delay(1000);
    Serial.println("is ready:"+String(Akeru.isReady()));
    Serial.println("Sending");
    bool status=Akeru.send(&p,sizeof(p));
    Serial.println("done result:"+String(status));
    // Wait for 10 minutes before sending another one
    for(int i=0;i<600;i++){
       delay(1000);
    }
    Serial.println("wake up");
  }
}

Of course, the program need some improvements, like waiting to have more satellite fixed before seding the first message, but let’s keep it simple.

The full source code is available

I had some minor issue, it seems that the Akeru API needs to be “reseted” before sending new data

The payload is quite simple, just a structure with three float: lat , lon, and alt (the last one might not be the most interesting but it was just a test).

The most interesting things is that now opensensorcloud support Sigfox devices. Sigfox allows you to send data in a limited way. You are allowed only 22 bytes per message, so no way to do ascii or json. So you usually send binary information, typically raw C or C++ structure. Then you have to decode them on the server.
Hopefully, OpenSensorCloud provides full support for such feature:
When creating a sensor for an Akeru enabled device, you just need to specify the format of the payload, this will be automatically converted to corresponding sensor values.
SigFoxConf

Make sure that your sigfox device id is well defined
Format are described in a generic json way, for instance, in may case, the payload contained 3 float field, so format is the following:

[
  {'name':"lat",'type':'float'},
  {'name':"lon",'type':'float'},
  {'name':"alt",'type':'float'}
]

As I wanted to map this to a “Position” object, and not to thre sensors, I ad to add a little bit of definition


[{'name':'pos',
  'type':'position',
  'struct':[
   {'name':"lat",'type':'float'},
   {'name':"lon",'type':'float'},
   {'name':"alt",'type':'float'}
  ]
 }
]

As a Sigfox callback, put the URL of the OpenSensorCloud callback, like this

http://api.opensensorcloud.com/device/sigfox?api_key=YOUR_API_KEY


Replace with the value of the platform.
OpenSensorCloud expect two extra parameters: device value, with the device_id you’ve just entered before, and data with the binary payload.

The API_KEY is visible on your profile,
And let’s go

The map location appears on the platform

You can see the result on the
Real Time Dashboard live
capture

In next part, we will add geofencing and twitter features thanks to the Trigger functions of the OpenSensorCloud platform…

Akeru board completed

Akeru unboxing!

SigFox logo
Juste received my Akeru board from Snootlab today, and here are my first reactions. Akeru is a Arduino with a SigFox chip inside, allowing to send (small) messages – shorter than sms – but with very low poswer consumption.

Here is some screen shot of the kit: looks nice, close to a Uno, but need a little bit of soldering

photo 3

photo 2

A few minutes later, it’s done, the board is up and ready to be plugged.

Trying to follow the instruction of Snootlab, I’ve downloaded the FDTI drivers, and connect the board, and….nothing happens….no led blinking, nothing

So I started to investigate: in fact, it appears that the driver for mac seems to have a lot of problems. I need to plug/unplug several time to start the usb driver. But even, the board was not visible.

So I used a PC to do the same manipulation. Good news, the board was recognized, and wake up, and I was able to program it but the driver was very unstable, I had to plug/unplug several time to program the board.

But finally, I was able to do it, and to send my first message to the Sigfox network and receive it to . I still need to use the Snooplab proprietary gateway (actoboard) which is quite limited: it can not pas directly the raw SigFox message as raw data, you need to specify one and only one format, which is not very practical.
Also, there is no statistic about your usage. You should not send more than 140 messages a day but there seems no indication of your daily usage. Beyond the 140 messages, your are limited but also you have to paye some “proportinal penalties”, which seems to be contrary to the limitation.

But back to the Mac , impossible to make it works: the driver have problems to load, and even when the loaded, the card was not recognized. So the bad news is that I can not program it using my everyday computer (MacBook air, OS X 10.9.2) and I have to go back to the PC

So I’ll start to play a little bit with it, and keep you updated with the results!

Playing with Arduino: real HashtagBattle Twitter display

I just started a few days ago to play with Arduino.

First, I must say it’s a great platform: setup is easy, samples are great. I’ve made some embeeded development by the past, and I would dreamed of such platform!

So here is my first small project: an “Hashtagbattle display”

Real World hashtagBattle
The idea is to show the result of an Hashtagbattle on Twitter using a something more physical, in that case an arrow indicating the tendance of the results.
So we are couting the number of tweets containing hashtag a (let’s say “iphone”) and compare it to the numbers of tweet containing hashtag b (let’s say “#android”) and the direction of the arrow depend of the result. If we have the same amount of hashtag b and hashtag a, the arrow will indicate the center.

The small difficulty here was to use the streaming API of Twitter to do this real time

The board:

- the board is quite simple: there is a servo that will display the arrow direction, and two leds, a red and green, each one flashing when there is on tweet on HashTag A (red) or HashTag B are coming

sketch
The software:

As I currently have only an Arduino UNO, there is no direct connection with the internet. But I use the PC for this task, and communicate with the board through the serial line.
So the PC sends every 100 ms a line with 3 informations:
- the angle of the servo
- 0 or 1 to indicate if first leds need to be enabled
- 0 or 1 to indicate if second leds needs to be enabled

For instance:

32 1 0

Will tell that the servo need to be of 32° and red led needs to blink

So the program is the following:

require 'tweetstream'
require "serialport"

input=ARGV.shift || "#bordeaux,#strasbourg"
#params for serial port
port_str = "/dev/tty.usbmodem1411"  #may be different for you

sp = SerialPort.new(port_str, 9600, 8, 1, SerialPort::NONE)

TweetStream.configure do |config|
  config.consumer_key       = '<YOUR KEY>'
  config.consumer_secret    = '<YOUR CONSUMER KEY>'
  config.oauth_token        = '<OAUTH TOKEN>'
  config.oauth_token_secret = '<TOKEN SECRET>'
  config.auth_method        = :oauth
end

@client=TweetStream::Client.new;

total=0
last=Time.now
words=input.split(',')
flags=Array.new(words.size)
buffers=words.map{|w| []}
sp.puts "512 0 0" # reset the servo
@client.track(words) do |tweet|
  begin
    search=tweet.text.downcase
    words.each_with_index do |word,i|
      if search.include? word
        flags[i]=1
        buffers[i]<<tweet
      end
    end
    if (Time.now-last)>0.11
      begin
        ratio=(180*buffers[0].size/(buffers[0].size+buffers[1].size)).to_i
        str=ratio.to_s+" "+flags.join(' ')
        puts str
        sp.puts str
        last=Time.now
        flags=Array.new(words.size,0)
      end
    end
  end
end

Note that you canstart with parameters:

ruby notifyTweets.rb “#apple,#orange”
or use words instead of hashtags:
ruby notifTweets.rb “apple,orange”
The program needs to be run on the PC where the arduino is connected!

The Arduino part is here
We just se the serial line, wait for a line, and extract the informations from it.

#include <Servo.h> 
int ledPin=13;    // select the input pin for the potentiometer
int nbLed=2;
Servo myservo;  // create servo object to control a servo 

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }
  for(int i=0;i<nbLed;i++){
   pinMode(ledPin-i, OUTPUT); 
  }
 myservo.attach(9);  // attaches the servo on pin 9 to the servo object 
}

# Utility function to get a value from a string at a given pos
String getValue(String data, char separator, int index)
{
  int found = 0;
  int strIndex[] = {0, -1};
  int maxIndex = data.length()-1;

  for(int i=0; i<=maxIndex && found<=index; i++){
    if(data.charAt(i)==separator || i==maxIndex){
        found++;
        strIndex[0] = strIndex[1]+1;
        strIndex[1] = (i == maxIndex) ? i+1 : i;
    }
  }
  return found>index ? data.substring(strIndex[0], strIndex[1]) : "";
}


void loop() {
  if(Serial.available() >0) {
    String str=Serial.readStringUntil('\n');
    Serial.println("Read:"+str);
    for(int i=0;i<nbLed;i++){
      // look for the next valid integer in the incoming serial stream:
      if(!getValue(str,' ',i+1).equals("0")){
        digitalWrite(ledPin-i, HIGH);  
      }
    myservo.write(getValue(str,' ',0).toInt());
    }
  }
  delay(100);  
  for(int i=0;i<nbLed;i++){
    digitalWrite(ledPin-i, LOW); 
  } 
}

That’s all for this first project, was really fast to develop, thanks to Arduino (and Ruby!)

SpaceSaving algorithm (heavyhitter): compute real time data in streams

I’ve just discovered recently the SpaceSaving algorithm ( ) which is very interesting.

The purpose of this algorithm is to approximate computation of data from an infinite stream using a data structure with a finite size. The typical problem we try to solve is to count the number of occurrences of items coming from an infinite stream.

The obvious approach, is to have a map of items,count somewhere (memory, database) and to increment the count (or create it if not present) in the map

Depending of the distribution, the map size can be as big as the number of items in the feed (so infinite!)

Thanks to the heavy hitter algorithm, you just maintain a small subset of items, and an error associate with them.

If an item is not yet present in the map, you remove the previous minus and replace it with this one, and set the error count to the new one to the total count of the previous one.

So basically, you can have an item appears in the map but with an high error count.

The result depends of the number chosen for the size of the map , and of the distribution of the data, but I wanted to try by myself and Ive done a small implementation in ruby and a sample using the Twitter real time stream to cunt URL’s occurrence in the feed

class SpaceSaving
  attr_accessor :counts,:errors
  def initialize in_max_entries=1_000
    @in_max_entries=in_max_entries
    @counts={}
    @errors={}
  end
  #Add a value in the array
  def add_entry value,inc_count=1
     count=@counts[value]
     if count==nil   #newentry
      if @counts.size>=@in_max_entries
        min=counts.values.min
        old=counts.key min
        @counts.delete old
        @errors.delete old
        @errors[value]=min
        count=min
      else
        count=0
      end
     end
     @counts[value]=count+inc_count
  end
  
end

I just wanted to keep it as simple as possible for now, but in a real world example, it would be good to encapsulate the way to access to the @counts field.
By default, the size of the map is set to 1.000, but this can be modified….

And the sample.rb

require 'bundler/setup'
require 'tweetstream'
require './SpaceSaving.rb'

TweetStream.configure do |config|
  config.consumer_key       = <CONSUMER_KEY>
  config.consumer_secret    = <CONSUMER_SECRET>
  config.oauth_token        = <OAUTH_TOKEN>
  config.oauth_token_secret = <OAUTH_SECRET>
  config.auth_method        = :oauth
end

@client=TweetStream::Client.new;

urls=SpaceSaving.new

total=0

@client.track('http') do |status|
  begin
    # add each URL in the tweet , and count it once
    status.urls.each do |url|
     urls.add_entry(url.expanded_url,1)
    end
    total+=1
    if total%1000==0 then
      puts "Treated:#{total}"
      limit=urls.counts.values.sort[-30]
      urls.counts.each do|u,c|
        if c>limit 
          puts "Counts #{c} #{u} errors:#{urls.errors[u]}"
        end
      end
    end
  end
end

The complete sourcecode is available on Github: https://github.com/tomsoft1/SpaceSaving

So I’ve made a few tests and comparaison with an exact approach, and the result for the top 20 was similar, even after 600.000 tweets parsed, with a “small” 1.000 size comparing to around 300.000 entries for the exact algo.

Facebook vs Twitter on Social TV, where are real numbers?

Our friends from Trendrr recently disclosed some interesting numbers about Facebook usage vs Twitter usage around the NBA playoff kickoff. According to them, there is 5x more Facebook usage than Twitter usage.
Interesting because nobody for now have these data to compare, so it’s hard to get a real view, and secondly because nobody that I know hardly publish things on Facebook about TV (while many of them use Twitter for this) so may be I am not in the right category.
However, Trendrr published some stat on the NBA:


(click to see full size image)

If you take a look at the number, you see:

Total on air activity: 3.190.816
Facebook on air activity: 2.377.251

So total is probably facebook+other networks, like Twitter (so 813 565 tweets or maybe less, that’s where come the 5x more than Twitter)

But take a look at the other numbers:

FB on air posts + comments +shares=924.266. So, what are the others? Like? Yes, majority of FB usage seems to be like even if there is a decent amount of comments, but more in the range of Twitter

The last interesting data, is FB on air uniques: 210.760 . Waow, this is not bad in SocialTV metric, but typically in the Twitter range. I would be curious to know how many unique users have tweeted on this same event, but I guess it’s probably more than 210.760 unique users.

Typically, there is around 2 to 3 tweet in average per event, so it should be around 300.000 unique twitter users….

Even, these are much bigger numbers thanthe 95% twitter vs 5% facebook that are usually used in the industry, but below the 1 vs 5 for Facebook.
Facebook can easily bypass these kind of issue by providing better metrics to companies like us ( TrendsMotion) without compromising security and privacy of conversation! Let’s start to work together…

TvTweet: realtime analytics around SocialTv and Twitter

TvTweet.

Thanks to TvTweet, you can have real time information about all tvshows and their twitter audience, for France and UK.
Beyonf real time information, you have anlytics and historical data about these Tv show.

You have a cool real time dashboard:

Dashboard

or some more detailled information about a show:

For more informations, visit us, there are plenty of features on the way!

Fast Gaussian Blur alogrithm (on iPhone)

I had to implement a Gaussian Blur for an ihpone project, and the direct approach was really slow. For each point, you compute a rxrx4 pixels for each point (r is that rayon of the blur).

I’ve tried several other approach, but I found the stackblur alorgirthm. The idea of the stackblur is instead of recomputed all the neighbouround points for each new points, you juste maintain a stack of value, and remove/add value on the fly.

The source code is availabe on git hub with a sample project:
https://github.com/tomsoft1/StackBluriOS

The source code is adpated from the Mario Kingemann algorithm, check his web site, full of good stuff

Diawi : Easy upload of iPhone App


One feature that I’ve totally missed on iPhone4, was the ability to download an .ipa other the air. You can now easily download an app if you specify a link. The requirement, is that your application need to have a valid certificate for this device.

So it’s not possible to download your app on any device, as it will require Apple signature, but it can be done for AdHoc release for instance (developer releases). But the management of the certificate, device id, and so on is usually a pain.

There is a simple service that do it very well: diawi . Just upload your zip or ipa, and then you’ll get back a link that you can forward to all your tester, with an optional password

Note: this works only on iOS4….

Gret service

Twitter Site Stream using EventMachine

I’ve spend some time to try to use Twitter Site Streaming API with event machine, so here is just a small snipet of code on how to do it. In fact it’s pretty simple

Just go to your app page on twitter ( http://www.twitter.com/apps ) and go to your application. Get the access token here. You will find also a button “get my access token” where you can get the oauth access token and access token secret.

With these, you will be able to sign the request

require 'rubygems'
require 'eventmachine'
require 'em-http'
require 'json'
require 'oauth'
require 'oauth/client/em_http'

# Edit in your details.
CONSUMER_KEY = "<put your consumer secret key here>"
CONSUMER_SECRET = "<put your consumer key here>"
ACCESS_TOKEN = "<put your access token here>"
ACCESS_TOKEN_SECRET = "<put your access token secrethere>"

def twitter_oauth_consumer
  @twitter_oauth_consumer ||= OAuth::Consumer.new(CONSUMER_KEY, CONSUMER_SECRET, :site => "http://twitter.com")
end

def twitter_oauth_access_token
  @twitter_oauth_access_token ||= OAuth::AccessToken.new(twitter_oauth_consumer, ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
end

EventMachine.run do
		  # now, let's subscribe to twitter site stream
		  # check informaiton on twitter site
		  # here we are followig to user that have signed to our app...
		  toFollow=[17590452,2071231]
		 http = EventMachine::HttpRequest.new('https://betastream.twitter.com/2b/site.json'
).post(:body=>{"follow"=>toFollow.join(",")},
		 	:head => {"Content-Type" => "application/x-www-form-urlencoded"},
		 	:timeout => -1) do |client|
    		twitter_oauth_consumer.sign!(client, twitter_oauth_access_token)
  		end

	  	buffer = ""

		http.stream do |chunk|
    		buffer += chunk
   			while line = buffer.slice!(/.+\r?\n/)
   				puts "handling a new event:"+line
    		end
  		end
   		http.errback { puts "oops" }
   		http.disconnect { puts "oops, dropped connection?" }

 end

For this, I use the eventmachine plugins for ruby, as well as the oauth and em-http plugins.

Please note that you must you https with twitter in order for this to work.

TvTweet: Real time Twitt about Tv Shows

I’ll present you a small side project, named “TvTweet“. Basically, the idea of TvTweet is to be able to check real time what are the discussion in twitter about TvChannel. And even better, to help you if you see something interesting and hot happening on a channel.

You can also react on Tv Shows, by sending Tweets.

Here are some screen shots:




Unfortunately, the App is only available in French and for French channels, but anybody interested by a converstion for US or UK channels (or other countries) is welcome.

There is a nice server part, “TvTweet.fr” where you can see the tweets, and also the statistics. The application is mostly client side, but the initial idea of the server side was to get statistics, like this one:

It was done quite rapidly and this was fun to create. Was build of course in Coca on the iPhone side, and using several technologies on the server side:
* Ruby On Rails for the server front end
* Node.js for a server side dispatcher, allowing you to follow real time twitter stream without being authentified.
* php for the server side stream tracker

So what I’ve learned:

* Node.js is definitvely too young, it was hard to make something work as expected , but the principle was nice. But thanks to node.js, I’ve take a deeper look to EventMachine for Ruby, and it’s probable that this part will be rewrite using rails

So last reminder: Download TvTweet for iPhone now!

AlertMachine 1.5 is out: real time push notification for everyone

AlertMachine icon

We just release “AlertMachine” on iPhone, the latest iteration of our product. It was previously named “iFlow”, but “AlertMachine” is more self explenatory.
Basically, with Alert Machine, you can be notified in real time of any “events” hapenning on the internet and easily manage these event and notifications. You can be notified of:

  • New RSS article (for instance, each time that techrunch is posting a new article)
  • Twitter notification or direct message
  • Facebook event
  • etc….

There is no limitation on the number of feed/event that you can follow.

An API is available if you want to create your own Alerts for this product, more detail soon.

You can download AlertMachine from the AppStor here.

iWebwag for iPhone is out ! RSS reader, tweet, widget and more….

We’ve been quite silent the past monthes, because one of the big project was the launch of our iPhone app, iWebwag! iWebwag is a full widget engine for iPhone, with a very powerfull RSS reader and sharing capacities trough email or Tweeter. Go to our app page description to have a nice overview of the features, but here are a few screen shot of this App. One of the cool feature is the “exposé mode”, where you see all your pages in a single view:

This is a page view. Note that each widget can have different size, choosed by the user:

And now the great news reader:



Twitter Hacked!

This is a translation of an article published in French, on the Korben.info website

twitter fail whale Hack de Twitter   La suite...

I’ve been contacted yesterday by the guy who have hacked Twitter. His pseudo his Hacker Croll (here is the initial reference to Hacker Croll, but in French) and explained to me that he was able to access to the various email boxes of the twitter employee including Evan Williams ones and his wife. This allowed him to have access to all a number of astonishing informations.

He had access to the Paypal, Amazon, Apple , AT&T, MobileMe and Gmail accounts of Evan Williams, Sara Morishige Williams, Margaret Utgoff and Kevin Thau (twitter employees)

Here are the snapshots that the hacker sent to me:

Evan Williams1 Hack de Twitter   La suite...

He was able to access to the Registar information of the Twitter domain name, and he could have been able to redirect twitter domain name to any other IP address (or simply steal the domain name)

Capture 1201 Hack de Twitter   La suite...

Capture 115 Hack de Twitter   La suite...

But the most incredible, was the quantity of internal information that he was able to get on Twitter:

  • the complete list of employees
  • their food preferences
  • their credit card numbers
  • some confidential contracts with Nokia, Samsung, Dell, AOL, Microsoft and others
  • direct emails with web and showbizz personalities
  • phone numbers
  • meeting reports (very informatives)
  • internal document templates
  • time sheet
  • applicant resumes
  • salary grid (time for me to move..lol)

But amongst all these information, you can see some funny things like:

  • the “possible” launch of a TV reality show where contestant will go across USA et will win contests thanks to their followers, with a 100 000$ price at the end (but for a nonprofit organization)
  • Some growing predictions that target 25 millions of users end of 2009, 100 millions ends of 2010, 350 millions ends of 2010…with revenue  that I will not disclose here…
  • A list of new star account like Wyclief Jean, DuranDuran, Cartoon Network, Cisco, UCLA, Guillaume Pepy (CEO of one of the biggest french company, the SNCF), Nirvana, Toshiba, 50 Cents,…. etc…

Capture 1101 Hack de Twitter   La suite...

  • The plan of their new offices with a list of whishes from the employee who would like a sleeping room, a playing room, plants, a chief cuisto, a meditation room, bicycle room, adjustable desks, sport room,washer/dryer, wifi, lockers, wine cellar, an aquarium and others…They seems to have imagination….

Capture 93 Hack de Twitter   La suite...

  • We learn also their idea about Twitter monetization…Of course, we’ve got certified accounts, but also advertising with the ability to put AdSense widget,  or sponsored tweets. Twitter whish also to be the first service to reach the billion of  users (which is highly probable). They defined themself more as a “nervous system” than an alert system.
  • We also learn that french president will soon use Twitter (@NicolasSarkozy ) and that Nicolas Princen which will do this.
  • And we’ve got also some “test” of t-shirt and cap designs

Capture 10621 Hack de Twitter   La suite...

Capture 1032 Hack de Twitter   La suite...

So Twitter has been visited by this hacker. Since then, everything is back to normal thanks to security recommendations:

Capture 1121 Hack de Twitter   La suite...

passwords have been changed. The information given by Hacker Croll is from beginning of may, but are still very instructive. In his mail, Hacker Croll explains the things to learn from this misadventure:

What I would like to say is that even the biggest and the strongest do silly things without realizing it and I hope that my action will help them to realize that nobody is safe on the net. If I did this it’s to educate those people who feel more secure than simple Internet novices.And security starts with simple things like secret questions because many people don’t realise the impact of these question on their life if somebody is able to crack them.

concerning me, I’ve put here only the information that are not against twitter because I am a big fan of Evan and his team works. I’ve just relayed some information of Hacker Croll and what I can tell to Twitter team is that this hacker seems to have a conduct code which will not give any prejudice to the company.

Now, clearly, we see from this hacking demonstration that it’s very easy to guess a simple password from a secrete question, and from this to enter into other account (Facebook, GMail and others) and from this enter in the heart of a company, both in accessing confidential data  but also by paralyzing business symply by getting a few domain names or admin accounts.

So, don’t stop to be paranoid. Don’t use secret question, use a different password for each of your account, don’t put inline sensible documents, etc… In short, be careful..

Know all about your app: Flurry analytics

Flurry, an early developer of mobile mail client for J2me seems to be reborn as a great analytic tools. This tool is available for four platforms: J2ME, iPhone, Android and Blackberry.

I’ve tested the J2me and iPhone version, and it’s one of the best tool I’ve found so far. Before, I’ve used “mobileZoo” from Stephane, and do the job, but started to be slow.

Flurry follow the same principle: you get stats about your users, their mobile, where do they come from, but give you much more details:

  • session length,
  • http usage (number of requests, time spent/request, data downloaded and received)…
  • number of canvas displayed
  • etc…

One of the coolest feature, is the notion of “events”. You can track events within your app (with a maximum of 100 events).
In J2ME, adding events is dead simple: whe you upload your jar file, flurry introspect it and you can select which procedure you want to track. If your binary is obfuscated, you can send the mapping file generated by the obfuscator. And then, by miracle, all calls to a specfific function are send back to the server.

Here is an exemple of two events that I’ve added in the 8Motions app:

Want to know if this feature is used or not in your app? Want to know the user path within it, want to track error generated…All of this is possible quite easily.

What is missing: the good thing for J2ME is it’s not a library, but it’s also a bad thing. I can not regenerate a new binary without manual interaction with Flurry. I would like to see a lib, like the iPhone version, that could be embeeded n my J2me app during the build process. Also, this could be used to add more details to event reporting, like additional parameters (this s what is done in iPhone version).

But it’s a very good product, and I strongly recomment any serious developper to check Flurry !

Webwag widgetize PNDs!

We just announced a few days ago a very exciting things: widgets are coming to your PNDs: Webwag Drives the First Widgets into cars and portable navigation devices.

This is interesting as a stand alone news, but also in the Webwag vision of a converging world where you can access to services around all your connected devices, from your computer, netbook, photo frame, settop box, PND or any connected devices, through a service taking advantage of devices specificites (like GPS for a PND) while taking care of device limitation.
That’s our challenge, and result if exciting for now!
And remember, see you at MWC, Hall2 , Booth 2E59

Reblog this post [with Zemanta]