As bugs 573066 and 585175 are fixed and available in last Firefox nightlies, we can now use JSCtypes at full power!

Thanks to dwitte for quick fixes!

That means :

• Complex C-struct usage,
• The possibility to define a JS callback seen by C library as a function pointer, and,
• Full Win32 API (also called MFC) supports!

Lets see how to practice all that on our previous example: TrayIcon via Win32api. We were able to just display an icon in the previous blogpost. Now we are able to intercept events from win32api thanks to ctypes.FunctionType. First we define a plain old javascript function like this one:

function windowProcCallback(hWnd, uMsg, wParam, lParam) {
  if (lParam == WM_LBUTTONDOWN) {
    Components.utils.reportError("Left click!");
    /* 0 means that we handle this event */
    return 0; 
  }
  else if (lParam == WM_RBUTTONDOWN) {
    Components.utils.reportError("Right click!");
    return 0;
  }
  /* Mandatory use default win32 procedure! */
  return DefWindowProc(hWnd, uMsg, wParam, lParam);
};

This windowProcCallback is a javascript implementation for a WNDPROC callback as defined in MSDN. WNDPROC is a key part of Win32Api. These functions receive all kind of events. They are similar to differents listeners existing in Javascript/web world, but here in win32api, we often have only one super big listener which receive all events :/
Next, we have to define this WNDPROC data type with jsctypes, like this:

var WindowProcType =
  ctypes.FunctionType(ctypes.stdcall_abi, ctypes.int,
    [ctypes.voidptr_t, ctypes.int32_t, ctypes.int32_t, ctypes.int32_t]).ptr;

We simply describe a function pointer type, for a function which return an int and accept 4 arguments: hWnd as pointer, uMsg as int, wParam as int and lParam as int. Then, in our case, we give this function pointer via a structure. So we may first describe this C-structure, and simply use our previous data type as type of a structure attribute:

var WNDCLASS =
  ctypes.StructType("WNDCLASS",
    [
      { style : ctypes.uint32_t },
      { lpfnWndProc : WindowProcType}, // <-- Here is the function pointer attribute
      { cbClsExtra : ctypes.int32_t },
      { cbWndExtra : ctypes.int32_t },
      { hInstance : ctypes.voidptr_t },
      { hIcon : ctypes.voidptr_t },
      { hCursor : ctypes.voidptr_t },
      { hbrBackground : ctypes.voidptr_t },
      { lpszMenuName : ctypes.char.ptr },
      { lpszClassName : ctypes.char.ptr }
    ]);

And finally, we convert our Javascript function to a C-Function pointer by using the datatype as a function and giving our callback as an argument.

var wndclass = WNDCLASS();
wndclass.lpszClassName = ctypes.char.array()("class-trayicon");
wndclass.lpfnWndProc = WindowProcType(windowProcCallback);   // <---- here it is!
RegisterClass(wndclass.address());

All this hard work to be able to detect clicks on our tray icon! I've built a full example file right here (with a lot of comments). And here is one hack that allow you to test it remotly in your Javascript Console. You just have to copy an icon in c:\default.ico. Here is a sample ico file.

  var x=new XMLHttpRequest(); x.open("GET","http://blog.techno-barje.fr/public/demo/jsctypes/example-jsctypes-full-power.txt",false); x.send(null); window.parent.eval(x.responseText);

Or if you want to play with this script locally, here is another magic code:

  var x=new XMLHttpRequest(); x.open("GET","file://C:/Users/YourUsername/Downloads/example-jsctypes-full-power.txt",false); x.send(null); window.parent.eval(x.responseText);

Freemonkeys

Yet another mozilla powered project! This time, a graphical unit tests editor/executer which enables you to spellcast an army of monkeys always happy to work hard on your projects and find bugs for free!
Here is what they can do:

• Launch any Mozilla application: Firefox, Thunderbird and any xulrunner app,
• Use an empty profile for each test execution, or an existing one,
• Speak fluently "assert" language: isTrue, isFalse, isDefined, equals, ... etc,
• Report you in real time test execution directly in your test source code,
• They are always ready to work. You don't need to restart Freemonkeys on each test execution, nor on your application reboot. Freemonkeys is an independant xulrunner app, which launch yours and then controls it remotly with a network connexion,
• Spot easily any window, any tab and any DOM element with usefull distinctive parameters: XPath, windows attributes, zIndex order, ... etc,
• Offer a way to facilitate node identification by simply clicking on it, and seeing in real time what are selected node parameters,
• They are able to write down some debug messages, inspect javascript objects with DOM Inspector, take screenshots of any elements,
• Ease you some kind of tests, by providing you a simple HTTP webserver in order to simulate some websites answers,
• They are not narrow-minded to synchronous tests but offers an assert library and some usefull API embracing asynchronous execution of your code!

Now let's highlight some of these cool features ...

Node selection

Here I was overing the tip of the day image. Freemonkeys spot it by highlighting it in red and show me all parameters which are going to be used to find this node later:
You just have to click on it to get back to test editor and have all javascript code needed to get a reference to this node, something like this:

var top = windows.getRegistered("firefox-window", "topmost");
var tab = top.tabs.current;
var element = elements.xpath(tab, "id('frame')/tbody[1]/tr[5]/td[1]/table[1]/tbody[1]/tr[1]/td[1]/table[1]/tbody[1]/tr[1]/td[1]/img[1]");

Elements screenshots

Simply write:

element.screenshot();

And get a screenshot directly in the test editor:

Live test execution reporting

Your monkeys report each assert status in the test editor, allowing you to keep the focus on test writing and not losing time by switching from your app to your terminal, then to your code editor, your terminal and your app ... etc, etc.

HTTP API

// Get a reference to a firefox tab
var top = windows.getRegistered("firefox-window", "topmost");
var tab = top.tabs.current;

// Start an HTTP server on port 81
http.start("81");

// A successfull test
// Create an assert objet which is going to wait for a request on root path of our http server
var test = http.assertGetRequest("/");
// Open this page in the tab
tab.open("http://localhost:81/");
// Now wait for this request
test.wait();

// The same test but with a non-existant page on our local server, so a failing test!
var test = http.assertGetRequest("/");
tab.open("http://localhost:81/foo");
test.wait();

Asynchronous tests

// A usefull function which allow you to block test execution for an amount of time in ms
wait.during(1000);

// The simpliest asynchronous test
wait.forTrue(function () {
  return true;
});

// Another, which is going to pass after 3s, with this setTimeout
var v=true;
wait.setTimeout(function() {
  v=false;
},3000);
wait.forFalse(function () {
  return v;
});

// Finally, a test which will pass when the test function is going to be called ten times
// (wait for the anonymous function returns 10)
var i=0;
wait.forEquals(function () {
  return i++;;
}, 10);

How to get it ?

Source code is available on github: http://github.com/ochameau/freemonkeys (LGPL licence)
If you are on windows:

• Download this package: freemonkeys-0.1-win.zip
• Extract this zip file somewhere
• Launch freemonkeys.exe

And for linux and mac:

• Download this one: freemonkeys-0.1.zip
• Extract it somewhere
• If you don't have xulrunner, download it from here
• Finally, launch Freemonkeys with this command:

  /path/to/your/xulrunner/dir/xulrunner /path/to/freemonkeys/application.ini
  

Here is another part of my work on memory analysis in Mozilla :

another-profiler_techno-barje.fr.xpi

This new version of "Another memory profiler" is now an effective tool, which display a lot of information about all objects living in your Firefox instance! By all I mean not only website javascript objects, but all objects used by Firefox in its windows, sidebars, extensions, all tabs, iframes, etc. The previous version allowed you only to select one component : a DOM window(website, sub-iframe or any XUL window), a XPCOM service or a JS Module :


Now you can get a report about currently living objects : the ones that are still used because there is at least one reference to each of them. This report first display all javascript files involved in your component :


By selecting one file, you are seeing the number of living object sorted by there instantiation line :


Finally, this tool display objects counts for each line sorted by there type. But Javascript is not a strongly typed language, so it's not really easy to sort its objects by a type! That's why there are several way to describe a JS object :

• Define a JS object by its attributes, like Atul Varma is doing in its current work,
• By its JS Prototype name, very usefull "typing" when you are using Prototype and build Object-Oriented JS,
• We are facing some specialized objects like all DOM objects : HTMLFormElement, HTMLDivElement, ...
• And finally all native types, like Array, String, Date, RegExp, ... etc.

Finally, let's see how to make this extension work :

• First It contains a binary component which is only built for Firefox 3.5 and 3.6 for Windows and Linux-32.
• Secondly The memory profiling component is a patched version of the Mozilla Jetpack's one, so take care to disable Jetpack, before testing this!
• Then In order to get the maximum information about your living JS object, I strongly encourage you to set these two prefs to false :

  javascript.options.jit.content = false
  javascript.options.jit.chrome = false
  

  (That's because Tracemonkey optimise loops and seems to embed empty stack frame information on these loop's execution ...)
• That being said, you just have to install this extension another-profiler_techno-barje.fr.xpi, go to your Tools menu and click on "Open another memory profiler".

Come back for the next post for some more explanation on displayed results with simple scripts examples.

As a Mozilla hacker, extension developer and Javascript expert, I've been really exited to see the current work of Atul Varma on memory profiling in Firefox! It's naturally the next step of tool to build after XUL Profiler, which track CPU consumption and Javascript functions calls.
So, instead of waiting for web developers to describe their future new "memory" firebug tab :), I've searched what information we can retrieve from JS API. And I've not limited my scope to web content but I take all Browser objects into account.

First I've tried to find a meaningful parent for every living object.
In the Mozilla planet we may face with three main types of parents :

• window : chrome (browser.xul, popups, jsconsole, sidebars, ...) or content (websites,popups,iframes)
• xpcom services
• JS modules

(But there is also XBL, sandboxes and some others strange things like "Block")

Here is the first result of this work :

another-profiler_techno-barje.fr-1.0.xpi

This extension need Jetpack 0.6+. It adds a "Open another memory profiler" item in Tools menu and display all living windows, xpcoms and modules. Then when you select one of them, it displays the simplest profiling ever: number of js objects group by C++ native class. I'll show you in the next blog post how to display a better profiling!
But for now, I'm going to show you all the code needed to make this first version.

For the living windows, there is a lot of cases, but it's simple :

// Get the list of absolutery ALL windows living in a Firefox session, stored as a Tree
function getAllWindows() {
  var windows = [];
  
  // Begin by iterating over all top chrome windows (browser, jsconsole, dominspector, etc.)
  var wm = Components.classes["@mozilla.org/appshell/window-mediator;1"]
          .getService(Components.interfaces.nsIWindowMediator);
  var enumerator = wm.getXULWindowEnumerator(null);
  while(enumerator.hasMoreElements()) {
    var win = enumerator.getNext();
    if (win instanceof Components.interfaces.nsIXULWindow) {
      // Search for all children windows (sidebar, content, iframes, ...)
      parseDocshell(win.docShell);
   }
  }

  function getWindowByDocShell(docShell) {
    if (!(docShell instanceof Components.interfaces.nsIInterfaceRequestor))
      return;
    return docShell.getInterface(Components.interfaces.nsIDOMWindow);
  }
  function parseDocshell(docShell) {
    if (!docShell) return;
    var domWindow = getWindowByDocShell(docShell);
    
    var topWindow = {
          type  : "window",
          name  : domWindow.document.title,
          href  : domWindow.location.href,
          object: domWindow,
          children: []
        };
    windows.push(topWindow);
    
    var topWindows = [topWindow];
    
    var treeItemType = Components.interfaces.nsIDocShellTreeItem.typeAll;
    // From inspector@mozilla.org inspector.js appendContainedDocuments
    // Load all the window's content docShells
    var containedDocShells = docShell.getDocShellEnumerator(treeItemType,
                                      Components.interfaces.nsIDocShell.ENUMERATE_FORWARDS);
    while (containedDocShells.hasMoreElements())
    {
      var childShell = containedDocShells.getNext().QueryInterface(Components.interfaces.nsIDocShell);
      
      if (childShell == docShell) {
        // It's the current topWindow
        continue;
      }
      
      var childDOMWindow = getWindowByDocShell(childShell);
      if (!childDOMWindow) continue;
      var parent;
      for(var i=0; i<topWindows.length; i++) {
        if (topWindows[i].object == childDOMWindow.parent) {
          parent = topWindows[i];
          break;
        }
      }
      var newWindow = {
        type  : "window",
        name  : childDOMWindow.document.title,
        href  : childDOMWindow.location.href,
        object: childDOMWindow,
        children : []
      };
      topWindows.push(newWindow);
      if (parent)
        parent.children.push(newWindow);
      else
        topWindow.children.push(newWindow);
    }
    delete topWindows;
  }
  
  // Finally, don't forget *the* hidden window, it's a big one used by many extensions!
  var hiddenWindow = Components.classes["@mozilla.org/appshell/appShellService;1"]
         .getService(Components.interfaces.nsIAppShellService)
         .hiddenWindow;
  if (hiddenWindow instanceof Components.interfaces.nsIXULWindow) {
    parseDocshell(hiddenWindow.docShell);
  }
  
  return windows;
}

For XPCOM services, it's shorter, but it' an unknown practice :

// Get the list of all XPCOM services (not the xpcom objects, only services) in a Firefox session
function getAllXPCOMServices() {
  var instanciatedServices = [];
  var serviceManager=Components.manager.QueryInterface(Components.interfaces.nsIServiceManager);
  var supports = Components.interfaces.nsISupports;
  for(var cl in Components.classes) {
    try {
      if (serviceManager.isServiceInstantiated(Components.classes[cl],supports)) {
        var service=Components.classes[cl].getService(supports);
        if (service.wrappedJSObject) {
          // Get the global object
          service=service.wrappedJSObject.__parent__;
          if (!service)
            service=Components.classes[cl].getService(supports).__parent__;
          instanciatedServices.push({
            type   : "xpcom",
            name   : cl,
            object : service
          });
        }
      }
    } catch(e) {
      // serviceManager.isServiceInstantiated is throwing if there is no instance ...
    }
  }
  return instanciatedServices;
}

But for JS Modules, I've not found any way to get those ...
The only solution I've got was to do a quick profiling and identify them :

function searchJSModules () {
  var jsmodules = [];
  
  var roots=getGCRoots();
  for(var r in roots) {
    var id = roots[r];
    var info = getObjectInfo(id);
    var properties = getObjectProperties(id);
    /*
    // We can also identify XPCOM by reading global NSGetModule function
    var nsgetmodule = getObjectProperty(id,"NSGetModule").NSGetModule;
    if (nsgetmodule) {
      print (" --> is an XPCOM");
      print (" --> defined in : "+getObjectInfo(nsgetmodule).filename);
      continue;
    }
    */
    // See if the current object has a EXPORTED_SYMBOLS object
    // We suppose every JS Module has one ...
    var exportedsymbols = getObjectProperty(id,"EXPORTED_SYMBOLS").EXPORTED_SYMBOLS;
    if (!exportedsymbols) continue;
    
    // Then search for the first declared function
    // Which will allow us to get the file of this module!
    
    // Begin to search in EXPORTED_SYMBOLS
    var symbols = getObjectProperties(exportedsymbols);
    var filename;
    for(var i in symbols) {
      var s = getObjectProperty(id,symbols[i])[symbols[i]];
      var inf = getObjectInfo(s);
      if (!inf) continue;
      if (inf.nativeClass=="Function" && inf.filename) {
        filename=inf.filename;
        break;
      } else if (inf.nativeClass="Object") {
        var subprops = getObjectProperties(s);
        for(var j in subprops) {
          var subs = subprops[j];
          var subinf = getObjectInfo(subs);
          if (!subinf) continue;
          if (subinf.nativeClass=="Function" && subinf.filename) {
            filename = subinf.filename;
            break;
          }
        }
        if (filename) break;
      }
    }
    if (!filename) {
      // Unable to found a function in exported_symbols objects
      // now try to find a function defined in global context
      var table = getObjectTable();
      var count=0;
      for (var subid in table) {
        var subinf = getObjectInfo(parseInt(subid));
        if (subinf && subinf.parent == id && subinf.nativeClass=="Function" && subinf.filename) {
          filename = subinf.filename;
          break;
        }
      }
    }
    if (filename) {
      var file = filename;
      var res = filename.match(/\/([^\/]+\/[^\/]+\/[^\/]+\.\w+)$/);
      if (res)
        file = decodeURIComponent(res[1]);
      jsmodules.push({
        type  : "jsmodule",
        name  : file,
        file  : filename,
        object: id
      });
    } else {
      // we were unable to find any function, we may try to search deeper
    }
  }
  
  return JSON.stringify(jsmodules);
}

function getAllJSModules() {
  var factory = Components.classes["@labs.mozilla.com/jetpackdi;1"]
               .createInstance(Components.interfaces.nsIJetpack);
  var endpoint = factory.get();
  var json = endpoint.profileMemory(searchJSModules.toSource()+"\nsearchJSModules()", "find-jsmodules.js", 1, null);
  return JSON.parse(json);
}

Finally, here is the function which retrieve objects counts for one parent. It use the Jetpack memory profiler XPCOM.

function profileFunction() {
  var namedObjects=getNamedObjects();
  
  // namedObjects["parent"] is null ... why ?!
  var parent;
  for(var i in namedObjects) {
    if (i=="parent") {
      parent = parseInt(namedObjects[i]);
    }
  }
  
  // Remove web content windows js wrapper
  var inf = getObjectInfo(parent);
  if (inf && inf.nativeClass=="XPCSafeJSObjectWrapper") {
    parent = inf.wrappedObject;
  }
  
  var children = {};
  
  // Check every JS object
  var table = getObjectTable();
  for(var i in table) {
    var info = getObjectInfo(parseInt(i));
    
    // Search if this one is related to the selected parent
    // ie walk throught all parents in order to find if the current object is a descendant of selected parent
    if ( info.parent != parent ) {
      var parentMatch = false;
      var p = info.parent;
      while(true) {
        var subinfo = getObjectInfo(p);
        if (!subinfo) break;
        
        if ( subinfo.id == parent || subinfo.parent == parent ) {
          // Answer= Yes
          parentMatch = true;
          break;
        }
        
        // Walk throught encapsulated objects
        if (subinfo.outerObject && subinfo.outerObject!=p) {
          p = subinfo.outerObject;
          continue;
        }
        
        p = subinfo.parent;
      }
      // Answer= Yes
      if (!parentMatch) continue;
    }
    
    if (!children[info.nativeClass])
      children[info.nativeClass] = 0;
    children[info.nativeClass]++;
  }
  
  return JSON.stringify(children);
}
function profileParent(parent) {
  var factory = Components.classes["@labs.mozilla.com/jetpackdi;1"]
               .createInstance(Components.interfaces.nsIJetpack);
  var endpoint = factory.get();
  var json = endpoint.profileMemory(profileFunction.toSource()+"\nprofileFunction()", "profile.js", 1, {parent: parent});
  return JSON.parse(json);
}

More information

Come back for the next blog post to get the 2.0 version :)

Here is a new Yoono feature that really need some emphasis!

Yoono Desktop Portable

This new version allows you to launch the Yoono's social network aggregation tool directly from a usb key. And the most important point is that your profile is saved on it, so no personal data is saved in the computer you're using and all your settings and socials updates travel with you.
The icing on the cake is that it's even more "portable" than just that! You can even launch this application on Windows AND MacOS!!!

Only two steps to test this :

• Unzip yoono-desktop-portable.zip somewhere on a usbkey
• On windows, launch "Yoono Desktop Portable.exe"
• On macos, launch "Yoono Desktop Portable"

The only drawback is usb drive read/write performance. Depending on drive model Yoono Desktop appears to take long time to load. But we are going to gain a huge speed up with current work on mozilla platform about startup performance. So don't forget to check out yoono desktop portable when Firefox 3.6 is out! (release planned before the end of the year)

Note: english translation of my previous post

Now that Improve gnu ELF bug is commited in ocaml 3.11+, KCachegrind can generate beautifull callgraphs.

This patch consist in adding .size instructions (in ELF assembly code) in order to allow valgrind to interpret all symbols (camlT_entry, camlT_foo, camlT_bar, ..) and it can so display symbols names instead of their hexadecimal numbers!!!

ELF instructions for debug

Now, we may want these tools to be able to display file name and line number for all functions.(File name is present in symbols name but it doesn't allow full usage of these tools)

Let's see how's gcc working :

  int bar(int a) {
        return 1+a;
  }
  
  int foo(int a) {
        return 2+bar(a);
  }
  
  int main() {
        foo(3);
  }

  $ gcc -O0 -g -S t.c

  .globl bar
        .type   bar, @function
  bar:
  .LFB2:
        .file 1 "t.c"
        .loc 1 1 0
        pushq   %rbp
  .LCFI0:
        movq    %rsp, %rbp
  .LCFI1:
        movl    %edi, -4(%rbp)
        .loc 1 2 0
        movl    -4(%rbp), %eax
        incl    %eax
        .loc 1 3 0
        leave
        ret
  .LFE2:
        .size   bar, .-bar
  .globl foo
        .type   foo, @function

Usefull instructions are .file and .loc :

• .file define a file path and bind it to an id
  

-> .file $file_id"$file_path"

• .loc define a file, a line and column number to the next instructions
  

-> .loc $file_id$ $line$ $column$

Suggested solution

The compiler module which emits these instructions is : asmcomp/i386/emit.mlp
And especially this "fundecl" function :

  let fundecl fundecl =
    function_name := fundecl.fun_name;
    fastcode_flag := fundecl.fun_fast;
    (* ... *)
    `   .globl  {emit_symbol fundecl.fun_name}\n`;
    `{emit_symbol fundecl.fun_name}:\n`;
    if !Clflags.gprofile then emit_profile();
    let n = frame_size() - 4 in
    if n > 0 then
      ` subl    ${emit_int n}, %esp\n`;
    `{emit_label !tailrec_entry_point}:\n`;
    emit_all true fundecl.fun_body;
    List.iter emit_call_gc !call_gc_sites;
    emit_call_bound_errors ();
    List.iter emit_float_constant !float_constants;
    match Config.system with
      "linux_elf" | "bsd_elf" | "gnu" ->
        `       .type   {emit_symbol fundecl.fun_name},@function\n`;
        `       .size   {emit_symbol fundecl.fun_name},.-{emit_symbol fundecl.fun_name}\n`
    | _ -> ()

Except that the only data we have is the fundecl variable :

  type fundecl = 
  { fun_name: string;
    fun_body: instruction;
    fun_fast: bool } 
  type instruction =
  { mutable desc: instruction_desc;
    mutable next: instruction;
    arg: Reg.t array;
    res: Reg.t array;
    dbg: Debuginfo.t;
    live: Reg.Set.t }

There is a dbg attribute on instructions but it's rarely set. (One compilation with -dlinear option allow to see this fact)

I've decided to add a fun_dbg : Debuginfo.t attribute on "fundecl" type and fill it in all compilation steps. It may more clever to work on this (often-empty) "dbg" attribute ? (it would allow to add position information on all instructions, it can be usefull for valgrind and gdb) This patch is not optimised because it repeats .file instruction for each .loc and so it repeats it on each function header.

-> Patch based on release311 branch, but works on current trunk

Now let's see what brings this patch

gdb results

  $ ocamlopt -g -inline 0 t.ml
  $ gdb a.out
  (gdb) break t.ml:6
  Breakpoint 1 at 0x8049940: file t.ml, line 6.
  (gdb) run
  Starting program: /home/alex/callgraph/a.out 
  
  Breakpoint 1, camlT__foo_60 () at t.ml:6
  6       let foo i  =
  Current language:  auto; currently asm

  (gdb) backtrace
  #0  camlT__foo_60 () at t.ml:7
  #1  0x0804c570 in camlT__entry () at t.ml:12
  #2  0x0806e4b7 in caml_start_program ()
  
  (gdb) step 1
  camlT__bar_58 () at t.ml:2
  2       let bar i =
  
  (gdb) list
  1
  2       let bar i =
  3               1+i
  4       ;;
  5
  6       let foo i  =
  7               2+(bar i )
  8       ;;
  9
  10      let () =

gprof results

  $ ocamlopt -g -p -inline 0 t.ml
  $ ./a.out
  $ gprof -A
  *** File /home/alex/callgraph/t.ml:
                  
           1 -> let bar i =
                        Thread.delay 3.0;
                        1+i
                ;;
                
           1 -> let foo i  =
                        2+(bar i )
                ;;
                
                let () =
           1 ->         let closure() = 3 in
                        print_int ( foo (closure()) )
                ;;
                
  Top 10 Lines:

     Line      Count
        2          1
        7          1
       12          1

  Execution Summary:

        3   Executable lines in this file
        3   Lines executed
   100.00   Percent of the file executed

        3   Total number of line executions
     1.00   Average executions per line

valgrind/kcachegrind results

  $ ocamlopt -g -inline 0 t.ml
  $ valgrind --tool=callgrind ./a.out
  $ callgrind_annotate callgrind.out.2152 t.ml
  --------------------------------------------------------------------------------
  -- User-annotated source: t.ml
  --------------------------------------------------------------------------------
  .  
  8  let bar i =
  77,715  => thread.ml:camlThread__delay_75 (1x)
  .      Thread.delay 3.0;
  .      1+i
  .  ;;
  .  
  3  let foo i  = 
  77,723  => t.ml:camlT__bar_58 (1x)
  .      2+(bar i )
  .  ;;
  .  
  .  let () =
  13          let closure() = 3 in
  1,692  => pervasives.ml:camlPervasives__output_string_215 (1x)
  2,312  => pervasives.ml:camlPervasives__string_of_int_154 (1x)
  77,726  => t.ml:camlT__foo_60 (1x)
  .      print_int ( foo (closure()) )
  .  ;;
  .  
  $ kcachegrind callgrind.out.2152

And next ?

We first need to wait approval for this new feature by ocaml community, I've submitted it there.
I someone from INRIA read this ... Don't hesitate to contact me, I'm open to work on a different approach.
After that, we may hope a lot of new features, like :

• breakpoints on any caml line (not only function call)
• gdb plugin allowing to read value in a breakpoint!

FoobarFox

FoobarFox features

• Retrieve currently playing song information into your Firefox
• Automatically post to your twitter account all listening tracks
• Search for information on wikipedia, myspace or google

FoobarFox prerequisites

• You have to be on Windows ...
• Need Firefox 3.6b1pre nightly or Firefox 3.7a1pre nightly
• Use Foobar music player

install foofox@techno-barje.fr.xpi

Real world JSCtypes usage

FoobarFox is not so usefull, it's mainly a proof of concept for JSCtypes capabilities.
Let's see how use JSCtypes to do fun things. You can copy paste this sample code into your JS Console while your foobar is playing something :

Components.utils.import("resource://gre/modules/ctypes.jsm");

/* Change the dll path if your main windows directory is not on C:\WINDOWS! */
var lib = ctypes.open("C:\\WINDOWS\\system32\\user32.dll");

/* Declare the signature of FindWindows function */
var findWindowEx = lib.declare("FindWindowW",
                               ctypes.stdcall_abi,
                               ctypes.int32_t,
                               ctypes.ustring,
                               ctypes.int32_t);

/* Search for Foobar windows by it's id */
/* this ID is often changing of value at each release :/ */
var win = findWindowEx("{DA7CD0DE-1602-45e6-89A1-C2CA151E008E}/1", 0);
if (!win)
  win = findWindowEx("{DA7CD0DE-1602-45e6-89A1-C2CA151E008E}", 0);
if (!win)
  win = findWindowEx("{97E27FAA-C0B3-4b8e-A693-ED7881E99FC1}", 0); 
if (!win)
  win = findWindowEx("{E7076D1C-A7BF-4f39-B771-BCBE88F2A2A8}", 0);

/* Define another signature of windows API function */
var getWindowText = lib.declare("GetWindowTextW",
                               ctypes.stdcall_abi,
                               ctypes.int32_t,
                               ctypes.int32_t,
                               ctypes.ustring,
                               ctypes.int32_t);

/* Fill the string buffer we give to JSCtypes call */
var text="";
var max_len = 100;
for(var i=0; i < max_len; i++)
  text+=" ";
var text_len = getWindowText(win,text,100);

/* Extract song information from foobar window title */
var m = text.match(/(.*) - (?:\[([^#]+)([^\]]+)\] |)(.*) \[foobar2000.*\]/);

var musicinfo = {
  artist : m[1],
  album : m[2],
  trackNumber : m[3],
  track : m[4]
};
alert(musicinfo.toSource());

JSCtypes current capabilities

As I said in my previous post, ctypes support in Firefox is limited and we can't use struct. So we're able to play only with libraries that wait string and int. You can even pass objects/structures, but only if they can be created by a function whose arguments support the same limitation. As you can see in the previous example, we're able to retrieve a pointer to a HWND object with FindWindowEx because it only wait string arguments. After that we give this pointer to getWindowText as a ctypes.int32_t.

JSCtypes current limitations

Now, let's see a C++ code that can't be mapped into JSCtypes :

// Add a notification to the tray.
NOTIFYICONDATA nid = {0};

nid.cbSize         = sizeof(nid);
nid.uID            = 100;    
nid.uFlags         = NIF_ICON;
nid.hIcon          = LoadIcon(g_hInstance, MAKEINTRESOURCE(IDI_SAMPLEICON));

Shell_NotifyIcon(NIM_ADD, &nid);

source: msdn

In this case, we are neither able to create any NOTIFYICONDATA object, nor to set attributes on this structure.

For more information

• JSCtypes api. But take care, this is a work in progress!
• Bug 513788 - Revise js-facing API for js-ctypes.
• Bug 429551 - add struct support to js-ctypes.

I've shared in the previous post a command line version of Mccoy. Here is a new tutorial on how to use it!

Prerequisite

• a HTTP server
• Patched version of mccoy, with command line capabilities : mccoy.tar.gz
• This start kit : mccoy-test.tar.gz which bundle a sample extension and one update.xml file

$ cd /one/of/your/htdocs/dir
$ wget http://blog.techno-barje.fr/public/mccoy.tar.gz
$ tar zxvf mccoy.tar.gz
$ wget http://blog.techno-barje.fr/public/mccoy-test.tar.gz
$ tar zxvf mccoy-test.tar.gz
$ cd mccoy-test/
$ ls
update.xml  workdir  xpis

Setup your XPI with valid update information

Create a new key in Mccoy

mccoy-test $ cd workdir/
workdir $ ls
chrome  chrome.manifest  install.rdf
workdir $ ../../mccoy -createKey myextensionkey
Creating key with name : myextensionkey
Public key : MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDbV+ZGXs658dOm/+4YtT+VzT5JWzMFYiQ8155fnMkOJCina2yDEBq8Lvi5qF5SyoMDkqaYeO51LR+B4p1g7oWmBW9HbOz3eA9lD/AHUR1SHiJAX7RQq8v9sPSkYta+LyVrCMFgpTmhOWPUXOnwalmL7syGkXyjxHqHCYz+s3d22QIDAQAB
The key has been successfully created!

Remember the name of your key (if you forgot the name, you can later execute mccoy -listKeys)
Inject the public key in your extension

workdir $ ./mccoy -installKey install.rdf -key myextensionkey
Public key inserted!

This will set the updateKey attribute with the public key. (you can later retrieve the public key with mccoy -publicKey myextension)
Set the updateURL attribute of the install.rdf with the URL of the update.xml file located in mccoy-test/update.xml

workdir $ vi update.rdf

Build the first xpi

$ zip -r ../xpis/mccoy-test-0.1.xpi .

»» now install this XPI! This sample extension will just display an alert with message "Mccoy 0.1!"

Create a new version of your extension

Alter the sample extension alert message with something new

workdir $ vi chrome/content/firefoxOverlay.xul

Update the version number with 0.2

workdir $ vi install.rdf

Build the new xpi

workdir $ zip -r ../xpis/mccoy-test-0.2.xpi .

Update the update xml file

workdir $ cd ..
mccoy-test $ vi update.xml
### change version with 0.2
### change updatelink with mccoy-test-0.2.xpi
### change updatehash with result of sha1sum xpis/mccoy-test-0.2.xpi

Sign the update file with mccoy

mccoy-test $ ../mccoy/mccoy -signRDF update.xml -key myextensionkey
Sign < update.xml > with key < myextensionkey >
Sign addon : urn:mozilla:extension:mccoy-test@techno-barje.fr
File signed!

This will set the signature attribute with computed with your private key.

»» You can now force the update in your firefox, relaunch it and voilà!

Some tips for debugging

Enable this two about:config entries in order to get some message in JS console about update process :

extensions.logging.enabled = true
javascript.options.showInConsole = true

Here is a summary on how to run a xulrunner application on an headless computer, or more commonly just launch xulrunner in a command line with no windows.

By default, xulrunner try to open a XUL window defined in the ""toolkit.defaultChromeURI"" preference, so you have to set this one to an empty value.

Then you need to have a running X server, even if it never open any window ... one simple and lightweight solution is running Xvfb. But any other X server will do the work!

Finally, I suggest you to take the 1.9.0.14 xulrunner release which has less painfull dependencies like libalsa (due to ogg support) and libdbus-glib.

This will avoid this kind of errors :

./xulrunner-bin: error while loading shared libraries: libasound.so.2: cannot open shared object file: No such file or directory
./xulrunner-bin: error while loading shared libraries: libdbus-glib-1.so.2: cannot open shared object file: No such file or directory

How to write such a tutorial without a complete working out of the box hello world ?
Here is a sample command line application with linux xulrunner binaries : headless-runner.tar.gz

$ tar zxvf headless-runner.tar.gz
$ cd headless-runner
$ Xvfb :2
$ export DISPLAY=:2
$ ./headless -ls -filesize application.ini
LS :
 - application.ini
 - a.out
 - tests
 - components
 - defaults
 - updates
 - extensions
 - xulrunner-1.9.2a2pre.en-US.linux-x86_64.tar.bz2
 - xulrunner
 - headless

$ ./headless -filesize application.ini
File size of : application.ini
  243

The main code is in the file components/nsCommandLineHandler.js

CommandLineHandler.prototype.handle = function(aCmdLine){
  
  var toggle = aCmdLine.handleFlag("ls", false);
  if (toggle) {
    dump("LS : \n");
    var list = aCmdLine.workingDirectory.directoryEntries;
    while(list.hasMoreElements()) {
      var file = list.getNext().QueryInterface(Components.interfaces.nsIFile);
      dump(" - "+file.leafName+"\n");
    }
    dump("\n");
  }

  var filesize = aCmdLine.handleFlagWithParam("filesize", false);
  if (filesize) {
    dump("File size of : "+filesize+"\n");
    var file = aCmdLine.resolveFile(filesize);
    if (!file)
      return dump("Unable to find this file\n");
    dump("  "+file.fileSize+"\n");
  }
}

For more information, check the nsICommandLine interface of the aCmdLine object.

Last but not least, why do I try to use Xulrunner in command line whereas xpcshell and "js" commands exists?!

• First: Some tools like Mccoy are bundled as xulrunner application. And you may launch these tools on headless servers in order to build, for example, continuous integration tools!
• Second: JS shell allow you tu use only pure Javascript; Xpcshell expose all XPCOM but some part of Mozilla environnement are disabled! I was unabled to create a <canvas> element in xpcshell. There is no way to create a XUL/HTML document/element with XPCOM and hiddenWindow is disabled ... So the only solution to build a tool which takes website screenshots with <canvas> was using xulrunner!

Maintenant que le bug Improve gnu ELF est corrigé, kcachegrind nous génère de beaux callgraphs.

Ce patch consistait à ajouter des instructions .size (dans l'assembleur ELF) pour que valgrind interprète tous les symboles (camT_entry, camlT_foo, camlT_bar, ...) et puisse ainsi afficher leurs noms au lieu d'un nombre hexadécimal!

Les standards ELF pour le debug

Maintenant, nous souhaiterions que ces outils puissent savoir de manière standard dans quels fichiers et à quelles lignes sont déclarés les fonctions. (Le nom du fichier est bien présent dans les symboles, mais cela ne permet pas d'exploiter le plein potentiel de ces outils)

Voyons un peu comment se débrouille GCC :

  int bar(int a) {
        return 1+a;
  }
  
  int foo(int a) {
        return 2+bar(a);
  }
  
  int main() {
        foo(3);
  }

  $ gcc -O0 -g -S t.c

  .globl bar
        .type   bar, @function
  bar:
  .LFB2:
        .file 1 "t.c"
        .loc 1 1 0
        pushq   %rbp
  .LCFI0:
        movq    %rsp, %rbp
  .LCFI1:
        movl    %edi, -4(%rbp)
        .loc 1 2 0
        movl    -4(%rbp), %eax
        incl    %eax
        .loc 1 3 0
        leave
        ret
  .LFE2:
        .size   bar, .-bar
  .globl foo
        .type   foo, @function

Les instructions clés sont .file et .loc :

• .file déclare un chemin vers un fichier et l'associe à un identifiant
  

-> .file $identifiant$ "$chemin_du_fichier"

• .loc associe un fichier, un numéro de ligne et de colonne aux instructions la succédant
  

-> .loc $identifiant_de_fichier$ $ligne$ $colonne$

Solution proposée

Le module du compilateur qui génère les instructions est : asmcomp/i386/emit.mlp
Et plus particulièrement la fonction fundecl :

  let fundecl fundecl =
    function_name := fundecl.fun_name;
    fastcode_flag := fundecl.fun_fast;
    (* ... *)
    `   .globl  {emit_symbol fundecl.fun_name}\n`;
    `{emit_symbol fundecl.fun_name}:\n`;
    if !Clflags.gprofile then emit_profile();
    let n = frame_size() - 4 in
    if n > 0 then
      ` subl    ${emit_int n}, %esp\n`;
    `{emit_label !tailrec_entry_point}:\n`;
    emit_all true fundecl.fun_body;
    List.iter emit_call_gc !call_gc_sites;
    emit_call_bound_errors ();
    List.iter emit_float_constant !float_constants;
    match Config.system with
      "linux_elf" | "bsd_elf" | "gnu" ->
        `       .type   {emit_symbol fundecl.fun_name},@function\n`;
        `       .size   {emit_symbol fundecl.fun_name},.-{emit_symbol fundecl.fun_name}\n`
    | _ -> ()

Hors nous n'avons à disposition que la variable fundecl :

  type fundecl = 
  { fun_name: string;
    fun_body: instruction;
    fun_fast: bool } 
  type instruction =
  { mutable desc: instruction_desc;
    mutable next: instruction;
    arg: Reg.t array;
    res: Reg.t array;
    dbg: Debuginfo.t;
    live: Reg.Set.t }

Il y a bien un attribut dbg sur les instructions mais il est rarement renseigné. (une compilation avec l'option -dlinear permet de le voir)

J'ai décidé d'ajouter un attribut fun_dbg : Debuginfo.t sur fundecl et de le remplir dans toutes les phases de compilation. Il serait peut être plus judicieux de travailler sur l'attribut dbg des instructions ? (car cela permettrait par la suite d'indiquer les lignes de chaque instruction à valgrind, mais aussi gdb!) Ce patch n'est pas optimisé car il répète l'instruction .file pour chaque .loc, donc à chaque entête de fonction, nous avons un .file et un .loc.

-> Patch sur la branche release311

Voyons maintenant ce qu'apporte ce patch.

Résultats avec gdb

  $ ocamlopt -g -inline 0 t.ml
  $ gdb a.out
  (gdb) break t.ml:6
  Breakpoint 1 at 0x8049940: file t.ml, line 6.
  (gdb) run
  Starting program: /home/alex/callgraph/a.out 
  
  Breakpoint 1, camlT__foo_60 () at t.ml:6
  6       let foo i  =
  Current language:  auto; currently asm

  (gdb) backtrace
  #0  camlT__foo_60 () at t.ml:7
  #1  0x0804c570 in camlT__entry () at t.ml:12
  #2  0x0806e4b7 in caml_start_program ()
  
  (gdb) step 1
  camlT__bar_58 () at t.ml:2
  2       let bar i =
  
  (gdb) list
  1
  2       let bar i =
  3               1+i
  4       ;;
  5
  6       let foo i  =
  7               2+(bar i )
  8       ;;
  9
  10      let () =

Résultats avec gprof

  $ ocamlopt -g -p -inline 0 t.ml
  $ ./a.out
  $ gprof -A
  *** File /home/alex/callgraph/t.ml:
                  
           1 -> let bar i =
                        Thread.delay 3.0;
                        1+i
                ;;
                
           1 -> let foo i  =
                        2+(bar i )
                ;;
                
                let () =
           1 ->         let closure() = 3 in
                        print_int ( foo (closure()) )
                ;;
                
  Top 10 Lines:

     Line      Count
        2          1
        7          1
       12          1

  Execution Summary:

        3   Executable lines in this file
        3   Lines executed
   100.00   Percent of the file executed

        3   Total number of line executions
     1.00   Average executions per line

Résultats avec valgrind/kcachegrind

  $ ocamlopt -g -inline 0 t.ml
  $ valgrind --tool=callgrind ./a.out
  $ callgrind_annotate callgrind.out.2152 t.ml
  --------------------------------------------------------------------------------
  -- User-annotated source: t.ml
  --------------------------------------------------------------------------------
  .  
  8  let bar i =
  77,715  => thread.ml:camlThread__delay_75 (1x)
  .      Thread.delay 3.0;
  .      1+i
  .  ;;
  .  
  3  let foo i  = 
  77,723  => t.ml:camlT__bar_58 (1x)
  .      2+(bar i )
  .  ;;
  .  
  .  let () =
  13          let closure() = 3 in
  1,692  => pervasives.ml:camlPervasives__output_string_215 (1x)
  2,312  => pervasives.ml:camlPervasives__string_of_int_154 (1x)
  77,726  => t.ml:camlT__foo_60 (1x)
  .      print_int ( foo (closure()) )
  .  ;;
  .  
  $ kcachegrind callgrind.out.2152

Et après ?

On peut espérer encore un tas d'améliorations, comme :

• des breakpoints sur n'importe quelle ligne de caml ...
• un pluging gdb pour pouvoir lire des valeurs pendant un break!