Techno Barje

SabnzbFox

Now, a real world usage of my previous JS Module.


SabnzbdFox


SabnzbdFox features

  • Catch absolutely all NZB file requests
  • Automatically save all these files to a specific directory (not limited to sabnzbd usage!)
  • Or Automatically upload to sabnzbd server via its web-API
  • Display current downloads count

SabnzbdFox prerequisites

  • Firefox
  • Sabnzbd or any newsreader which automatically read nzb files in a directory.

install SabnzbdFox on addons.mozilla.org

sabnzbdfox

Ocaml Native Code Debugging

Note: english translation of my previous post

t-caml-valid-callgraph.png

Now that Improve gnu ELF bug is commited in ocaml 3.11+, KCachegrind can generate beautifull callgraphs.

This patch consist in adding .size instructions (in ELF assembly code) in order to allow valgrind to interpret all symbols (camlT_entry, camlT_foo, camlT_bar, ..) and it can so display symbols names instead of their hexadecimal numbers!!!

ELF instructions for debug

Now, we may want these tools to be able to display file name and line number for all functions.(File name is present in symbols name but it doesn’t allow full usage of these tools)

Let’s see how’s gcc working :

  int bar(int a) {
        return 1+a;
  }
  
  int foo(int a) {
        return 2+bar(a);
  }
  
  int main() {
        foo(3);
  }
  $ gcc -O0 -g -S t.c
  .globl bar
        .type   bar, @function
  bar:
  .LFB2:
        .file 1 "t.c"
        .loc 1 1 0
        pushq   %rbp
  .LCFI0:
        movq    %rsp, %rbp
  .LCFI1:
        movl    %edi, -4(%rbp)
        .loc 1 2 0
        movl    -4(%rbp), %eax
        incl    %eax
        .loc 1 3 0
        leave
        ret
  .LFE2:
        .size   bar, .-bar
  .globl foo
        .type   foo, @function

Usefull instructions are .file and .loc :

  • .file define a file path and bind it to an id

-> .file $file_id"$file_path"

  • .loc define a file, a line and column number to the next instructions

-> .loc $file_id$ $line$ $column$

Suggested solution

The compiler module which emits these instructions is : asmcomp/i386/emit.mlp
And especially this "fundecl" function :

  let fundecl fundecl =
    function_name := fundecl.fun_name;
    fastcode_flag := fundecl.fun_fast;
    (* ... *)
    `   .globl  {emit_symbol fundecl.fun_name}
`;
    `{emit_symbol fundecl.fun_name}:
`;
    if !Clflags.gprofile then emit_profile();
    let n = frame_size() - 4 in
    if n > 0 then
      ` subl    ${emit_int n}, %esp
`;
    `{emit_label !tailrec_entry_point}:
`;
    emit_all true fundecl.fun_body;
    List.iter emit_call_gc !call_gc_sites;
    emit_call_bound_errors ();
    List.iter emit_float_constant !float_constants;
    match Config.system with
      "linux_elf" | "bsd_elf" | "gnu" ->
        `       .type   {emit_symbol fundecl.fun_name},@function
`;
        `       .size   {emit_symbol fundecl.fun_name},.-{emit_symbol fundecl.fun_name}
`
    | _ -> ()

Except that the only data we have is the fundecl variable :

  type fundecl = 
  { fun_name: string;
    fun_body: instruction;
    fun_fast: bool } 
  type instruction =
  { mutable desc: instruction_desc;
    mutable next: instruction;
    arg: Reg.t array;
    res: Reg.t array;
    dbg: Debuginfo.t;
    live: Reg.Set.t }

There is a dbg attribute on instructions but it’s rarely set. (One compilation with -dlinear option allow to see this fact)

I’ve decided to add a fun_dbg : Debuginfo.t attribute on "fundecl" type and fill it in all compilation steps. It may more clever to work on this (often-empty) "dbg" attribute ? (it would allow to add position information on all instructions, it can be usefull for valgrind and gdb) This patch is not optimised because it repeats .file instruction for each .loc and so it repeats it on each function header.

-> Patch based on release311 branch, but works on current trunk

Now let’s see what brings this patch

gdb results

  $ ocamlopt -g -inline 0 t.ml
  $ gdb a.out
  (gdb) break t.ml:6
  Breakpoint 1 at 0x8049940: file t.ml, line 6.
  (gdb) run
  Starting program: /home/alex/callgraph/a.out 
  
  Breakpoint 1, camlT__foo_60 () at t.ml:6
  6       let foo i  =
  Current language:  auto; currently asm

  (gdb) backtrace
  #0  camlT__foo_60 () at t.ml:7
  #1  0x0804c570 in camlT__entry () at t.ml:12
  #2  0x0806e4b7 in caml_start_program ()
  
  (gdb) step 1
  camlT__bar_58 () at t.ml:2
  2       let bar i =
  
  (gdb) list
  1
  2       let bar i =
  3               1+i
  4       ;;
  5
  6       let foo i  =
  7               2+(bar i )
  8       ;;
  9
  10      let () =

gprof results

  $ ocamlopt -g -p -inline 0 t.ml
  $ ./a.out
  $ gprof -A
  *** File /home/alex/callgraph/t.ml:
                  
           1 -> let bar i =
                        Thread.delay 3.0;
                        1+i
                ;;
                
           1 -> let foo i  =
                        2+(bar i )
                ;;
                
                let () =
           1 ->         let closure() = 3 in
                        print_int ( foo (closure()) )
                ;;
                
  Top 10 Lines:

     Line      Count
        2          1
        7          1
       12          1

  Execution Summary:

        3   Executable lines in this file
        3   Lines executed
   100.00   Percent of the file executed

        3   Total number of line executions
     1.00   Average executions per line

valgrind/kcachegrind results

  $ ocamlopt -g -inline 0 t.ml
  $ valgrind --tool=callgrind ./a.out
  $ callgrind_annotate callgrind.out.2152 t.ml
  --------------------------------------------------------------------------------
  -- User-annotated source: t.ml
  --------------------------------------------------------------------------------
  .  
  8  let bar i =
  77,715  => thread.ml:camlThread__delay_75 (1x)
  .      Thread.delay 3.0;
  .      1+i
  .  ;;
  .  
  3  let foo i  = 
  77,723  => t.ml:camlT__bar_58 (1x)
  .      2+(bar i )
  .  ;;
  .  
  .  let () =
  13          let closure() = 3 in
  1,692  => pervasives.ml:camlPervasives__output_string_215 (1x)
  2,312  => pervasives.ml:camlPervasives__string_of_int_154 (1x)
  77,726  => t.ml:camlT__foo_60 (1x)
  .      print_int ( foo (closure()) )
  .  ;;
  .  
  $ kcachegrind callgrind.out.2152

kcachegrind-file-and-line.png



And next ?

We first need to wait approval for this new feature by ocaml community, I’ve submitted it there.
I someone from INRIA read this … Don’t hesitate to contact me, I’m open to work on a different approach.
After that, we may hope a lot of new features, like :

  • breakpoints on any caml line (not only function call)
  • gdb plugin allowing to read value in a breakpoint!

Catch All Requests to a Specific Mime Type/file Extension in Firefox

Here is a Mozilla Javascript Module which allow to catch absolutely all requests based on their Content-Type Http header (ie Mime type) or their filename.

contentTypeObserver.js
(under LGPL License)

Normally, it would be simple to catch all firefox request by adding a nsIURIContentListener like this :

Components.classes["@mozilla.org/uriloader;1"].getService(Components.interfaces.nsIURILoader).registerContentListener( ...nsIURIContentListener... );

But for some reason, this listener is bypassed here when the HTTP request contains a Content-Disposition header :(
So I give you there all Mozilla black magic needed to catch really all requests.

Hello world

Components.utils.import("resource://your-extension/contentTypeObserver.js"); 
var contentTypeObserver = {};

// Tell if we must catch requests with this content-type
// requestInfo is an object with 3 attributes : contentType, contentLength and fileName.
contentTypeObserver.getRequestListener = function (requestInfo) {
  // Return a new instance of nsIWebProgressListener
  // (a new instance to avoid conflicts with multiple simultaneous downloads)
  return {
    onStartRequest : function (request, context) {

    },
    onStopRequest : function (request, context, statusCode) {

    },
    onDataAvailable : function (request, context, inputStream, offset, count) {

    }
  };
  // There is an helper function that allow to automatically save this request to a file,
  // you just have to pass destinationFile argument which hold a nsIFile instance :
  return createSaveToFileRequestListener(requestInfo, destinationFile, function () { dump("file : "+destinationFile.spec+" downloaded!
"); }
}

addContentTypeObserver(contentTypeObserver);

FoobarFox : JSCtypes Putting Foobar Into Your Firefox!

FoobarFox

FoobarFox features

  • Retrieve currently playing song information into your Firefox
  • Automatically post to your twitter account all listening tracks
  • Search for information on wikipedia, myspace or google

FoobarFox prerequisites

install foofox@techno-barje.fr.xpi

foobarfox foobarfox



Real world JSCtypes usage

FoobarFox is not so usefull, it’s mainly a proof of concept for JSCtypes capabilities.
Let’s see how use JSCtypes to do fun things. You can copy paste this sample code into your JS Console while your foobar is playing something :

Components.utils.import("resource://gre/modules/ctypes.jsm");

/* Change the dll path if your main windows directory is not on C:\WINDOWS! */
var lib = ctypes.open("C:\\WINDOWS\\system32\\user32.dll");

/* Declare the signature of FindWindows function */
var findWindowEx = lib.declare("FindWindowW",
                               ctypes.stdcall_abi,
                               ctypes.int32_t,
                               ctypes.ustring,
                               ctypes.int32_t);

/* Search for Foobar windows by it's id */
/* this ID is often changing of value at each release :/ */
var win = findWindowEx("{DA7CD0DE-1602-45e6-89A1-C2CA151E008E}/1", 0);
if (!win)
  win = findWindowEx("{DA7CD0DE-1602-45e6-89A1-C2CA151E008E}", 0);
if (!win)
  win = findWindowEx("{97E27FAA-C0B3-4b8e-A693-ED7881E99FC1}", 0); 
if (!win)
  win = findWindowEx("{E7076D1C-A7BF-4f39-B771-BCBE88F2A2A8}", 0);

/* Define another signature of windows API function */
var getWindowText = lib.declare("GetWindowTextW",
                               ctypes.stdcall_abi,
                               ctypes.int32_t,
                               ctypes.int32_t,
                               ctypes.ustring,
                               ctypes.int32_t);

/* Fill the string buffer we give to JSCtypes call */
var text="";
var max_len = 100;
for(var i=0; i < max_len; i++)
  text+=" ";
var text_len = getWindowText(win,text,100);

/* Extract song information from foobar window title */
var m = text.match(/(.*) - (?:\[([^#]+)([^\]]+)\] |)(.*) \[foobar2000.*\]/);

var musicinfo = {
  artist : m[1],
  album : m[2],
  trackNumber : m[3],
  track : m[4]
};
alert(musicinfo.toSource());

JSCtypes current capabilities

As I said in my previous post, ctypes support in Firefox is limited and we can’t use struct. So we’re able to play only with libraries that wait string and int. You can even pass objects/structures, but only if they can be created by a function whose arguments support the same limitation. As you can see in the previous example, we’re able to retrieve a pointer to a HWND object with FindWindowEx because it only wait string arguments. After that we give this pointer to getWindowText as a ctypes.int32_t.

JSCtypes current limitations

Now, let’s see a C++ code that can’t be mapped into JSCtypes :

// Add a notification to the tray.
NOTIFYICONDATA nid = {0};

nid.cbSize         = sizeof(nid);
nid.uID            = 100;    
nid.uFlags         = NIF_ICON;
nid.hIcon          = LoadIcon(g_hInstance, MAKEINTRESOURCE(IDI_SAMPLEICON));

Shell_NotifyIcon(NIM_ADD, &nid);

source: msdn

In this case, we are neither able to create any NOTIFYICONDATA object, nor to set attributes on this structure.


For more information

  • JSCtypes api. But take care, this is a work in progress!
  • Bug 513788 - Revise js-facing API for js-ctypes.
  • Bug 429551 - add struct support to js-ctypes.

JS Ctypes

Bug 518721 - Implement jsctypes with raw JSAPI
Status: SOLVED FIXED!!!

That’s an awesome news for mozilla’s developpers!
JS Ctypes aims to provide the same library than Python Ctypes :

You can load any dynamic library (dll, so, dylib) and call C-functions directly from your Javascript. In the current implementation only simple types are supported : numbers, string, boolean. For now, we can’t play with pointers, nor structures, but these features are planned.


simpler code -> less work -> simpler review

This new feature is going to greatly ease platform specific developpement like windows management, system calls, … For example we’re able to call Windows API directly from Javascript, without having to create, compile and maintain any C++ XPCOM!
One side effect is that it will ease code review for https://addons.mozilla.org/ too! Instead of shipping an obscure dynamic library with our extension, we may build only a JS-Ctypes wrapper and call directly OS libraries or call a common library that can be validated by reviewers with some MD5 checks.


simpler code -> less knownledge -> better learning curve

This is going to simplify the use of native code too! You can now build native code without having to learn any mozilla "things" (XPCOM, specific build layout/system, …) You will just have to expose your library with a C api and write a simple JS-CTypes wrapper.


Hello World!

  • First retrieve Firefox 3.6b1pre nightly or Firefox 3.7a1pre nightly
  • On windows, copy and paste this code in your JS console.
    This will display an OS native dialog.
    (Change the dll path if your main windows directory is not on C:\WINDOWS!)
    /* Load JS Ctypes Javascript module */
    Components.utils.import("resource://gre/modules/ctypes.jsm");
    var Types = ctypes.types;
    
    /* Load windows api dll */
    var lib = ctypes.open("C:\\WINDOWS\\system32\\user32.dll");
    
    /* Declare the signature of the function we are going to call */
    var msgBox = lib.declare("MessageBoxW",
                             ctypes.stdcall_abi,
                             ctypes.int32_t,
                             ctypes.int32_t,
                             ctypes.ustring,
                             ctypes.ustring,
                             ctypes.int32_t);
    var MB_OK = 3;
    
    /* Do it! */
    var ret = msgBox(0, "Hello world", "title", MB_OK);
    
    /* Display the returned value */
    alert("MessageBox result : "+ret);
    
    lib.close();
    

How to Setup a Mozilla Extension Update Server

I’ve shared in the previous post a command line version of Mccoy. Here is a new tutorial on how to use it!

Prerequisite

  • a HTTP server
  • Patched version of mccoy, with command line capabilities : mccoy.tar.gz
  • This start kit : mccoy-test.tar.gz which bundle a sample extension and one update.xml file
$ cd /one/of/your/htdocs/dir
$ wget http://blog.techno-barje.fr/public/mccoy.tar.gz
$ tar zxvf mccoy.tar.gz
$ wget http://blog.techno-barje.fr/public/mccoy-test.tar.gz
$ tar zxvf mccoy-test.tar.gz
$ cd mccoy-test/
$ ls
update.xml  workdir  xpis


Setup your XPI with valid update information

Create a new key in Mccoy

mccoy-test $ cd workdir/
workdir $ ls
chrome  chrome.manifest  install.rdf
workdir $ ../../mccoy -createKey myextensionkey
Creating key with name : myextensionkey
Public key : MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDbV+ZGXs658dOm/+4YtT+VzT5JWzMFYiQ8155fnMkOJCina2yDEBq8Lvi5qF5SyoMDkqaYeO51LR+B4p1g7oWmBW9HbOz3eA9lD/AHUR1SHiJAX7RQq8v9sPSkYta+LyVrCMFgpTmhOWPUXOnwalmL7syGkXyjxHqHCYz+s3d22QIDAQAB
The key has been successfully created!

Remember the name of your key (if you forgot the name, you can later execute mccoy -listKeys)
Inject the public key in your extension

workdir $ ./mccoy -installKey install.rdf -key myextensionkey
Public key inserted!

This will set the updateKey attribute with the public key. (you can later retrieve the public key with mccoy -publicKey myextension)
Set the updateURL attribute of the install.rdf with the URL of the update.xml file located in mccoy-test/update.xml

workdir $ vi update.rdf

Build the first xpi

$ zip -r ../xpis/mccoy-test-0.1.xpi .

»» now install this XPI! This sample extension will just display an alert with message "Mccoy 0.1!"


Create a new version of your extension

Alter the sample extension alert message with something new

workdir $ vi chrome/content/firefoxOverlay.xul

Update the version number with 0.2

workdir $ vi install.rdf

Build the new xpi

workdir $ zip -r ../xpis/mccoy-test-0.2.xpi .

Update the update xml file

workdir $ cd ..
mccoy-test $ vi update.xml
### change version with 0.2
### change updatelink with mccoy-test-0.2.xpi
### change updatehash with result of sha1sum xpis/mccoy-test-0.2.xpi

Sign the update file with mccoy

mccoy-test $ ../mccoy/mccoy -signRDF update.xml -key myextensionkey
Sign < update.xml > with key < myextensionkey >
Sign addon : urn:mozilla:extension:mccoy-test@techno-barje.fr
File signed!

This will set the signature attribute with computed with your private key.


»» You can now force the update in your firefox, relaunch it and voilà!


Some tips for debugging

Enable this two about:config entries in order to get some message in JS console about update process :

extensions.logging.enabled = true
javascript.options.showInConsole = true

Mozilla Mccoy Tool From the Command Line

We have seen in the previous post how to build an headless xulrunner application.
My first use case was enabling the Mozilla Mccoy application to launch from command line. That allows me to build a bash script which create nightly builds of a firefox extension automatically updated by Firefox every day!.


Here is the resulting Mccoy version :


Now, a summary of each command line arguments :

# Create a new named public/private key
$ ./mccoy -createKey mykey
Creating key with name : mykey
Public key : MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDbV+ZGXs658dOm/+4YtT+VzT5JWzMFYiQ8155fnMkOJCina2yDEBq8Lvi5qF5SyoMDkqaYeO51LR+B4p1g7oWmBW9HbOz3eA9lD/AHUR1SHiJAX7RQq8v9sPSkYta+LyVrCMFgpTmhOWPUXOnwalmL7syGkXyjxHqHCYz+s3d22QIDAQAB
The key has been successfully created!

# List all keys in the current mccoy profile
# /!\ Don't forget to save your profile!
$ ./mccoy -listKeys
Registered keys :
 - mykey

# Inject the public key in your extension's install.rdf
$ ./mccoy -installRDF myextension_workdir/install.rdf -key myextensionkey
Public key inserted!

# Update the signature of the update xml file
$ ./mccoy -signRDF update.xml -key mykey
Sign < update.xml > with key < mykey >
Sign addon : urn:mozilla:extension:mccoy@techno-barje.fr
File signed!

# Check if the update xml file is correctly signed
$ ./mccoy -verifyRDF update.xml -key myextensionkey
Check rdf : update.xml with key myextensionkey
Valid signature!

Headless Xulrunner

Here is a summary on how to run a xulrunner application on an headless computer, or more commonly just launch xulrunner in a command line with no windows.


By default, xulrunner try to open a XUL window defined in the ""toolkit.defaultChromeURI"" preference, so you have to set this one to an empty value.


Then you need to have a running X server, even if it never open any window … one simple and lightweight solution is running Xvfb. But any other X server will do the work!


Finally, I suggest you to take the 1.9.0.14 xulrunner release which has less painfull dependencies like libalsa (due to ogg support) and libdbus-glib.


This will avoid this kind of errors :

./xulrunner-bin: error while loading shared libraries: libasound.so.2: cannot open shared object file: No such file or directory
./xulrunner-bin: error while loading shared libraries: libdbus-glib-1.so.2: cannot open shared object file: No such file or directory

How to write such a tutorial without a complete working out of the box hello world ?
Here is a sample command line application with linux xulrunner binaries : headless-runner.tar.gz

$ tar zxvf headless-runner.tar.gz
$ cd headless-runner
$ Xvfb :2
$ export DISPLAY=:2
$ ./headless -ls -filesize application.ini
LS :
 - application.ini
 - a.out
 - tests
 - components
 - defaults
 - updates
 - extensions
 - xulrunner-1.9.2a2pre.en-US.linux-x86_64.tar.bz2
 - xulrunner
 - headless

$ ./headless -filesize application.ini
File size of : application.ini
  243

The main code is in the file components/nsCommandLineHandler.js

CommandLineHandler.prototype.handle = function(aCmdLine){
  
  var toggle = aCmdLine.handleFlag("ls", false);
  if (toggle) {
    dump("LS : 
");
    var list = aCmdLine.workingDirectory.directoryEntries;
    while(list.hasMoreElements()) {
      var file = list.getNext().QueryInterface(Components.interfaces.nsIFile);
      dump(" - "+file.leafName+"
");
    }
    dump("
");
  }

  var filesize = aCmdLine.handleFlagWithParam("filesize", false);
  if (filesize) {
    dump("File size of : "+filesize+"
");
    var file = aCmdLine.resolveFile(filesize);
    if (!file)
      return dump("Unable to find this file
");
    dump("  "+file.fileSize+"
");
  }
}

For more information, check the nsICommandLine interface of the aCmdLine object.


Last but not least, why do I try to use Xulrunner in command line whereas xpcshell and "js" commands exists?!

  • First: Some tools like Mccoy are bundled as xulrunner application. And you may launch these tools on headless servers in order to build, for example, continuous integration tools!
  • Second: JS shell allow you tu use only pure Javascript; Xpcshell expose all XPCOM but some part of Mozilla environnement are disabled! I was unabled to create a <canvas> element in xpcshell. There is no way to create a XUL/HTML document/element with XPCOM and hiddenWindow is disabled … So the only solution to build a tool which takes website screenshots with <canvas> was using xulrunner!

XUL Profiler

addon_thumb.png XUL Profiler ? Qu’est-ce donc ? Et bien c’est l’extension Firefox que j’ai pu développer chez Yoono. Elle a pour but de donner des pistes aux développeurs XUL (mais aussi aux développeurs Web) pour optimiser les performances de leurs applications.

Pour l’instant, cette extension permet de récolter deux informations :

  • un callgraph Javascript : Chaque appel de fonction est consigné dans un arbre et classé par son temps d’exécution. On peut ainsi rapidement repérer les fonctions qui ralentissent le navigateur.
  • une vidéo des rafraichissements de Firefox : Toutes les opérations de mise à jour graphique de firefox sont enregistrées dans une vidéo qui nous permet d’apprendre à optimiser notre Javascript ainsi que les CSS afin de soulager Firefox dans son travail de layout.

L’extension est disponible sur Mozilla addons

Voici quelques résultats sur des exemples simples.

Callgraph Javascript

(Fichier html de test)

function fun_root() {
  fun_A();
  fun_B();
  fun_C();
}
function fun_A() {
  dump("fun A");
}
function fun_B() {
  dump("fun B");
  var s="";
  fun_D();
  for(var i=0; i<1000; i++) {
    s+="CPU INTENSIVE FUNCTION";
    fun_D();
  }
  fun_D();
}
function fun_C() {
  dump("fun C");
}
function fun_D() {
  dump("fun D");
}

test-xulprofiler-callgraph.png

On voit ici la hiérarchie des appels entre les fonction grâce à la présentation sous forme d’arbre, et l’on peut conclure que la majorité du temps de calcul de ce script est effectué dans la fonction "fun_B".

Paint events

(Fichier html de test)

function delayEachInserts() {
  for(var i=0;i<20;i++) {
    window.setTimeout(insertItem,100*i,i);
  }
}
function insertItem(i) {
  var container=document.getElementById("container");
  var item=document.createElement("div");
  item.setAttribute("class","item");
  item.textContent="Item "+i;
  container.appendChild(item);
}
window.addEventListener("load",delayEachInserts,false);

  => Résutat

Cet exemple montre que lorsqu’on ajoute un élement DOM, Firefox est obligé de rafraichir son conteneur à chaque ajout.

Ocaml Native Code Debugging

t-caml-valid-callgraph.png Maintenant que le bug Improve gnu ELF est corrigé, kcachegrind nous génère de beaux callgraphs.

Ce patch consistait à ajouter des instructions .size (dans l’assembleur ELF) pour que valgrind interprète tous les symboles (camT_entry, camlT_foo, camlT_bar, …) et puisse ainsi afficher leurs noms au lieu d’un nombre hexadécimal!

Les standards ELF pour le debug

Maintenant, nous souhaiterions que ces outils puissent savoir de manière standard dans quels fichiers et à quelles lignes sont déclarés les fonctions. (Le nom du fichier est bien présent dans les symboles, mais cela ne permet pas d’exploiter le plein potentiel de ces outils)

Voyons un peu comment se débrouille GCC :

  int bar(int a) {
        return 1+a;
  }
  
  int foo(int a) {
        return 2+bar(a);
  }
  
  int main() {
        foo(3);
  }
  $ gcc -O0 -g -S t.c
  .globl bar
        .type   bar, @function
  bar:
  .LFB2:
        .file 1 "t.c"
        .loc 1 1 0
        pushq   %rbp
  .LCFI0:
        movq    %rsp, %rbp
  .LCFI1:
        movl    %edi, -4(%rbp)
        .loc 1 2 0
        movl    -4(%rbp), %eax
        incl    %eax
        .loc 1 3 0
        leave
        ret
  .LFE2:
        .size   bar, .-bar
  .globl foo
        .type   foo, @function

Les instructions clés sont .file et .loc :

  • .file déclare un chemin vers un fichier et l’associe à un identifiant

-> .file $identifiant$ "$chemin_du_fichier"

  • .loc associe un fichier, un numéro de ligne et de colonne aux instructions la succédant

-> .loc $identifiant_de_fichier$ $ligne$ $colonne$

Solution proposée

Le module du compilateur qui génère les instructions est : asmcomp/i386/emit.mlp
Et plus particulièrement la fonction fundecl :

  let fundecl fundecl =
    function_name := fundecl.fun_name;
    fastcode_flag := fundecl.fun_fast;
    (* ... *)
    `   .globl  {emit_symbol fundecl.fun_name}
`;
    `{emit_symbol fundecl.fun_name}:
`;
    if !Clflags.gprofile then emit_profile();
    let n = frame_size() - 4 in
    if n > 0 then
      ` subl    ${emit_int n}, %esp
`;
    `{emit_label !tailrec_entry_point}:
`;
    emit_all true fundecl.fun_body;
    List.iter emit_call_gc !call_gc_sites;
    emit_call_bound_errors ();
    List.iter emit_float_constant !float_constants;
    match Config.system with
      "linux_elf" | "bsd_elf" | "gnu" ->
        `       .type   {emit_symbol fundecl.fun_name},@function
`;
        `       .size   {emit_symbol fundecl.fun_name},.-{emit_symbol fundecl.fun_name}
`
    | _ -> ()

Hors nous n’avons à disposition que la variable fundecl :

  type fundecl = 
  { fun_name: string;
    fun_body: instruction;
    fun_fast: bool } 
  type instruction =
  { mutable desc: instruction_desc;
    mutable next: instruction;
    arg: Reg.t array;
    res: Reg.t array;
    dbg: Debuginfo.t;
    live: Reg.Set.t }

Il y a bien un attribut dbg sur les instructions mais il est rarement renseigné. (une compilation avec l’option -dlinear permet de le voir)

J’ai décidé d’ajouter un attribut fun_dbg : Debuginfo.t sur fundecl et de le remplir dans toutes les phases de compilation. Il serait peut être plus judicieux de travailler sur l’attribut dbg des instructions ? (car cela permettrait par la suite d’indiquer les lignes de chaque instruction à valgrind, mais aussi gdb!) Ce patch n’est pas optimisé car il répète l’instruction .file pour chaque .loc, donc à chaque entête de fonction, nous avons un .file et un .loc.

-> Patch sur la branche release311

Voyons maintenant ce qu’apporte ce patch.

Résultats avec gdb

  $ ocamlopt -g -inline 0 t.ml
  $ gdb a.out
  (gdb) break t.ml:6
  Breakpoint 1 at 0x8049940: file t.ml, line 6.
  (gdb) run
  Starting program: /home/alex/callgraph/a.out 
  
  Breakpoint 1, camlT__foo_60 () at t.ml:6
  6       let foo i  =
  Current language:  auto; currently asm

  (gdb) backtrace
  #0  camlT__foo_60 () at t.ml:7
  #1  0x0804c570 in camlT__entry () at t.ml:12
  #2  0x0806e4b7 in caml_start_program ()
  
  (gdb) step 1
  camlT__bar_58 () at t.ml:2
  2       let bar i =
  
  (gdb) list
  1
  2       let bar i =
  3               1+i
  4       ;;
  5
  6       let foo i  =
  7               2+(bar i )
  8       ;;
  9
  10      let () =

Résultats avec gprof

  $ ocamlopt -g -p -inline 0 t.ml
  $ ./a.out
  $ gprof -A
  *** File /home/alex/callgraph/t.ml:
                  
           1 -> let bar i =
                        Thread.delay 3.0;
                        1+i
                ;;
                
           1 -> let foo i  =
                        2+(bar i )
                ;;
                
                let () =
           1 ->         let closure() = 3 in
                        print_int ( foo (closure()) )
                ;;
                
  Top 10 Lines:

     Line      Count
        2          1
        7          1
       12          1

  Execution Summary:

        3   Executable lines in this file
        3   Lines executed
   100.00   Percent of the file executed

        3   Total number of line executions
     1.00   Average executions per line

Résultats avec valgrind/kcachegrind

  $ ocamlopt -g -inline 0 t.ml
  $ valgrind --tool=callgrind ./a.out
  $ callgrind_annotate callgrind.out.2152 t.ml
  --------------------------------------------------------------------------------
  -- User-annotated source: t.ml
  --------------------------------------------------------------------------------
  .  
  8  let bar i =
  77,715  => thread.ml:camlThread__delay_75 (1x)
  .      Thread.delay 3.0;
  .      1+i
  .  ;;
  .  
  3  let foo i  = 
  77,723  => t.ml:camlT__bar_58 (1x)
  .      2+(bar i )
  .  ;;
  .  
  .  let () =
  13          let closure() = 3 in
  1,692  => pervasives.ml:camlPervasives__output_string_215 (1x)
  2,312  => pervasives.ml:camlPervasives__string_of_int_154 (1x)
  77,726  => t.ml:camlT__foo_60 (1x)
  .      print_int ( foo (closure()) )
  .  ;;
  .  
  $ kcachegrind callgrind.out.2152

kcachegrind-file-and-line.png



Et après ?

On peut espérer encore un tas d’améliorations, comme :

  • des breakpoints sur n’importe quelle ligne de caml …
  • un pluging gdb pour pouvoir lire des valeurs pendant un break!