Hello, I'm trying to migrate from Windows to Linux as a Java development platform, and while the transition has generally been pretty painless, there are a few points of uncertainty that I'd like some feedback on. I'm running openSUSE 11.4, but I'm open to hear what works on other distros.

1. Where do you install your JDK from? This one is surprisingly not as cut and dry as most people make it out to be. OpenJDk 6 is available in the openSUSE repositories, and was very easy to install. However it's currently update 21, and right now the Oracle release is at update 24. I'm used to a little alert in Windows notifying me that my Java needs updating but that doesn't appear to be the norm in Linux. Do Java developers forgo the JDK in their package manager and install the binary directly? Or is there another way?
2. Where do you install Eclipse? There seems to be a general agreement online that Eclipse is best installed by simply downloading the binary and extracting it somewhere, but where's the usual place I would extract a program like Eclipse or Ant? I've seen votes for /usr/local and /opt online, but no definitive answer.
3. Where do you put your Jetty/Tomcat? Similar to the eclipse question, where do most Linux Java developers put their Jetty/Tomcat/other container.
4. What are some of the differences between the way you setup development versus production At the very least it seems I don't want to run my servlet container as root, that makes sense to me. But what other practices should I watch out for? Is there anything else that could make my development environment easier, but perhaps less secure?

I found this question was similar but ultimately too high level and didn't get into details of how actual developers are setting up their environment. If there's other resources you feel answer these questions, please share them here.

Thanks for your time.

Q> Where do you install your JDK from?
A> I never bother with other JDKs coming from outside Sun/Oracle mainly because our product is only certified to work with Sun/Oracle JRE. On my desktop, I run Kubuntu, but I never use apt-get for this but always download them manually. Reasons:

• distro maintainers rarely rush to upgrade packages, as their primary concern is to make dependant apps (such as OpenOffice) work. If JDK changes from 1.6.0_20 to 1.6.0_21, they simply don't care. I might do because a newer patch might have an important bugfix or I simply want to try if my app still passes all the unit tests.
• it might be a nightmare to retain old JDK versions. We still support older versions of our product and if I upgrade to a newer Kubuntu, I don't have guarantees that some ancient JDK will still be available as a package.
• I am not sure some distros even support multiple existence of JDKs on the same machine.

My preference is to keep all JDKs/JREs in /opt and make a symlink to the newest one or the one I need most. I simply don't see why installing JDK manually is a problem.

I also set the PATH to the newest JDK/JRE.

Same thing (and similar arguments) apply to Ant and Maven.

Q> Where do you install Eclipse?
A> I use IntelliJ but the same applies. I store IDE in my home folder. This allows me to have different versions of it, update them without needing sudo, etc. I could as well install it in /opt but I guess I got this habit when I was downloading and testing newest IntelliJ IDEA EAP every week so I can quickly delete the older versions and do not pollute /opt. Finally, other programs might require Ant/Maven/JDK but it's only me who uses IntelliJ hence the different approach.

Q> Where do you put your Jetty/Tomcat?
A> I have a separate folder tomcats under /home where I have ~10 different Tomcat instances. Each of Tomcats is used for a different version of my app (we bundle Tomcat with our app). This is necessary because one deployment of our app can have different Tomcat settings (or even version) than another.

Q> What are some of the differences between the way you setup development versus production
A> It very much depends on your app. For example, we need some partitions to have lower access latencies but having less space (e.g. gigabytes for Lucene indexes) VS others which can have higher latencies but require more space (e.g. terabytes for content repositories). We, however, design our app so that all these different aspects can reside on different partitions which are configurable. Some partitions need to have special limitations (e.g. file upload) so this doesn't overflow other partitions. There is no simple one-for-all answer to this question, but obviously most of these concerns don't matter that much for a development environment.

When a process requests for memory and an operating system is giving some new pages to the process, the kernel should initialize the pages (with zeros for instance) in order to avoid showing potentially confident data that another process used. The same when a process is starting and receives some memory, for example the stack segment.

When I execute the following code in Linux, the result is that the majority of allocated memory is indeed 0, but something about 3-4 kB at the bottom of the stack (the last elements of the array, the highest addresses) contains random numbers.

#include <cstdlib>
#include <iostream>
using namespace std;

int main()
{
    int * a = (int*)alloca(sizeof(int)*2000000);
    for(int i = 0; i< 2000000; ++i)
        cout << a[i] << endl;
    return 0;
}

1. Why isn't it set to zero too?
2. Could it be because it is being reused by the process?
3. If yes, could it be the initialization code that had used those 3-4 kB of memory earlier?

I am pretty sure that when the OS starts your process, the stack is only zeros. What you observe is another phenomenon, I think. You seem to have compiled your program as C++. C++ does a lot of code (constructors and stuff like that) before your main starts. So what you see are the left over values of your own execution.

If you'd compile your code as C (change to "stdio.h" etc) you'd probably see a much reduced "pollution" if not even none at all. In particular if you'd link your program statically to a minimalist version of a C library.

This is in C++ on CentOS 64bit using G++ 4.1.2.

We're writing a test application to load up the memory usage on a system by n Gigabytes. The idea being that the overall system load gets monitored through SNMP etc. So this is just a way of exercising the monitoring.

What we've seen however is that simply doing:

char* p = new char[1000000000];

doesn't affect the memory used as shown in either top or free -m

The memory allocation only seems to become "real" once the memory is written to:

memcpy(p, 'a', 1000000000);   //shows an increase in mem usage of 1GB

But we have to write to all of the memory, simply writing to the first element does not show an increase in the used memory:

p[0] = 'a';    //does not show an increase of 1GB.

Is this normal, has the memory actually been allocated fully? I'm not sure if it's the tools we are using (top and free -m) that are displaying incorrect values or whether there is something clever going on in the compiler or in the runtime and/or kernel.

This behavior is seen even in a debug build with optimizations turned off.

It was my understanding that a new[] allocated the memory immediately. Does the C++ runtime delay this actual allocation until later on when it is accessed. In that case can an out of memory exception be deferred until well after the actual allocation of the memory until the memory is accessed?

As it is it is not a problem for us, but it would be nice to know why this is occurring the way it is!

Cheers!

Edit:

I don't want to know about how we should be using Vectors, this isn't OO / C++ / the current way of doing things etc etc. I just want to know why this is happening the way it is, rather than have suggestions for alternative ways of trying it.

When your library allocates memory from the OS, the OS will just reserve an address range in the process's virtual address space. There's no reason for the OS to actually provide this memory until you use it - as you demonstrated.

If you look at e.g. /proc/self/maps you'll see the address range. If you look at top's memory use you won't see it - you're not using it yet.

I understand that each CPU/architecture has it's own instruction set, therefore a program(binary) written for a specific CPU cannot run on another. But what i don't really understand is why an executable file (binary like .exe for instance) cannot run on Linux but can run on windows even on the very same machine.

This is a basic question, and the answer i'm expecting is that .exe and other binary formats are probably not Raw machine instructions but they contain some data that is operating system dependent. If this is true, then what this OS dependent data is like? and as an example what is the format of an .exe file and the difference between it and Linux executables?

Is there a source i can get brief and detailed information about this?

In order to do something meaningful, applications will need to interface with the OS. Since system calls and user-space infrastructure look fundamentally different on Windows and Unix/Linux, having different formats for executable programs is the smallest trouble. It's the program logic that would need to be changed.

(You might argue that this is meaningless if you have a program that solely depends on standardized components, for example the C runtime library. This is theoretically true - but irrelevant for most applications since they are forced to use OS-dependent stuff).

The other differences between Windows PE (EXE,DLL,..) files and Linux ELF binaries are related to the different image loaders and some design characteristics of both OSs. For example on Linux a separate program is used to resolve external library imports while this functionality is built-in on Windows. Another example: Linux shared libraries function differently than DLLs on Windows. Not to mention that both formats are optimized to enable the respective OS kernels to load programs as quick as possible.

Emulators like Wine try to fill the gap (and actually prove that the biggest problem is not the binary format but rather the OS interface!).

It's not clear to me what encodings are used where in C's argv. In particular, I'm interested in the following scenario:

• A user uses locale L1 to create a file whose name, N, contains non-ASCII characters
• Later on, a user uses locale L2 to tab-complete the name of that file on the command line, which is fed into a program P as a command line argument

What sequence of bytes does P see on the command line?

I have observed that on Linux, creating a filename in the UTF-8 locale and then tab-completing it in (e.g.) the zw_TW.big5 locale seems to cause my program P to be fed UTF-8 rather than Big5. However, on OS X the same series of actions results in my program P getting a Big5 encoded filename.

Here is what I think is going on so far (long, and I'm probably wrong and need to be corrected):

Windows

File names are stored on disk in some Unicode format. So Windows takes the name N, converts from L1 (the current code page) to a Unicode version of N we will call N1, and stores N1 on disk.

What I then assume happens is that when tab-completing later on, the name N1 is converted to locale L2 (the new current code page) for display. With luck, this will yield the original name N -- but this won't be true if N contained characters unrepresentable in L2. We call the new name N2.

When the user actually presses enter to run P with that argument, the name N2 is converted back into unicode, yielding N1 again. This N1 is now available to the program in UCS2 format via GetCommandLineW/wmain/tmain, but users of GetCommandLine/main will see the name N2 in the current locale (code page).

OS X

The disk-storage story is the same, as far as I know. OS X stores file names as Unicode.

With a unicode terminal, I think happens is that the terminal builds the command line in a Unicode buffer. So when you tab complete, it copies the file name as a Unicode file name to that buffer.

When you run the command, that unicode buffer is converted to the current locale, L2, and fed to the program via argv, and the program can decode argv with the current locale into Unicode for display.

Linux

On Linux, everything is different and I'm extra-confused about what is going on. Linux stores file names as byte strings, not in Unicode. So if you create a file with name N in locale L1 that N as a byte string is what is stored on disk.

When I later run the terminal and try and tab-complete the name, I'm not sure what happens. It looks to me like the command line is constructed as a byte buffer, and the name of the file as a byte string is just concatenated onto that buffer. I assume that when you type a standard character it is encoded on the fly to bytes that are appended to that buffer.

When you run a program, I think that buffer is sent directly to argv. Now, what encoding does argv have? It looks like any characters you typed in the command line while in locale L2 will be in the L2 encoding, but the file name will be in the L1 encoding. So argv contains a mixture of two encodings!

Question

I'd really like it if someone could let me know what is going on here. All I have at the moment is half-guesses and speculation, and it doesn't really fit together. What I'd really like to be true is for argv to be encoded in the current code page (Windows) or the current locale (Linux / OS X) but that doesn't seem to be the case...

Extras

Here is a simple candidate program P that lets you observe encodings for yourself:

#include <stdio.h>

int main(int argc, char **argv)
{
    if (argc < 2) {
        printf("Not enough arguments\n");
        return 1;
    }

    int len = 0;
    for (char *c = argv[1]; *c; c++, len++) {
        printf("%d ", (int)(*c));
    }

    printf("\nLength: %d\n", len);

    return 0;
}

You can use locale -a to see available locales, and use export LC_ALL=my_encoding to change your locale.

Thanks everyone for your responses. I have learnt quite a lot about this issue and have discovered the following things that has resolved my question:

1. As discussed, on Windows the argv is encoded using the current code page. However, you can retrieve the command line as UTF-16 using GetCommandLineW. Use of argv is not recommended for modern Windows apps with unicode support because code pages are deprecated.

2. On Unixes, the argv has no fixed encoding:

   a) File names inserted by tab-completion/globbing will occur in argv verbatim as exactly the byte sequences by which they are named on disk. This is true even if those byte sequences make no sense in the current locale.

   b) Input entered directly by the user using their IME will occur in argv in the locale encoding (Ubuntu seems to use LOCALE to decide how to encode IME input, whereas OS X uses the Terminal.app encoding Preference)

This is annoying for languages such as Python or Haskell which wants to treat command line arguments as Strings, because it needs to decide how to decode argv into whatever encoding is used internally for String (which is UTF-16 for both of those languages). However, if they just use the locale encoding to do this decoding then valid filenames in the input may fail to decode, causing an exception.

The solution to this problem adopted by Python 3 is a surrogate byte encoding scheme (http://www.python.org/dev/peps/pep-0383/) which represents any undecodable byte in argv as special Unicode code points. When that code point is decoded back to a byte stream, it just becomes the original byte again. This allows for roundtripping data from argv that is not valid in the current encoding (i.e. a filename named in something other than the current locale) through the native Python string type and back to bytes with no loss of information.

As you can see, the situation is pretty messy :-)

I've started using vimdiff today, and wanted to do some of the things that I've taken for granted on Windows based diff editors (like expand/collapse a diff section, have full file expansion/only diffs with 3 context lines above or below). I currently know only the following commands :

Keyboard Shortcuts:

do - Get changes from other window into the current window.

dp - Put the changes from current window into the other window.

]c - Jump to the next change.

[c - Jump to the previous change.

Ctrl W + Ctrl W - Switch to the other split window.

Could someone point to a good link that has all the frequently used commands (not the manpage - I'm looking for users' opinions, not the full list of capabilities) ? It would be nice if I could expand/collapse lines around diffs, for example....

Aside from the ones you mention, I only use frequently when diffing the following:

• :diffupdate :diffu -> recalculate the diff, useful when after making several changes vim's isn't showing minimal changes anymore.
• :set noscrollbind -> temporarily disable simultaneous scrolling on both buffers, reenable by :set scrollbind and scrolling.

Most of what you asked for is folding: vim user manual's chapter on folding. Outside of diffs I sometime use:

• zo -> open fold.
• zc -> close fold.

But you'll probably be better served by:

• zr -> reducing folding level.
• zm -> one more folding level, please.

or even:

• zR -> Reduce completely the folding, I said!.
• zM -> fold Most!.

The other thing you asked for, use n lines of folding, can be found at the vim reference manual section on options, via the section on diff:

• set diffopt=<TAB>, then update or add context:n.

You should also take a look at the user manual section on diff.

I am using the function os.path.getsize() which gives the size of the file in bytes.

As my one file size is 10gb it give me size in negative(bytes).

so can anyone give me any idea why this happen?

This is my code:

import os
ospathsize = os.path.getsize('/home/user/Desktop/test1.nrg')
print (ospathsize) 

Your Linux kernel obviously has large file support, since ls -l works correctly. Thus, it's your Python installation that is lacking the support. (Are you using your distribution's Python package? What distribution is it?)

The documentation on POSIX large file support in Python states that Python should typically make use of large file support if it is available on Linux. It also suggests to try and configure Python with the command line

CFLAGS='-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64' OPT="-g -O2 $CFLAGS" \
    ./configure

And finally, quoting the man page of the stat system call:

This can occur when an application compiled on a 32-bit platform without -D_FILE_OFFSET_BITS=64 calls stat() on a file whose size exceeds (1<<31)-1 bits.

(I believe the last word should be "bytes".)

I have a portal in my university LAN where people can upload code to programming puzzles in C/C++. I would like to make the portal secure so that people cannot make system calls via their submitted code. There might be several workarounds but I'd like to know if I could do it simply by setting some clever gcc flags. libc by default seems to include <unistd.h>, which appears to be the basic file where system calls are declared. Is there a way I could tell gcc/g++ to 'ignore' this file at compile time so that none of the functions declared in unistd.h can be accessed?

Some particular reason why chroot("/var/jail/empty"); setuid(65534); isn't good enough (assuming 65534 has sensible limits)?

OK, this isn't really a question to achieve anything practical, but still it is a serious question and I hope it will be taken seriously and mods won't punish me for this :)

I'm sure majority of you have seen some good action movie, where CIA or FBI or hackers or any other "pc nerds" are "retrieving some information" and when they actually show their screens/monitors/desktops, there is a lot of random data being displayed and it's just so thrilling :D

So, we're shooting a movie and I need to reconstruct such a scene. My OS is ubuntu 10.10.

I think i've read somewhere on the internet once that shell can actually be recorded and then played back, but I'm not sure how it worked.

Basically, any script/program/code/solution which does the trick is very well welcome.

If there's anyone who could come up with a solution, it would be so cool!

Let's make this fun, shall we?

BOUNTY EDIT: Still need some more ideas, so I'm offering a bounty for the best new upcoming idea.

There's a utility call script (ironically) that does what you're talking about. It can even record timing data so the playback is done at the same rate the original actions were performed.

To start recording and capture timing data:

$ script -t script.out 2>timing.out

When you're finished, enter exit.

To replay the recorded session including the original timing:

$ scriptreplay timing.out script.out

Edit:

You can simulate typing or slow dialup data transmission using the pv utility. The command below will output the file at 37 characters per second (roughly approximating a 300 baud modem).

pv -q -L 37 somefile

Here's another idea:

hexdump -C /dev/urandom | pv -q -L 1200

This gives Matrix-like output on the screen:

#!/bin/bash
printf "\e[32m\n"
while :
do
    for i in {1..16}
    do
        ((r = $RANDOM % 2))
        if (($RANDOM % 5 == 1))
        then
            if (($RANDOM % 4 == 1))
            then
                v+="\e[1m $r   "
            else
                v+="\e[2m $r   "
            fi
        else
            v+="     "
        fi
    done
    printf "$v\n"
    v=""
done

Suppose an application is blocked at a cancellation point, for example read, and a signal is received and a signal handler invoked. Glibc/NPTL implements cancellation points by enabling asynchronous cancellation for the duration of the syscall, so as far as I can tell, asynchronous cancellation will remain in effect for the entire duration of the signal handler. This would of course be horribly wrong, as there are plenty of functions that are not async-cancel-safe but which are required to be safe to call from signal handlers.

This leaves me with two questions:

• Am I wrong or is the glibc/NPTL behavior really this dangerously broken? If so, is such dangerous behavior conformant?
• What, according to POSIX, is supposed to happen if a signal handler is invoked while the process is executing a function which is a cancellation point?

Edit: I've almost convinced myself that any thread which is a potential target of pthread_cancel must ensure that functions which are cancellation points can never be called from a signal handler in that thread's context:

On the one hand, any signal handler that can be invoked in a thread that might be cancelled and which uses any async-cancel-unsafe functions must disable cancellation before calling any function which is a cancellation point. This is because, from the perspective of the code interrupted by the signal, any such cancellation would be equivalent to asynchronous cancellation. On the other hand, a signal handler cannot disable cancellation, unless the code that will be running when the signal handler is invoked only uses async-signal-safe functions, because pthread_setcancelstate is not async-signal-safe.

To answer the first half of my own question: glibc does exhibit the behavior I predicted. Signal handlers that run while blocked at a cancellation point run under asynchronous cancellation. To see this effect, simply create a thread that invokes a cancellation point that will block forever (or for a long time), wait a moment, send it a signal, wait a moment again, and cancel and join it. The signal handler should fiddle with some volatile variables in a way that makes it clear that it ran for an unpredictable amount of time before being terminated asynchronously.

As for whether POSIX allows this behavior, I'm still not 100% certain. POSIX states:

Whenever a thread has cancelability enabled and a cancellation request has been made with that thread as the target, and the thread then calls any function that is a cancellation point (such as pthread_testcancel() or read()), the cancellation request shall be acted upon before the function returns. If a thread has cancelability enabled and a cancellation request is made with the thread as a target while the thread is suspended at a cancellation point, the thread shall be awakened and the cancellation request shall be acted upon. It is unspecified whether the cancellation request is acted upon or whether the cancellation request remains pending and the thread resumes normal execution if:

• The thread is suspended at a cancellation point and the event for which it is waiting occurs

• A specified timeout expired

before the cancellation request is acted upon.

Presumably executing a signal handler is not a case of being "suspended", so I'm leaning towards interpreting glibc's behavior here as non-conformant.

Let's say I compiled the application below and stripped it's symbols.

#include <stdio.h>

int main()
{
    printf("Hello\n");
}

Build procedure:

gcc -o hello hello.c
strip --strip-unneeded hello

If the application wasn't stripped, disassembling the main function would be easy. However, I have no idea how to disassemble the main function of a stripped application.

(gdb) disas main
No symbol table is loaded.  Use the "file" command.

(gdb) info line main
Function "main" not defined.

How could I do it, gentlemen? Is it even possible?

Notes: this must be done with GDB only. Forget objdump. Assume that I don't have access to the code.

A step-by-step example would be greatly appreciated.

Hi there karlphillip,

Ok, here a big edition of my previous answer. I think I found a way now.

You (still :) have this specific problem:

(gdb) disas main
No symbol table is loaded.  Use the "file" command.

Now, if you compile the code (I added a return 0 at the end), you will get with gcc -S:

    pushq   %rbp
    movq    %rsp, %rbp
    movl    $.LC0, %edi
    call    puts
    movl    $0, %eax
    leave
    ret

Now, you can see that your binary gives you some info:

Striped:

(gdb) info files
Symbols from "/home/beco/Documents/fontes/cpp/teste/stackoverflow/distrip".
Local exec file:
    `/home/beco/Documents/fontes/cpp/teste/stackoverflow/distrip', file type elf64-x86-64.
    Entry point: 0x400440
    0x0000000000400238 - 0x0000000000400254 is .interp
    ...
    0x00000000004003a8 - 0x00000000004003c0 is .rela.dyn
    0x00000000004003c0 - 0x00000000004003f0 is .rela.plt
    0x00000000004003f0 - 0x0000000000400408 is .init
    0x0000000000400408 - 0x0000000000400438 is .plt
    0x0000000000400440 - 0x0000000000400618 is .text
    ...
    0x0000000000601010 - 0x0000000000601020 is .data
    0x0000000000601020 - 0x0000000000601030 is .bss

The most important entry here is .text. It is a common name for a assembly start of code, and from our explanation of main bellow, from its size, you can see that it includes main. If you disassembly it, you will see a call to __libc_start_main. Most important, you are disassembling a good entry point that is real code (you are not misleading to change DATA to CODE).

disas 0x0000000000400440,0x0000000000400618
Dump of assembler code from 0x400440 to 0x400618:
   0x0000000000400440:  xor    %ebp,%ebp
   0x0000000000400442:  mov    %rdx,%r9
   0x0000000000400445:  pop    %rsi
   0x0000000000400446:  mov    %rsp,%rdx
   0x0000000000400449:  and    $0xfffffffffffffff0,%rsp
   0x000000000040044d:  push   %rax
   0x000000000040044e:  push   %rsp
   0x000000000040044f:  mov    $0x400540,%r8
   0x0000000000400456:  mov    $0x400550,%rcx
   0x000000000040045d:  mov    $0x400524,%rdi
   0x0000000000400464:  callq  0x400428 <__libc_start_main@plt>
   0x0000000000400469:  hlt
   ...

   0x000000000040046c:  sub    $0x8,%rsp
   ...
   0x0000000000400482:  retq   
   0x0000000000400483:  nop
   ...
   0x0000000000400490:  push   %rbp
   ..
   0x00000000004004f2:  leaveq 
   0x00000000004004f3:  retq   
   0x00000000004004f4:  data32 data32 nopw %cs:0x0(%rax,%rax,1)
   ...
   0x000000000040051d:  leaveq 
   0x000000000040051e:  jmpq   *%rax
   ...
   0x0000000000400520:  leaveq 
   0x0000000000400521:  retq   
   0x0000000000400522:  nop
   0x0000000000400523:  nop
   0x0000000000400524:  push   %rbp
   0x0000000000400525:  mov    %rsp,%rbp
   0x0000000000400528:  mov    $0x40062c,%edi
   0x000000000040052d:  callq  0x400418 <puts@plt>
   0x0000000000400532:  mov    $0x0,%eax
   0x0000000000400537:  leaveq 
   0x0000000000400538:  retq   

The call to __libc_start_main gets as its first argument a pointer to main(). So, the last argument in the stack just immediately before the call is your main() address.

   0x000000000040045d:  mov    $0x400524,%rdi
   0x0000000000400464:  callq  0x400428 <__libc_start_main@plt>

Here it is 0x400524 (as we already know). Now you set a breakpoint an try this:

(gdb) break *0x400524
Breakpoint 1 at 0x400524
(gdb) run
Starting program: /home/beco/Documents/fontes/cpp/teste/stackoverflow/disassembly/d2 

Breakpoint 1, 0x0000000000400524 in main ()
(gdb) n
Single stepping until exit from function main, 
which has no line number information.
hello 1
__libc_start_main (main=<value optimized out>, argc=<value optimized out>, ubp_av=<value optimized out>, 
    init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, 
    stack_end=0x7fffffffdc38) at libc-start.c:258
258 libc-start.c: No such file or directory.
    in libc-start.c
(gdb) n

Program exited normally.
(gdb) 

Now you can disassembly it using:

(gdb) disas 0x0000000000400524,0x0000000000400600
Dump of assembler code from 0x400524 to 0x400600:
   0x0000000000400524:  push   %rbp
   0x0000000000400525:  mov    %rsp,%rbp
   0x0000000000400528:  sub    $0x10,%rsp
   0x000000000040052c:  movl   $0x1,-0x4(%rbp)
   0x0000000000400533:  mov    $0x40064c,%eax
   0x0000000000400538:  mov    -0x4(%rbp),%edx
   0x000000000040053b:  mov    %edx,%esi
   0x000000000040053d:  mov    %rax,%rdi
   0x0000000000400540:  mov    $0x0,%eax
   0x0000000000400545:  callq  0x400418 <printf@plt>
   0x000000000040054a:  mov    $0x0,%eax
   0x000000000040054f:  leaveq 
   0x0000000000400550:  retq   
   0x0000000000400551:  nop
   0x0000000000400552:  nop
   0x0000000000400553:  nop
   0x0000000000400554:  nop
   0x0000000000400555:  nop
   ...

This is primarily the solution.

BTW, this is a different code, to see if it works. That is why the assembly above is a bit different. The code above is from this c file:

#include <stdio.h>

int main(void)
{
    int i=1;
    printf("hello %d\n", i);
    return 0;
}

But!

---

if this does not work, then you still have some hints:

You should be looking to set breakpoints in the beginning of all functions from now on. They are just before a ret or leave. The first entry point is .text itself. This is the assembly start, but not the main.

The problem is that not always a breakpoint will let your program run. Like this one in the very .text:

(gdb) break *0x0000000000400440
Breakpoint 2 at 0x400440
(gdb) run
Starting program: /home/beco/Documents/fontes/cpp/teste/stackoverflow/disassembly/d2 

Breakpoint 2, 0x0000000000400440 in _start ()
(gdb) n
Single stepping until exit from function _start, 
which has no line number information.
0x0000000000400428 in __libc_start_main@plt ()
(gdb) n
Single stepping until exit from function __libc_start_main@plt, 
which has no line number information.
0x0000000000400408 in ?? ()
(gdb) n
Cannot find bounds of current function

So you need to keep trying until you find your way, setting breakpoints at:

0x400440
0x40046c
0x400490
0x4004f4
0x40051e
0x400524

---

From the other answer, we should keep this info:

In the non-striped version of the file, we see:

(gdb) disas main
Dump of assembler code for function main:
   0x0000000000400524 <+0>: push   %rbp
   0x0000000000400525 <+1>: mov    %rsp,%rbp
   0x0000000000400528 <+4>: mov    $0x40062c,%edi
   0x000000000040052d <+9>: callq  0x400418 <puts@plt>
   0x0000000000400532 <+14>:    mov    $0x0,%eax
   0x0000000000400537 <+19>:    leaveq 
   0x0000000000400538 <+20>:    retq   
End of assembler dump.

Now we know that main is at 0x0000000000400524,0x0000000000400539. If we use the same offset to look at the striped binary we get the same results:

(gdb) disas 0x0000000000400524,0x0000000000400539
Dump of assembler code from 0x400524 to 0x400539:
   0x0000000000400524:  push   %rbp
   0x0000000000400525:  mov    %rsp,%rbp
   0x0000000000400528:  mov    $0x40062c,%edi
   0x000000000040052d:  callq  0x400418 <puts@plt>
   0x0000000000400532:  mov    $0x0,%eax
   0x0000000000400537:  leaveq 
   0x0000000000400538:  retq   
End of assembler dump.

So, unless you can get some tip where the main starts (like using another code with symbols), another way is if you can have some info about the firsts assembly instructions, so you can disassembly at specifics places and look if it matches. If you have no access at all to the code, you still can read the ELF definition to understand how many sections should appear in the code and try a calculated address. Still, you need info about sections in the code!

That is hard work, my friend! Good luck!

Beco

It is quite easy to embed binary resources into PE images (EXE, DLL) via Windows API (refer to http://msdn.microsoft.com/en-us/library/ms648008(v=VS.85).aspx).

Is there any similar standard API in Linux?

or maybe some kind of de-facto approach to resource embedding?

The goal is to embed some static binary and/or textual data into executable, e.g. pictures, HTMLs, etc.. so that program binary distribution is as simple as making one file copy? (assuming all library dependencies are ok)

Update:

following bdk's suggestion, I've tried solution described in Embedding binary blobs using gcc mingw and it worked for me. Though, there were some issues that are worth mentioning: my project (in Code::Blocks) consists of a number of C++ files and adding binary data into any of corresponding object files rendered them useless breaking the build - objdump -x would show that most of the symbols have gone after embedding (and I didn't find how to fix it). To overcome this issue I added an empty dummy .cpp file into the project with the only purpose of providing an object file to play with and wrote the following custom build step for that file which did the job nicely (example uses Code::Blocks macros):

$compiler $options $includes -c $file -o $object
ld -Ur -b binary -o $object <binary payload path>

objcopy --add-section allows you to add an arbitrary file as a section in an ELF executable. (objcopy man page). However this is only half a solution, as I have not yet found a way to access this data from inside a C Program other than by loading and parsing the ELF Binary using an ELF Library.

Edit Additional Information:

If you have a compiled program called MyProgram and a resource file MyResource.dat which you want embedded into MyProgram, you can use the objcopy command like this:

objcopy MyProgram --add-section MyResource=MyResource.dat

Now if you look at your program using the command objdump -x MyProgram

You will see a section called MyResource which contains the contents of MyResource.dat. The file is now embedded inside of your executable.

The trick now, is how do you access the data from inside your program. My instinct tells me that the loader should place the file into memory somewhere and you should be able to get a pointer to it, however I'm not sure how to do that simply. Ideally I'd want to be able to dlopen my exeutable and dlsym the section, but that doesn't work because its a section and not a symbol.

The only alternative I know of to access the section from inside the program is to use the libelf library or something similar which is a little like using a sledgehammer to tap in a nail. You can use it in your application to load itself as an ELF Resource and retrieve the sections. Documentation is sparse, but here's an example

http://em386.blogspot.com/2007/03/quick-libelf-guide.html

I'd love if someone could chime in with an easier way to access the data from --add-section.

Edit 2 In My research I encoutered this question: Embedding binary blobs using gcc mingw

Which should work for gcc as well as mingw and shows a way to use ld instead of objcopy to add the data and be able to access it as a symbol. Looks promising

I have a password generator:

import random, string

def gen_pass():
    foo = random.SystemRandom()
    length = 64
    chars = string.letters + string.digits
    return ''.join(foo.choice(chars) for _ in xrange(length))

According to the docs, SystemRandom uses os.urandom which uses /dev/urandom to throw out random cryto bits. In Linux you can get random bits from /dev/urandom or /dev/random, they both use whatever entropy the kernel can get its hands on. The amount of entropy available can be checked with tail /proc/sys/kernel/random/entropy_avail, this will return a number like: 129. The higher the number more entropy is available. The difference between /dev/urandom and /dev/random is that /dev/random will only spit out bits if entropy_avail is high enough (like at least 60) and /dev/urandom will always spit out bits. The docs say that /dev/urandom is good for crypto and you only have to use /dev/random for ssl certs and the like.

My question is will gen_pass be good for making strong crypto grade passwords always? If I call this function as quickly as possible will I stop getting strong cryto bits at some point because the entropy pool is depleted?

The question could also be why does /dev/urandom always produce strong cryto bits and not care about the entropy_avail?

It is possible that /dev/urandom is designed so that its bandwidth is capped by the number of cycles you can guess will be correlated with an amount of entropy, but this is speculation and I can't find an answer.

Also this is my first stackoverflow question so please critique me. I am concerned that I gave to much background when someone who knows the answer probably knows the background.

Thanks

update

I wrote some code to look at the entropy pool while the /dev/urandom was being read from:

import subprocess
import time

from pygooglechart import Chart
from pygooglechart import SimpleLineChart
from pygooglechart import Axis

def check_entropy():
    arg = ['cat', '/proc/sys/kernel/random/entropy_avail']
    ps = subprocess.Popen(arg,stdout=subprocess.PIPE)
    return int(ps.communicate()[0])

def run(number_of_tests,resolution,entropy = []):
    i = 0
    while i < number_of_tests:        
        time.sleep(resolution)
        entropy += [check_entropy()]
        i += 1
    graph(entropy,int(number_of_tests*resolution))

def graph(entropy,rng):    
    max_y = 200    
    chart = SimpleLineChart(600, 375, y_range=[0, max_y])
    chart.add_data(entropy)
    char t.set_colours(['0000FF'])
    left_axis = range(0, max_y + 1, 32)
    left_axis[0] = 'entropy'
    chart.set_axis_labels(Axis.LEFT, left_axis)    
    chart.set_axis_labels(Axis.BOTTOM,['time in second']+get_x_axis(rng))
    chart.download('line-stripes.png')

def get_x_axis(rng):
    global modnum        
    if len(filter(lambda x:x%modnum == 0,range(rng + 1)[1:])) > 10:
        modnum += 1
        return get_x_axis(rng)
    return filter(lambda x:x%modnum == 0,range(rng + 1)[1:])

modnum = 1
run(500,.1)

If run this and also run:

while 1 > 0:
    gen_pass()

Then I pretty reliablly get a graph that looks like this:

Making the graph while running cat /dev/urandom looks smiler and cat /dev/random drops off to nothing and stays low very quickly (this also only reads out like a byte every 3 seconds or so)

update

If I run the same test but with six instances of gen_pass(), I get this:

So it looks like something is making it be the case that I have enough entropy. I should measure the password generation rate and make sure that it is actually being capped, because if it is not then something fishy may be going on.

update

I found this email chain

This says that urandom will stop pulling entropy once the pool only has 128 bits in it. This is very consistent with the above results and means that in those tests I am producing junk passwords often.

My assumption before was that if the entropy_avail was high enough (say above 64 bits) then /dev/urnadom output was good. This is not the case it seems that /dev/urandom was designed to leave extra entropy for /dev/random in case it needs it.

Now I need to find out how many true random bits a SystemRandom call needs.

There's a subtle difference between the output of /dev/random and /dev/urandom. As has been pointed out, /dev/urandom doesn't block. That's because it gets its output from a pseudo-random number generator, seeded from the 'real' random numbers in /dev/random.

The output of /dev/urandom will almost always be sufficiently random -- it's a high-quality PRNG with a random seed. If you really need a better source of random data, you could consider getting a system with a hardware random number generator -- my netbook has a VIA C7 in it, which can generate quite a lot of properly random data (I get a consistent 99.9kb/s out of /dev/random, 545kb/s out of /dev/urandom).

As an aside, if you're generating passwords then you might want to look at pwgen -- it makes nice pronounceable passwords for you :).

What is a core dump file in linux? What all information does it provide?

It's basically the process address space in use (from the mm_struct structure which contains all the virtual memory areas), and any other supporting information^*a, at the time it crashed.

For example, let's say you try to dereference a NULL pointer and receive a SEGV signal, causing you to exit. As part of that process, the operating system tries to write your information to a file for later post-mortem analysis.

You can load the core file into a debugger along with the executable file (for symbols and other debugging information, for example) and poke around to try and discover what caused the problem.

---

^*a: in kernel version 2.6.38, fs/exec.c/do_coredump() is the one responsible for core dumps and you can see that it's passed the signal number, exit code and registers. It in turn passes the signal number and registers to a binary-format-specific (ELF, a.out, etc) dumper.

The ELF dumper is fs/binfmt_elf.c/elf_core_dump() and you can see that it outputs non-memory-based information, like thread details, in fs/binfmt_elf.c/fill_note_info(), then returns to output the process space.

im writing a server in c++ that will handle safe connections where sensitive data will be sent.

the goal is never saving the data in unencrypted form anywhere outside memory, and keeping it at a defined space in the memory (to be overwritten after its no longer needed)

will allocating a large chunk of memory and using it to store the sensitive data be sufficient and ensure that there is no leakage of data ?

From the manual of a tool that handles passwords:

It's also debatable whether mlock() is a proper way to protect sensitive information. According to POSIX, mlock()-ing a page guarantees that it is in memory (useful for realtime applications), not that it isn't in the swap (useful for security applications). Possibly an encrypted swap partition (or no swap partition) is a better solution.

However, Linux does guarantee that it is not in the swap and specifically discusses the security applications. It also mentions:

But be aware that the suspend mode on laptops and some desktop computers will save a copy of the system's RAM to disk, regardless of memory locks.

I currently need to provide multiple keyboard interrupts for a program. Is there an easy way to do this with the signal class? I currently use the SIGINT/Ctrl+C but I can't find any other keyboard mappings.

Would be nice to have more than 2 signals. How can I either define more signals or is there a better way to capture an "interrupt from a user"?

Here is a highlevel view of the current code:

 def shutdown(signal, frame):
       if(signal==2): #sigint
          print 'do something'
       elif signal==XX:
          print 'do something else'
       # continued...

 signal.signal(signal.SIGINT, shutdown)
 signal.signal(signal.SOMEOTHERTYPE, shutdown)


 print 'start'
 t = Thread(target=run)
 t.setDaemon(True)
 t.start()

 print 'Done, press ctrl c, or ctrl ? '
 signal.pause()

The Ctrl+\ that has been mentioned is interpreted by your terminal software, and the key binding is configured through stty. Unless you have some way of customizing your terminal software you'll only be able to use the few signals that are already built in.

Depending on how much functionality you need or how far you want to take it, another option is to write your own simple "process execution terminal". This would be a script that executes an app for you and places your terminal in raw mode so that it can process keystrokes which perform custom actions.

Below is an oversimplified example showing what I mean. You could also do something similar via curses or urwid if you like.

To handle process output you'd need to capture the stdout/stderr of and display it nicely to the screen, using ANSI escape sequences if you are manipulating the terminal manually, or using an urwid widget to display the output in a scrolling window, etc. The same idea would also extend to other GUI systems (wx, tkinter, etc) but terminal control was mentioned.

Here is term.py which implements a basic raw terminal interpreter:

import os, signal, subprocess, sys, tty, termios

sigmap = {
    '\x15': signal.SIGUSR1,     # ctrl-u
    '\x1c': signal.SIGQUIT,     # ctrl-\
    '\x08': signal.SIGHUP,      # ctrl-h
    '\x09': signal.SIGINT,      # ctrl-i
    }
# setup tty
fd = sys.stdin.fileno()
old_tc = termios.tcgetattr(fd)
tty.setraw(fd)
# spawn command as a child proc
cmd = sys.argv[1:]
proc = subprocess.Popen(cmd)
while 1:
    try:
        ch = sys.stdin.read(1)
        # example of ansi escape to move cursor down and to column 0
        print '\033[1Eyou entered', repr(ch)
        if ch == 'q':
            break
        signum = sigmap.get(ch)
        if signum:
            os.kill(proc.pid, signum)
    finally:
        pass
termios.tcsetattr(fd, termios.TCSANOW, old_tc)
sys.exit()

Here is a simple target.py script to spin and print the signals it receives:

import signal, sys, time

def handler(num, _):
    print 'got:', sigmap.get(num, '<other>')
    if num == signal.SIGINT:
        sys.exit(1)
    return 1

signames = ['SIGINT','SIGHUP','SIGQUIT','SIGUSR1']
sigmap = dict((getattr(signal, k), k) for k in signames)
for name in signames:
    signal.signal(getattr(signal, name), handler)
while 1:
    time.sleep(1)

Usage example:

% python term.py python target.py
you entered 'h'
you entered 'i'
you entered '\x1c'
                  got: SIGQUIT
you entered '\x15'
                  got: SIGUSR1
you entered '\x08'
                  got: SIGHUP
you entered '\t'
                got: SIGINT
you entered 'q'