I've always been amazed/frustrated with how long it takes to simply output to the terminal with a print statement. After some recent painfully slow logging I decided to look into it and was quite surprised to find that almost all the time spent is waiting for the terminal to process the results.

Can writing to stdout be sped up somehow?

I wrote a script ('print_timer.py' at the bottom of this question) to compare timing when writing 100k lines to stdout, to file, and with stdout redirected to /dev/null. Here is the timing result:

$python print_timer.py
this is a test
this is a test
<snipped 99997 lines>
this is a test
-----
timing summary (100k lines each)
-----
print                         :11.950 s
write to file (+ fsync)       : 0.122 s
print with stdout = /dev/null : 0.050 s

Wow. To make sure python isn't doing something behind the scenes like recognizing that I reassigned stdout to /dev/null or something, I did the redirection outside the script...

$ python print_timer.py > /dev/null
-----
timing summary (100k lines each)
-----
print                         : 0.053 s
write to file (+fsync)        : 0.108 s
print with stdout = /dev/null : 0.045 s

So it isn't a python trick, it is just the terminal. I always knew dumping output to /dev/null sped things up, but never figured it was that significant!

It amazes me how slow the tty is. How can it be that writing to physical disk is WAY faster than writing to the "screen" (presumably an all-RAM op), and is effectively as fast as simply dumping to the garbage with /dev/null?

This link talks about how the terminal will block I/O so it can "parse [the input], update its frame buffer, communicate with the X server in order to scroll the window and so on"... but I don't fully get it. What can be taking so long?

I expect there is no way out (short of a faster tty implementation?) but figure I'd ask anyway.

---

UPDATE: after reading some comments I wondered how much impact my screen size actually has on the print time, and it does have some significance. The really slow numbers above are with my Gnome terminal blown up to 1920x1200. If I reduce it very small I get...

-----
timing summary (100k lines each)
-----
print                         : 2.920 s
write to file (+fsync)        : 0.121 s
print with stdout = /dev/null : 0.048 s

That is certainly better (~4x), but doesn't change my question. It only adds to my question as I don't understand why the terminal screen rendering should slow down an application writing to stdout. Why does my program need to wait for screen rendering to continue?

Are all terminal/tty apps not created equal? I have yet to experiment. It really seems to me like a terminal should be able to buffer all incoming data, parse/render it invisibly, and only render the most recent chunk that is visible in the current screen configuration at a sensible frame rate. So if I can write+fsync to disk in ~0.1 seconds, a terminal should be able to complete the same operation in something of that order (with maybe a few screen updates while it did it).

I'm still kind of hoping there is a tty setting that can be changed from the application side to make this behaviour better for programmer. If this is strictly a terminal application issue, then this maybe doesn't even belong on StackOverflow?

What am I missing?

---

Here is the python program used to generate the timing:

import time, sys, tty
import os

lineCount = 100000
line = "this is a test"
summary = ""

cmd = "print"
startTime_s = time.time()
for x in range(lineCount):
    print line
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)

#Add a newline to match line outputs above...
line += "\n"

cmd = "write to file (+fsync)"
fp = file("out.txt", "w")
startTime_s = time.time()
for x in range(lineCount):
    fp.write(line)
os.fsync(fp.fileno())
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)

cmd = "print with stdout = /dev/null"
sys.stdout = file(os.devnull, "w")
startTime_s = time.time()
for x in range(lineCount):
    fp.write(line)
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)

print >> sys.stderr, "-----"
print >> sys.stderr, "timing summary (100k lines each)"
print >> sys.stderr, "-----"
print >> sys.stderr, summary

Thanks for all the comments! I've ended up answering it myself with your help. It feels dirty answering your own question, though.

Question 1: Why is printing to stdout slow?

Answer: Printing to stdout is not inherently slow. It is the terminal you work with that is slow. And it has pretty much zero to do with I/O buffering on the application side (eg: python file buffering). See below.

Question 2: Can it be sped up?

Answer: Yes it can, but seemingly not from the program side (the side doing the 'printing' to stdout). To speed it up, use a faster different terminal emulator.

Explanation...

I tried a self-described 'lightweight' terminal program called wterm and got significantly better results. Below is the output of my test script (at the bottom of the question) when running in wterm at 1920x1200 in on the same system where the basic print option took 12s using gnome-terminal:

-----
timing summary (100k lines each)
-----
print                         : 0.261 s
write to file (+fsync)        : 0.110 s
print with stdout = /dev/null : 0.050 s

0.26s is MUCH better than 12s! I don't know whether wterm is more intelligent about how it renders to screen along the lines of how I was suggesting (render the 'visible' tail at a reasonable frame rate), or whether it just "does less" than gnome-terminal. For the purposes of my question I've got the answer, though. gnome-terminal is slow.

So - If you have a long running script that you feel is slow and it spews massive amounts of text to stdout... try a different terminal and see if it is any better!

Note that I pretty much randomly pulled wterm from the ubuntu/debian repositories. This link might be the same terminal, but I'm not sure. I did not test any other terminal emulators.

---

Update: Because I had to scratch the itch, I tested a whole pile of other terminal emulators with the same script and full screen (1920x1200). My manually collected stats are here:

wterm           0.3s
aterm           0.3s
rxvt            0.3s
mrxvt           0.4s
konsole         0.6s
yakuake         0.7s
lxterminal        7s
xterm             9s
gnome-terminal   12s
xfce4-terminal   12s
vala-terminal    18s
xvt              48s

The recorded times are manually collected, but they were pretty consistent. I recorded the best(ish) value. YMMV, obviously.

As a bonus, it was an interesting tour of some of the various terminal emulators available out there! I'm amazed my first 'alternate' test turned out to be the best of the bunch.

I am working on a God script to monitor my Unicorns. I started with GitHub's examples script and have been modifying it to match my server configuration. Once God is running, commands such as god stop unicorn and god restart unicorn work just fine.

However, god start unicorn results in WARN: unicorn start command exited with non-zero code = 1. The weird part is that if I copy the start script directly from the config file, it starts right up like a brand new mustang.

This is my start command:

/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D

I have declared all paths as absolute in the config file. Any ideas what might be preventing this script from working?

I haven't used unicorn as an app server, but I've used god for monitoring before.

If I remember rightly when you start god and give your config file, it automatically starts whatever you've told it to watch. Unicorn is probably already running, which is why it's throwing the error.

Check this by running god status once you've started god. If that's not the case you can check on the command line what the comand's exit status is:

/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D; echo $?;

that echo will print the exit status of the last command. If it's zero, the last command reported no errors. Try starting unicorn twice in a row, I expect the second time it'll return 1, because it's already running.

I ran into this question while typing man 2 open. It says that there are two kinds of open, one with two args, and one with three! last time i checked we could not overload functions in C. How did they do this? did they write in C++?

int open(const char * pathname, int flags);
int open(const char * pathname, int flags, mode_t mode);

No, they just used variadic function.

int open(const char * pathname, int flags, ...);

This makes the last argument mode optional. The prototypes only show how the function should be used, not the actual interface.

Of course, unlike real overloading, the compiler cannot type-check the mode argument, so the user have to be extra careful to ensure only 2 or 3 arguments are passed, and the 3rd argument must be a mode_t.

---

BTW, if you check the man 2 open for BSD (including OS X) it shows the correct prototype as above.

I have to create a software that must work on several *nix platforms (Linux, AIX, ...).

I need to handle internationalization and my translation strings are in the following form:

"Hi %1, you are %2." // English
"Vous êtes %2, bonjour %1 !" // French

Here %1 stand for the name, and %2 for another word. I may change the format, that's not an issue.

I tried to use printf() but you cannot specify the order of the parameters, you just specify their types.

"Hi %s, you are %s"
"Vous êtes %s, bonjour %s !"

Now there is no way to know which parameter to use for replacement of %s: printf() just uses the first one, then the next.

Is there any alternative to printf() that deals with this ?

Note: gettext() is not an option.

POSIX printf() supports positional arguments.

printf("Hi %1$s, you are %2$s.", name, status);
printf("Vous êtes %2$s, bonjour %1$s !", name, status);

I have a php/mysql website with over 200,000 images in single folder (linux server). I don't think, that I will never need to see them in file explorer, instead they will be viewed on website on their individual pages. They are just displayed in product page on website. File system is ext3. so is it wise to save them in single folder? can it slow down the site's performance? I need your expert advise

Ext3 uses tree to hold directory contents, so it's capability to handle large amount of files in single directory is better than of those file systems with linear directory listings. Here you can read the description of the tree used to keep directory contents.

However, 200K files is still a huge number. It's reasonable to move them into subdirectories based on first n characters of file names. This approach lets you keep only file names and not directory names, and when you need to access the file, you know where (in which subdirectory) to look for it.

My webservers use the usual Java I/O with thread per connection mechanism. Nowadays, they are getting on their knees with increased user (long polling connection). However, the connections are mostly idle. While this can be solved by adding more webservers, I have been trying to do some research on the NIO implementation.

I got a mixed impression about it. I have read about benchmarks where regular I/O with the new NPTL library in Linux outperforms NIO.

What is the real life experience of configuring and using the latest NPTL for Linux with Java I/O? Is there any increased performance?

And on a larger scope question:

What is the maximum number of I/O and blocking threads (that we configure in the Tomcat thread pool) in a standard server class machine (Dell with a quad-core processor) we expect to perform normally (with Linux NPTL library?). What's the impact if the threadpool gets really big, say more than 1000 threads?

Any references and pointers will be very much appreciated.

Provocative blog posting, "Avoid NIO, get better throughput." Paul Tyma's(2008) blog claims ~5000 threads without any trouble; I've heard folks claim more:

1. With NPTL on, Sun and Blackwidow JVM 1.4.2 scaled easily to 5000+ threads. Blocking model was consistently 25-35% faster than using NIO selectors. Lot of techniques suggested by EmberIO folks were employed - using multiple selectors, doing multiple (2) reads if the first read returned EAGAIN equivalent in Java. Yet we couldn't beat the plain thread per connection model with Linux NPTL.

I think the key here is to measure the overhead and performance, and make the move to non-blocking I/O only when you know you need to and can demonstrate an improvement. The additional effort to write and maintain non-blocking code should be factored in to your decision. My take is, if your application can be cleanly expressed using synchronous/blocking I/O, DO THAT. If your application is amenable to non-blocking I/O and you won't just be re-inventing blocking I/O badly in application-space, CONSIDER moving to nio based on measured performance needs. I'm amazed when I poke around the google results for this how few of the resources actually cite any (recent) numbers!

Also, see Paul Tyma's presentation slides: The old way is new again. Based on his work at Google, concrete numbers suggest that synchronous threaded I/O is quite scalable on Linux, and consider "NIO is faster" a myth that was true for awhile, but no longer. Some good additional commentary here on Comet Daily. He cites the following (anecdotal, still no solid link to benchmarks, etc...) result on NPTL:

In tests, NPTL succeeded in starting 100,000 threads on a IA-32 in two seconds. In comparison, this test under a kernel without NPTL would have taken around 15 minutes

If you really are running into scalability problems, you may want to tune the thread stack size using XX:ThreadStackSize. Since you mention Tomcat see here.

Finally, if you're bound and determined to use non-blocking I/O, make every effort to build on an existing framework by people who know what they're doing. I've wasted far too much of my own time trying to get an intricate non-blocking I/O solution right (for the wrong reasons).

See also related on SO.

Is there any way to dump the call stack in a running process in C or C++ every time a certain function is called? What I have in mind is something like this:

void foo()
{
   print_stack_trace();

   // foo's body

   return
}

Where print_stack_trace works similarly to caller in Perl.

Or something like this:

int main (void)
{
    // will print out debug info every time foo() is called
    register_stack_trace_function(foo); 

    // etc...
}

where register_stack_trace_function puts some sort of internal breakpoint that will cause a stack trace to be printed whenever foo is called.

Does anything like this exist in some standard C library?

I am working on Linux, using GCC.

---

Background

I have a test run that behaves differently based on some commandline switches that shouldn't affect this behavior. My code has a pseudo-random number generator that I assume is being called differently based on these switches. I want to be able to run the test with each set of switches and see if the random number generator is called differently for each one.

For a linux-only solution you can use backtrace(3) that simply returns an array of void * (in fact each of these point to the return address from the corresponding stack frame). To translate these to something of use, there's backtrace_symbols(3).

Pay attention to the notes section in backtrace(3):

The symbol names may be unavailable without the use of special linker options. For systems using the GNU linker, it is necessary to use the -rdynamic linker option. Note that names of "static" functions are not exposed, and won't be available in the backtrace.

What's the easiest/quickest way to interleave the lines of two (or more) text files? Example:

File 1:

line1.1
line1.2
line1.3

File 2:

line2.1
line2.2
line2.3

Interleaved:

line1.1
line2.1
line1.2
line2.2
line1.3
line2.3

Sure it's easy to write a little Perl script that opens them both and does the task. But I was wondering if it's possible to get away with fewer code, maybe a one-liner using Unix tools?

paste -d '\n' file1 file2

hi I'm really frustrated, First I have no idea how to code the very complex (make files), so I'm using IDE's that would ease the job for me like (netbeans , eclipse ,Kdevelop .. etc) i almost tried every thing starting with

• Emacs (i'm very slow on it and I need autocompletion)
• Netbeans 6.9.1 (crashes , very slow editor,but amazing and very easy in project creation)
• Eclipse (fast editor, i just hate the project configurations even in php it was so annoying)
• Kdevelop(I couldn't even get through the Hello world , configuration and Cmake issues "no executable specified " :S :S
• code blocks not so many ppl recommends it
• anjuta the code completion really sux

you might think i'm lazy or give up quickly but I swear I've searched alot and read the pooooooor tutorials of each one that is bloated with much information that no body needs for getting started.And I think we're really in lack of good documentation in the world of programming. and you can very much see that in (Boost library website) I really hope someone gives me helpful method on how to survive all this mess.

either I'll end up reading CMAKE details that i don't need , and memorize EMACS short cuts (CTRL x CTRL bla bla) and give up the idea of the comfortable easy to use IDEs in 2010 !

---

After thankfully very good answers , I think it's a must That I read about CMAKE , makefiles concepts. then half of my problems will go away , And I think now using IDE's that blind me from understanding the "Make" things isn't going to be helpful as i'm planning for long term development on linux

thanks for the brilliant simple answers.

p.s (Qt creator is awesome !! it's so much MAC like neat, clean and user friendly )

---

10 days later: goin old school and using EMACS and CMAKE

maybe you can try qt creator, it designed for developing QT application, however you can use it for other c++ program. It supports cmake.

I have a long-running Python process that is generating more data than I planned for. My results are stored in a list that will be serialized (pickled) and written to disk when the program completes -- if it gets that far. But at this rate, it's more likely that the list will exhaust all 1+ GB free RAM and the process will crash, losing all my results in the process.

I plan to modify my script to write results to disk periodically, but I'd like to save the results of the currently-running process if possible. Is there some way I can grab an in-memory data structure from a running process and write it to disk?

I found code.interact(), but since I don't have this hook in my code already, it doesn't seem useful to me (http://stackoverflow.com/questions/1637198/method-to-peek-at-a-python-program-running-right-now).

I'm running Python 2.5 on Fedora 8. Any thoughts?

Thanks a lot.

Shahin

There is not much you can do for a running program. The only thing I can think of is to attach the gdb debugger, stop the process and examine the memory. Alternatively make sure that your system is set up to save core dumps then kill the process with kill --sigsegv <pid>. You should then be able to open the core dump with gdb and examine it at your leisure.

There are some gdb macros that will let you examine python data structures and execute python code from within gdb, but for these to work you need to have compiled python with debug symbols enabled and I doubt that is your case. Creating a core dump first then recompiling python with symbols will NOT work, since all the addresses will have changed from the values in the dump.

Here are some links for introspecting python from gdb:

http://wiki.python.org/moin/DebuggingWithGdb

http://chrismiles.livejournal.com/20226.html

or google for 'python gdb'

N.B. to set linux to create coredumps use the ulimit command.

ulimit -a will show you what the current limits are set to.

ulimit -c unlimited will enable core dumps of any size.

I successfully compiled and installed the latest version of the Boost library onto my linux machine. Now, I would like to be able to use pkg-config to ease the process of providing linking paremeters with GCC.

Since I am too lazy for hand-coding my own .pc file, is there a script/tool which would automatically generate the needed .pc file or in some other way update pkg-config with boost flags?

(If someone already has that .pc file, a share would be welcome as well.)

What you're looking for seems to be a bit complicated, and a long-requested feature, as indicated in this 3 year old post https://svn.boost.org/trac/boost/ticket/1094 on Boost's trac. Reading through it shows that the feature was repeatedly postponed and never implemented (as of 1.4.3). The cause of the inability to generate a .pc file usable by pkg-config happens to do with boost's inconsistency in naming their library versions / build variants.

FWIW, an alternative for "automating" your building process is to use autotools (autoconf/automake). There's a link that might be of use to you (which I can't post because SO thinks I'm a spammer instead of a newcomer!), just google "tsuna boost m4 github" and it should take you there :)

I know it's possible to rotate video output in X server to display in portrait mode as well as landscape.

I'm curious if it's possible to rotate the video output that occurs pre-X server. The white text on black background output as the machine boots (rc.sysinit, bringing up eth connections, etc.).

if you use the framebuffer-console you can use the fbcon=rotate:n command at boot time to rotate the console output.

(n = 0: no rotation, n=1 90deg clockwise, n=2 upside down, n=3: 90deg counterclockwise)

The framebuffer options are documented in the kernel source in the file

linux-source-2.6.32/Documentation/fb/fbcon.txt

The code i am trying to run is bellow. I use nasm util to convert it into object file. When i tried to execute it says "can not execute binary file".

I run the command: nasm -f elf -o helloworld.o helloworld.asm

segment .data
msg  db   "Hello, world!",10
len  equ  $ - msg

segment .text
global _start

_start:
   mov  eax,4
   mov  ebx,1
   mov  ecx,msg
   mov  edx,len
   int  80h

   mov  eax,1
   mov  ebx,0
   int  80h

it is my first program in assembly and i did not use ubuntu except compiling some
basic c programs. i need some help thanks all.

It works perfectly for me your code.

Did you link the final object file?

Try this: ld helloworld.o -o helloworld

This one has me rather confused. I've written a query which runs fine from my development client but fails on the production client with error "ORA-01652: unable to extend temp segment by....". In both instances, the database and user is the same. On my development machine (MS Windows) I've got SQL*PLUS (Release 9.0.1.4.0) and Toad 9.0 (both using version 9.0.4.0.1 of the oci.dll). Both run the code without errors.

However when I run the same file, against the same database, using the same username/password from a different machine, this time version 10.2.0.4.0 (from the 10.2.0.4-1 Oracle instant client) I get the error.

It does occur reproducibly.

Unfortunately I've only got limited access to the dictionary views on the database which is set up as read-only (can't even get an explain plan!).

I've tried working around the problem by tuning the query (I suspect that there is a large interim result set which is subsequently trimmed down) but have not managed to change the behaviour at either client.

It may be possible to deploy a different version of the client on the machine causing the problems - but currently that looks like downgrading to a previous version.

Any ideas?

TIA

Update

Based on Gary's answer below, I had a look at the glogin.sql scripts - the only difference was that 'SET SQLPLUSCOMPATIBILITY 8.1.7' was present on the working client but absent on failing client - but adding it in did not resolve the problem.

I also tried

alter session set workarea_size_policy=manual;
alter session set hash_area_size=1048576000;

and

alter session set sort_area_size=1048576000;

to no avail :(

Update 2

I managed to find the same behaviour, this time talking to an Oracle 8i backend. In this case the database was RW. That allowed me to confirm that the different clients were, as I suspected, generating different plans. But why????

Looking at the output of 'SHOW PARAMETERS' both clients reported exactly the same settings!

Not really an answer - but a bit more information....

Our local DBAs were able to confirm that the 16Gb (!) TEMP tablespace was indeed being used and had filled up, but only from the Linux clients (I was able to recreate the error making an oci8 call from PHP). In the case of the sqlplus client I was actually using exactly the same file to run the query on both clients (copied via scp without text conversion - so line endings were CRLF - i.e. byte for byte the same as was running on the Windows client).

So the only rational solution was that the 2 client stacks were resulting in different execution plans!

Running the query from both clients approx simultaeneously on a DBMS with very little load gave the same result - meaning that the two clients also generated different sqlids for the query.

(and also Oracle was ignoring my hints - I hate when it does that).

There is no way Oracle should be doing this - even if it were doing some internal munging of the query before presenting it to the DBMS (which would give rise to the different sqlids) the client stack used should be totally transparent regarding the choice of an execution plan - this should only ever change based on the content of the query and the state of the DBMS.

The problem was complicated by not being to see any explain plans - but for the query to use up so much temporary tablespace, it had to be doing a very ugly join (at least partially cartesian) before filtering the resultset. Adding hints to override this had no effect. So I resolved the problem by splitting the query into 2 cursors and doing a nested lookup using PL/SQL. A very ugly solution, but it solved my immediate problem. Fortunately I just need to generate a text file.

For the benefit of anyone finding themselves in a similar pickle:

BEGIN

DECLARE
CURSOR query_outer IS
    SELECT some_primary_key, some_other_stuff
    FROM atable
    WHERE....

CURSOR query_details (p_some_pk) IS
    SELECT COUNT(*), SUM(avalue)
    FROM btable
    WHERE fk=p_some_pk
    AND....

FOR m IN query_outer
LOOP
    FOR n IN query_details(m.some_primary_key)
    LOOP
        dbms_out.put_line(....);
    END LOOP;
END LOOP;

END;

The more I use Oracle, the more I hate it!

I'm trying to get the title of the active window. The application is a background task so if the user has Eclipse open the function returns "Eclipse - blabla", so it's not getting the window title of my own window. I'm developing this in Python 2.6 using PyQt4.

My current solution, borrowed and slightly modified from an old answer here at SO, looks like this:

def get_active_window_title():
    title = ''
    root_check = ''

    root = Popen(['xprop', '-root'],  stdout=PIPE)

    if root.stdout != root_check:
        root_check = root.stdout

        for i in root.stdout:
            if '_NET_ACTIVE_WINDOW(WINDOW):' in i:
                id_ = i.split()[4]
                id_w = Popen(['xprop', '-id', id_], stdout=PIPE)

        for j in id_w.stdout:
            if 'WM_ICON_NAME(STRING)' in j:
                if title != j.split()[2]:
                    return j.split("= ")[1].strip(' \n\"')

It works for most windows, but not all. For example it can't find my kopete chat windows, or the name of the application i'm currently developing.

My next try looks like this:

def get_active_window_title(self):
    screen = wnck.screen_get_default()
    if screen == None:
        return "Could not get screen"
    window = screen.get_active_window()
    if window == None:
        return "Could not get window"
    title = window.get_name()
    return title;

But for some reason window is always None.

Does somebody have a better way of getting the current window title, or how to modify one of my ways, that works for all windows?

Edit:

In case anybody is wondering this is the way I found that seems to work for all windows.

def get_active_window_title(self):
    root_check = ''
    root = Popen(['xprop', '-root'],  stdout=PIPE)

    if root.stdout != root_check:
        root_check = root.stdout

        for i in root.stdout:
            if '_NET_ACTIVE_WINDOW(WINDOW):' in i:
                id_ = i.split()[4]
                id_w = Popen(['xprop', '-id', id_], stdout=PIPE)
        id_w.wait()
        buff = []
        for j in id_w.stdout:
            buff.append(j)

        for line in buff:
            match = re.match("WM_NAME\((?P<type>.+)\) = (?P<name>.+)", line)
            if match != None:
                type = match.group("type")
                if type == "STRING" or type == "COMPOUND_TEXT":
                    return match.group("name")
        return "Active window not found"

xdotool can do that.

xdotool getactivewindow

On POSIX systems, termination signals usually have the following order (according to many MAN pages and the POSIX Spec):

1. SIGTERM - politely ask a process to terminate. It shall terminate gracefully, cleaning up all resources (files, sockets, child processes, etc.), deleting temporary files and so on.

2. SIGQUIT - more forceful request. It shall terminate ungraceful, still cleaning up resources that absolutely need cleanup, but maybe not delete temporary files, maybe write debug information somewhere; on some system also a core dump will be written (regardless if the signal is caught by the app or not).

3. SIGKILL - most forceful request. The process is not even asked to do anything, but the system will clean up the process, whether it like that or not. Most likely a core dump is written.

How does SIGINT fit into that picture? A CLI process is usually terminated by SIGINT when the user hits CRTL+C, however a background process can also be terminated by SIGINT using KILL utility. What I cannot see in the specs or the header files is if SIGINT is more or less forceful than SIGTERM or if there is any difference between SIGINT and SIGTERM at all.

UPDATE:

The best description of termination signals I found so far is in the GNU LibC Documentation. It explains very well that there is an intended difference between SIGTERM and SIGQUIT.

It says about SIGTERM:

It is the normal way to politely ask a program to terminate.

And it says about SIGQUIT:

[...] and produces a core dump when it terminates the process, just like a program error signal. You can think of this as a program error condition “detected” by the user. [...] Certain kinds of cleanups are best omitted in handling SIGQUIT. For example, if the program creates temporary files, it should handle the other termination requests by deleting the temporary files. But it is better for SIGQUIT not to delete them, so that the user can examine them in conjunction with the core dump.

And SIGHUP is also explained well enough. SIGHUP is not really a termination signal, it just means the "connection" to the user has been lost, so the app cannot expect the user to read any further output (e.g. stdout/stderr output) and there is no input to expect from the user any longer. For most apps that mean they better quit. In theory an app could also decide that it goes into daemon mode when a SIGHUP is received and now runs as a background process, writing output to a configured log file. For most daemons already running in the background, SIGHUP usually means that they shall reexamine their configuration files, so you send it to background processes after editing config files.

However there is no useful explanation of SIGINT on this page, other than that it is sent by CRTL+C. Is there any reason why one would handle SIGINT in a different way than SIGTERM? If so what reason would this be and how would the handling be different?

SIGTERM and SIGKILL are intended for general purpose "terminate this process" requests. SIGTERM (by default) and SIGKILL (always) will cause process termination. SIGTERM may be caught by the process (e.g. so that it can do its own cleanup if it wants to), or even ignored completely; but SIGKILL cannot be caught or ignored.

SIGINT and SIGQUIT are intended specifically for requests from the terminal: particular input characters can be assigned to generate these signals (depending on the terminal control settings). The default action for SIGINT is the same sort of process termination as the default action for SIGTERM and the unchangeable action for SIGKILL; the default action for SIGQUIT is also process termination, but additional implementation-defined actions may occur, such as the generation of a core dump. Either can be caught or ignored by the process if required.

SIGHUP, as you say, is intended to indicate that the terminal connection has been lost, rather than to be a termination signal as such. But, again, the default action for SIGHUP (if the process does not catch or ignore it) is to terminate the process in the same way as SIGTERM etc. .

There is a table in the POSIX definitions for signal.h which lists the various signals and their default actions and purposes, and the General Terminal Interface chapter includes a lot more detail on the terminal-related signals.