Possible Duplicate:
Please help me understanding the error a+++++b in C

In a discussion today the topic of pre/post increment came up.

We tried several combinations of addition and increment:

c = a+++b //works
c = a++ + ++b //works
c = a+++ ++b //works
c = a+++++b // doesn't work, compiler error "error: invalid lvalue in increment

Where is the difference? What exactly does that additional whitespace add for the parser so the program compiles?

Compilers are written in stages. The first stage is called the lexer and turns characters into a symbolic structure. So "++" becomes something like an enum SYMBOL_PLUSPLUS. Later, the parser stage turns this into an abstract syntax tree, but it can't change the symbols. You can affect the lexer by inserting spaces (which end symbols unless they are in quotes).

Normal lexers are greedy (with some exceptions), so your code is being interpreted as

a++ ++ +b

The input to the parser is a stream of symbols, so your code would be something like:

[ SYMBOL_NAME(name = "a"), 
  SYMBOL_PLUS_PLUS, 
  SYMBOL_PLUS_PLUS, 
  SYMBOL_PLUS, 
  SYMBOL_NAME(name = "b") 
]

Which the parser thinks is syntactically incorrect. (EDIT based on comments: Semantically incorrect because you cannot apply ++ to an r-value, which a++ results in)

a+++b 

is

a++ +b

Which is ok. So are your other examples.

Can you think of any legitimate (smart) uses for runtime code modification (program modifying it's own code at runtime)?

Modern operating systems seem to frown upon programs that do this since this technique has been used by viruses to avoid detection.

All I can think of is some kind of runtime optimization that would remove or add some code by knowing something at runtime which cannot be known at compile time.

There are many valid cases for code modification. Generating code at run time can be useful for:

• Some virtual machines use JIT compilation to improve performance.
• Generating specialized functions on the fly has long been common in computer graphics. See e.g. Rob Pike and Bart Locanthi and John Reiser Hardware Software Tradeoffs for Bitmap Graphics on the Blit (1984) or this posting (2006) by Chris Lattner on Apple's use of LLVM for runtime code specialization in their OpenGL stack.
• In some cases software resorts to a technique known as trampoline which involves the dynamic creation of code on the stack (or another place). Examples are GCC's nested functions and the signal mechanism of some Unices.

Sometimes code is translated into code at runtime (this is called dynamic binary translation):

• Emulators like Apple's Rosetta use this technique to speed up emulation. Another example is Transmeta's code morphing software.
• Sophisticated debuggers and profilers like Valgrind or Pin use it to instrument your code while it is being executed.
• Before extensions were made to the x86 instruction set, virtualization software like VMWare could not directly run privileged x86 code inside virtual machines. Instead it had to translate any problematic instructions on the fly into more appropriate custom code.

Code modification can be used to work around limitations of the instruction set:

• There was a time (long ago, I know), when computers had no instructions to return from a subroutine or to indirectly address memory. Self modifying code was the only way to implement subroutines, pointers and arrays.

More cases of code modification:

• Many debuggers replace instructions to implement breakpoints.
• Some dynamic linkers modify code at runtime. This article provides some background on the runtime relocation of Windows DLLs, which is effectively a form of code modification.

I recently had a discussion with another developer who claimed to me that JOINs (SQL) are useless. This is technically true but he added that using joins is less efficient than making several requests and link tables in the code (C# or Java).

For him joins are for lazy people that don't care about performance. Is this true? Should we avoid using joins?

No, we should avoid developers who hold such incredibly wrong opinions.

In many cases, a database join is several orders of magnitude faster than anything done via the client, because it avoids DB roundtrips, and the DB can use indexes to perform the join.

Off the top of my head, I can't even imagine a single scenario where a correctly used join would be slower than the equivalent client-side operation.

Edit: There are some rare cases where custom client code can do things more efficiently than a straightforward DB join (see comment by meriton). But this is very much the exception.

Hi all,

I'm an undergraduate CS student, and I'm currently taking my required Operating Systems course. I originally thought the material would be quite dry, but to my surprise I'm really interested in it.

What I'm wondering is, how are everyday machines (not so much computers and mobile devices as appliances, digital watches, etc) programmed? What kind of code goes into the programming of a Coca-Cola vending machine? How does my coffee maker accept a pre-programmed time and begin brewing a pot of coffee hours later, when that time arrives?

Do these kinds of machines have operating systems inside of them, or is it something even more basic? Are they written in Assembly, C, or some other language?

And, I would really like to find some resource that lists these operating systems or underlying code systems, possibly even with source code if possible. If anyone knows of such a resource (searching yielded nothing for me), that would be fantastic.

Most of what you're talking about are embedded based systems where C is a luxury often not available. They don't have software in the traditional sense. Most of the time the software is written in C, assembly, or even machine code. C and ASM require compilers to be written to use them for that platform. Machine code is written as binary w/o a compiler.

Your coffee pot and most simple systems like that don't carry an operating system. They simply load from a start address in memory and you put your code there. Often these systems have their "code" burned into EEPROMS that act as the hard drive of the system. Screw up the code after burning the proms, throw the chips away reburn the code on the chip, and start over. There are newer chips FPGA that higher end devices use to make testing, deploying, etc easier, but they are the same thing.

Coca-cola machines, routers, etc. typically use a realtime OS like QNX, EMBOS, or sometimes RTlinux if you're lucky. Most of these are proprietary OS you license for lots of money, but they have C compilers, drivers to work with hardware, etc.

http://www.qnx.com/

http://www.segger.com/cms/embos.html

http://www.microsoft.com/windowsembedded/en-us/campaigns/compact7/default.aspx?WT.srch=1&WT.mc_ID=SEARCH

RTLinux

Well, this is going to be a short one...

I want to parse /proc/net/tcp/, but is it safe? I mean, how to open and read it and not be afraid, that some other process (or the OS) will be changing it in the same time?

Although the files in /proc appear as regular files in userspace, they are not really files but rather entities that support the standard file operations from userspace (open, read, close). Note that this is quite different than having an ordinary file on disk that is being changed by the kernel.

All the kernel does is print its internal state into its own memory using a sprintf-like function, and that memory is copied into userspace whenever you issue a read(2) system call.

The kernel handles these calls in an entirely different way than for regular files, which could mean that the entire snapshot of the data you will read could be ready at the time you open(2) it, while the kernel makes sure that concurrent calls are consistent and atomic. I haven't read that anywhere, but it doesn't really make sense to be otherwise.

My advice is to take a look at the implementation of a proc file in your particular Unix flavour. This is really an implementation issue (as is the format and the contents of the output) that is not governed by a standard.

The simplest example would be the implementation of the uptime proc file in Linux: http://lxr.free-electrons.com/source/fs/proc/uptime.c. Note how the entire buffer is produced in the callback function supplied to single_open.

So we've been through this several times now, we release a game (for cheap) and someone hacks it and puts it up on a mirror. We setup Google Alerts for all our apps, so we get told daily who's doing the hacking. So far, we have implemented the licensing service as Google has suggested, our salt is randomly made each time the license is initiated with the unique device ID. We run the check service once, when the application is started for the first time. We then generate a 512 character hash for the key and the stored value that is compared against in SharedPreferences from there on out.

Now, I know that checking once is probably where the application is being blocked. Our bytecode has most likely been looked at and recompiled without the line that initiates the check.

From here, I don't want to obfuscate our code as I have seen it broken before. I want something a little more solid, and I also want to learn how to do this properly. I am more interested in learning than making money at this point since only 2% of people will ever look for a hacked version.

So far, on my own, I have come up with a random number generator that is placed in several startup areas of the game. When initiated (say, 1 out of 50 times) the license is checked. I know this would make it harder to hack because the cracker would have to eliminate each case, compile, eliminate, compile. This method however, is still crackable...so what do you guys suggest? Again, I am really interested in this process of security, so please educate, don't turn this into a discussion on obfuscation or checking periodically based on a timestamp.

Thanks

My idea isnt hacker proof, but might remove some of the interest for hacking the game.

Freemium model

1) Make the first 5-10 levels free so people can learn the game and have some fun without paying. Less will want to hack the first level and the game will spread even further by Freemium model.

Shareware/clustered levelpacks

2) Let part of the game levels or logic stay online. Eg. when reaching for level 5 or 10 or 15, then download small parts for the game, and every time submit the progress-log from the game and validate this against possible values + hashcodes. This could perhaps make it possible to automatically close down of hacked accounts.

Stealth cheater protection

3) You could also just count "small warning flags" that you place around in the game. Dont just check for the "validation" in the beginning, no build these flags into the game logic itself. Dont make it break the gameplay, because then noone will look for it. Then when the user reached the end of level monster, check if there were any logged warning flags. These will not show up inside the game, so the unknowing user with a hacked edition could be playing for hours/days and suddently realize that he/she couldnt finish the game or advance to next level, because the game had a "bug". What the user didnt know was that this bug only occures on hacked clients.

Conclusion

Be smarter than the crackers. Fool them into thinking the job was done. Make a copyprotection and know that the more advanced crackers will be able to remove it. But they probably dont want to play 50 levels to check if the crack also works all the way.

Once they realize this problem, they might start to crack it too. But if you break the game up into level-packs, you can still validate between each pack download. So once you receive hacked client hash data, then just execute an exeception and crash the game on the client. Whoops the game crashed. Dont tell its because its hacked. A program error can happend. :-)

Again, its not hacker proof. But it might annoy them enough to move on to the next game. Lastly, you could also put out regular updates for the game and only the latest version should be able to "post the records" etc. so the active users would have to update to keep in the loop.

I found this code to swap two numbers without using a third variable, using the XOR ^ operator.

Code:

int i = 25;
int j = 36;
j ^= i;       
i ^= j;
j ^= i;

Console.WriteLine("i:" + i + " j:" + j);

//numbers Swapped correctly
//Output: i:36 j:25

Now I changed the above code to this equivalent code.

My Code:

int i = 25;
int j = 36;

j ^= i ^= j ^= i;   // I have changed to this equivalent (???).

Console.WriteLine("i:" + i + " j:" + j);

//Not Swapped correctly            
//Output: i:36 j:0

Now, I want to know, Why does my code give incorrect output?

EDIT: Okay, got it.

The first point to make is that obviously you shouldn't use this code anyway. However, when you expand it, it becomes equivalent to:

j = j ^ (i = i ^ (j = j ^ i));

(If we were using a more complicated expression such as foo.bar++ ^= i, it would be important that the ++ was only evaluated once, but here I believe it's simpler.)

Now, the order of evaluation of the operands is always left to right, so to start with we get:

j = 36 ^ (i = i ^ (j = j ^ i));

This (above) is the most important step. We've ended up with 36 as the LHS for the XOR operation which is executed last. The LHS is not "the value of j after the RHS has been evaluated".

The evaluation of the RHS of the ^ involves the "one level nested" expression, so it becomes:

j = 36 ^ (i = 25 ^ (j = j ^ i));

Then looking at the deepest level of nesting, we can substitute both i and j:

j = 36 ^ (i = 25 ^ (j = 25 ^ 36));

... which becomes

j = 36 ^ (i = 25 ^ (j = 61));

The assignment to j in the RHS occurs first, but the result is then overwritten at the end anyway, so we can ignore that - there are no further evaluations of j before the final assignment:

j = 36 ^ (i = 25 ^ 61);

This is now equivalent to:

i = 25 ^ 61;
j = 36 ^ (i = 25 ^ 61);

Or:

i = 36;
j = 36 ^ 36;

Which becomes:

i = 36;
j = 0;

I think that's all correct, and it gets to the right answer... apologies to Eric Lippert if some of the details about evaluation order are slightly off :(

Which value is better to use? Boolean true or Integer 1?

The above topic made me do some experiments with bool and int in if condition. So just out of curiosity I wrote this program:

int f(int i) 
{
    if ( i ) return 99;   //if(int)
    else  return -99;
}
int g(bool b)
{
    if ( b ) return 99;   //if(bool)
    else  return -99;
}
int main(){}

g++ intbool.cpp -S generates asm code for each functions as follows:

• asm code for f(int)

  __Z1fi:
     LFB0:
           pushl  %ebp
     LCFI0:
            movl  %esp, %ebp
     LCFI1:
            cmpl  $0, 8(%ebp)
            je    L2
            movl  $99, %eax
            jmp   L3
     L2:
            movl  $-99, %eax
     L3:
            leave
     LCFI2:
            ret
  

• asm code for g(bool)

  __Z1gb:
     LFB1:
            pushl %ebp
     LCFI3:
            movl  %esp, %ebp
     LCFI4:
            subl  $4, %esp
     LCFI5:
            movl  8(%ebp), %eax
            movb  %al, -4(%ebp)
            cmpb  $0, -4(%ebp)
            je    L5
            movl  $99, %eax
            jmp   L6
     L5:
            movl  $-99, %eax
     L6:
            leave
     LCFI6:
            ret
  

Surprisingly, g(bool) generates more asm instructions! Does it mean that if(bool) is little slower than if(int)? I used to think bool is especially designed to be used in conditional statement such as if, so I was expecting g(bool) to generate less asm instructions, thereby making g(bool) more efficient and fast.

EDIT:

I'm not using any optimization flag as of now. But even absence of it, why does it generate more asm for g(bool) is a question for which I'm looking for a reasonable answer. I should also tell you that -O2 optimization flag generates exactly same asm. But that isn't the question. The question is what I've asked.

---

Makes sense to me. Your compiler apparently defines a bool as an 8-bit value, and your system ABI requires it to "promote" small (< 32-bit) integer arguments to 32-bit when pushing them onto the call stack. So to compare a bool, the compiler generates code to isolate the least significant byte of the 32-bit argument that g receives, and compares it with cmpb. In the first example, the int argument uses the full 32 bits that were pushed onto the stack, so it simply compares against the whole thing with cmpl.

I'm in a "special" situation about effeciency of my program. Now I'm at a phase where I want to maximize my application; being top notch and reducing battery consumption.

Before the question:

• First, my application work. It runs fine - no errors whatsoever.
• Second, I have read Designing for Performance on Android developers website and I have optimized small things they're mentioning. No problems whatsoever.

Now, I'm curious to hear about developers "special" tricks (fixes) that optimized their application(s). Stuff that users never gonna recognize or pay attention to; but instead increased the battery life or effectively made your application easier to maintain.

So, what's your unique optimizing trick(s)? (e.g. Object Pools? other patterns? unknown things? things that the website forgot to mention?) It doesn't have to be Android specific.

I'm in a particular situation where I'm really looking for knowledge and I think this will be a great opportunity to share developers knowledge about a situation they've all been in.

Please, vote up great answers as that will encourage great developers to share their knowledge.

At some point you are going to get to the point where using known tricks will hit their limits. The best thing to do at this point is profile your code and see what areas are the bottle-necks based on your specific requirements.

Profiling with Traceview and dmtracedump: an article on how to use the tools to profile your application.

Quick question: from a design point of view, why is that, in C++, there is no mother-of-all base-class, what's usually object in other languages?

The definitive ruling is found here. In short, it doesn't convey any semantic meaning. It will have a cost. Templates are more useful for containers.

Why doesn't C++ have a universal class Object?

• We don't need one: generic programming provides statically type safe alternatives in most cases. Other cases are handled using multiple inheritance.

• There is no useful universal class: a truly universal carries no semantics of its own.

• A "universal" class encourages sloppy thinking about types and interfaces and leads to excess run-time checking.

• Using a universal base class implies cost: Objects must be heap-allocated to be polymorphic; that implies memory and access cost. Heap objects don't naturally support copy semantics. Heap objects don't support simple scoped behavior (which complicates resource management). A universal base class encourages use of dynamic_cast and other run-time checking.

I struggled finding a how-to which provides a stable solution for using Qt with Visual Studio 2010, so after collecting all the bits of information and some trial and error, I would like to write my solution into a guide.

The problem, or why is it not possible to use prebuilt binaries?

It seems that using binaries built for Visual Studio 2008 might work in some special cases, but I found them not to work. In my case they compiled OK, but they produce runtime errors, like this:

or when started from Visual Studio 2010:

Update: I found a blog post analysing why does it work for some people, while it does not for others. In one word, it depends on whether you have Visual Studio 2008 installed on the same machine, or not. http://blog.paulnettleship.com/2010/11/11/troubleshooting-visual-studio-2010-and-qt-4-7-integration/

The most important thing (that I stupidly didn’t realize) was the fact that you CANNOT use the Visual Studio 2008 compiled libraries and dll’s (available on the Qt webpage) if you don’t have Visual Studio 2008 installed. The reason is because the Qt SDK you download is a debug build which is dependant on the VC9.0 DebugCRT, meaning it needs the Visual C++ 2008 Debug Runtime installed, which is NOT available as a redistributable installer. The only way to install the DebugCRT is to install the entirety of Visual Studio 2008.

First of all, it’s very important to understand that for using Qt with Visual Studio 2010, it's not possible to use the pre-built binaries which were made for Visual Studio 2008, but you have to compile it from source.

Downloading Qt

On http://qt.nokia.com/downloads, click LGPL.

You should not download Qt by clicking "Qt libraries 4.7.2 for Windows (Visual Studio 2008, 218 MB)", but by clicking on the "zip" link above it.

On that link, you get a big zip file like "qt-everywhere-opensource-src-4.7.2.zip". Unzip this into a folder and make its path something nice and small, for example "E:\Qt"

Visual Studio Command Prompt

Now that we have the sources, we need to build the binaries. To do it, open the Microsoft Visual Studio 2010\Visual Studio Tools\Visual Studio Command Prompt (2010) link from your start menu, or even pin it to the taskbar (a good idea). This is a special command prompt which has all the variables set for building with Visual Studio 2010 tools.

Once within the command prompt, navigate to your extracted Qt folder using old-school DOS way, which means you have to change drive letter by E:, enter directories by cd Qt and list dir contents by dir. You can use the tab key for helping you with the directory names. When you have arrived at the correct directory, a dir command should return something like this.

Building Qt

Now it’s time for configure and build. For configuring a minimalist Qt, I'm using the following flags with configure.exe. Just copy and paste it into the command line. Look in the Qt reference manual for what flag to use or not to use.

configure.exe -release -no-webkit -no-phonon -no-phonon-backend -no-script -no-scripttools -no-qt3support -no-multimedia -no-ltcg

Once configure.exe has finished (it was 10 minutes for me), you'll need to start the build process. It will take about 20-30 minutes with the above flags. To start it, just type:

nmake

Setting environment variables

Basically we are done. All you need to do is to set your environment variables (QTDIR and PATH), which tell programs where to find Qt. If you are on Windows 7, you can use the following command to set QTDIR to your installation dir.

setx QTDIR e:\Qt

For setting the PATH, I strongly recommend using Path Editor. Within Path Editor

add the directory of Qt\bin to your PATH

(it doesn't matter if it's in system path or user path)

If you prefer to use Control Panel\System\Environment Variables, then you can set these there too.

Qt Visual Studio Add-in

Here you go, after a logoff-logon or a restart, all the Qt demo applications should start correctly (I recommend have a look at bin\qtdemo.exe). Now you can download and install the Visual Studio Add-in (qt-vs-addin-1.1.9.exe) from the Qt download page, it will work perfectly.

Appendix A: Official Instructions:

There is a page at the official wiki at Qt called Qt 4.7 Installing Qt for Windows, but I found it lacking important information, but here is it for reference:

http://doc.qt.nokia.com/4.7/install-win.html

The references I used for this post

• Qt DevNet forums

• Recommended flags for a minimalistic Qt build

• Building Qt 4.5 with Visual C++ 2010

• How to compile Qt as static

• Qt 4.7: Configure options for Qt

• Edit the PATH environment variable in Windows without pain - op111.net

So..after developing a few Firefox & Chrome extensions, I've decided to try and expand my skillset by developing an Internet Explorer extension in C#. I went into it thinking it wouldn't be too bad.

Wow.

I was really wrong. So, my question is, does anyone here have experience with/in developing IE extensions that can share their knowledge? This would include code samples, or links to good ones, or documentation on the process, or anything.

I really want to do this, but I'm hitting a giant wall with lousy documentation, lousy code/example code/lack thereof. Any help/resources you could offer would be greatly appreciated.

Specifically, I would like to start with how to get access to/manipulate the DOM from within a IE extension.

EDIT:

Even more details:

Ideally, I would like to plant a toolbar button that, when clicked, popped a menu up that contains links to external sites. I would also like to access the DOM and plant Javascript on the page depending on some conditions. I would also like to know what is the best way to persist information in an IE extension. In Firefox/Chrome/Most modern browsers, you use window.localStorage, but obviously with IE8/IE7, that's not an option. Maybe a SQLite DB or such? It is ok to assume that .NET 4.0 will be installed on a user's computer.

Thanks!

EDIT:

I'm providing a 500 reputation bounty on this. I'm serious about learning how to build a Internet Explorer extension. I don't want to use Spice IE as I want to build one that is compatible with IE9 as well. I've added the C++ tag to this question as well, because if it's better to build one in C++, I can do that.

I hope it's worth 500 reputation to somebody to help not only me, but future people with the same intentions.

One last edit:

It looks like the general consensus is get a 24 pack of diet coke, two monitors, and lots of time. But if anyone else would like to offer up a great answer, please do so before I have to award the bounty.

Man... this has been a lot of work! I was so curious about how to do this, that I did it myself.

First of all... credit is not all mine. This is a compilation of what I found, on these sites:

• CodeProject article, how to make a BHO;
• 15seconds, but it was not 15 seconds, it took about 7 hours;
• Microsoft tutorial, helped me adding the command button.
• And this social.msdn topic, that helped me figure out that the assembly must be in the GAC.
• many other sites, in the discovery process...

And of course, I wanted my answer to have the features you asked:

• DOM traversal to find something;
• a button that shows a window (in my case to setup)
• persist the configuration (I will use regitry for that)
• and finally execute javascript.

I will describe it step by step, how I managed to do it working with Internet Explorer 8, in Windows 7 x64... note that I could not test in other configurations. Hope you understand =)

Creating a Working Internet Explorer 8 Addon

I am using Visual Studio 2010, C# 4, .Net Framework 4, so some of these steps might be slightly different for you.

Created a class library. I called mine InternetExplorerExtension.

Add these references to the project:

• Interop.SHDocVw
• Microsoft.mshtml

this is what my references section in csproj contains:

<Reference Include="Interop.SHDocVw, Version=1.1.0.0, Culture=neutral, PublicKeyToken=90ba9c70f846762e, processorArchitecture=MSIL">
  <SpecificVersion>False</SpecificVersion>
  <EmbedInteropTypes>True</EmbedInteropTypes>
  <HintPath>C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Interop.SHDocVw.dll</HintPath>
</Reference>
<Reference Include="Microsoft.CSharp" />
<Reference Include="Microsoft.mshtml, Version=7.0.3300.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
  <EmbedInteropTypes>True</EmbedInteropTypes>
</Reference>
<Reference Include="System" />
<Reference Include="System.Data" />
<Reference Include="System.Drawing" />
<Reference Include="System.Windows.Forms" />
<Reference Include="System.Xml" />

Create the following files:

IEAddon.cs

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using Interop.ShDocVw;
using mshtml;
using Microsoft.Win32;
using System.Windows.Forms;

namespace InternetExplorerExtension
{
    [ComVisible(true)]
    [ClassInterface(ClassInterfaceType.None)]
    [Guid("D40C654D-7C51-4EB3-95B2-1E23905C2A2D")]
    [ProgId("MyBHO.WordHighlighter")]
    public class WordHighlighterBHO : IObjectWithSite, IOleCommandTarget
    {
        const string DefaultTextToHighlight = "browser";

        IWebBrowser2 browser;

        #region Highlight Text
        void OnDocumentComplete(object pDisp, ref object URL)
        {
            var document = browser.Document as IHTMLDocument2;

            var window = document.parentWindow;
            window.execScript(@"function FncAddedByAddon() { alert('Message added by addon.'); }");

            Queue<IHTMLElement> queue = new Queue<IHTMLElement>();
            queue.Enqueue(document.body);
            while (queue.Count > 0)
            {
                var el = queue.Dequeue();
                // replacing desired text with a highlighted version of it
                el.innerHTML = el.innerHTML.Replace(TextToHighlight, "<span style='background-color: yellow; cursor: hand;' onclick='javascript:FncAddedByAddon()' title='Click to open script based alert window.'>" + TextToHighlight + "</span>");
                // adding children to collection
                var x = (HTMLElementCollection)(el.children);
                foreach (IHTMLElement eachChild in x)
                    queue.Enqueue(eachChild);
            }
        }
        #endregion
        #region Load and Save Data
        static string TextToHighlight = DefaultTextToHighlight;
        public static string RegData = "Software\\MyIEExtension";
        private static void SaveOptions()
        {
            RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(RegData, true);
            if (registryKey == null)
                registryKey = Registry.LocalMachine.CreateSubKey(RegData);
            registryKey.SetValue("Data", TextToHighlight);
            registryKey.Close();
        }
        private static void LoadOptions()
        {
            RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(RegData, true);
            if (registryKey == null)
                TextToHighlight = DefaultTextToHighlight;
            TextToHighlight = (string)registryKey.GetValue("Data");
            registryKey.Close();
        }
        #endregion

        #region Implementation of IObjectWithSite
        int IObjectWithSite.SetSite(object site)
        {
            if (site != null)
            {
                LoadOptions();
                browser = (IWebBrowser2)site;
                ((DWebBrowserEvents2_Event)browser).DocumentComplete +=
                    new DWebBrowserEvents2_DocumentCompleteEventHandler(this.OnDocumentComplete);
            }
            else
            {
                ((DWebBrowserEvents2_Event)browser).DocumentComplete -=
                    new DWebBrowserEvents2_DocumentCompleteEventHandler(this.OnDocumentComplete);
                browser = null;
            }
            return 0;
        }
        int IObjectWithSite.GetSite(ref Guid guid, out IntPtr ppvSite)
        {
            IntPtr punk = Marshal.GetIUnknownForObject(browser);
            int hr = Marshal.QueryInterface(punk, ref guid, out ppvSite);
            Marshal.Release(punk);
            return hr;
        }
        #endregion
        #region Implementation of IOleCommandTarget
        int IOleCommandTarget.QueryStatus(IntPtr pguidCmdGroup, uint cCmds, ref OLECMD prgCmds, IntPtr pCmdText)
        {
            return 0;
        }
        int IOleCommandTarget.Exec(IntPtr pguidCmdGroup, uint nCmdID, uint nCmdexecopt, IntPtr pvaIn, IntPtr pvaOut)
        {
            var form = new HighlighterOptionsForm();
            form.InputText = TextToHighlight;
            if (form.ShowDialog() != DialogResult.Cancel)
            {
                TextToHighlight = form.InputText;
                SaveOptions();
            }
            return 0;
        }
        #endregion

        #region Registering with regasm
        public static string RegBHO = "Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Browser Helper Objects";
        public static string RegCmd = "Software\\Microsoft\\Internet Explorer\\Extensions";

        [ComRegisterFunction]
        public static void RegisterBHO(Type type)
        {
            string guid = type.GUID.ToString("B");

            // BHO
            {
                RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(RegBHO, true);
                if (registryKey == null)
                    registryKey = Registry.LocalMachine.CreateSubKey(RegBHO);
                RegistryKey key = registryKey.OpenSubKey(guid);
                if (key == null)
                    key = registryKey.CreateSubKey(guid);
                key.SetValue("Alright", 1);
                registryKey.Close();
                key.Close();
            }

            // Command
            {
                RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(RegCmd, true);
                if (registryKey == null)
                    registryKey = Registry.LocalMachine.CreateSubKey(RegCmd);
                RegistryKey key = registryKey.OpenSubKey(guid);
                if (key == null)
                    key = registryKey.CreateSubKey(guid);
                key.SetValue("ButtonText", "Highlighter options");
                key.SetValue("CLSID", "{1FBA04EE-3024-11d2-8F1F-0000F87ABD16}");
                key.SetValue("ClsidExtension", guid);
                key.SetValue("Icon", "");
                key.SetValue("HotIcon", "");
                key.SetValue("Default Visible", "Yes");
                key.SetValue("MenuText", "&Highlighter options");
                key.SetValue("ToolTip", "Highlighter options");
                //key.SetValue("KeyPath", "no");
                registryKey.Close();
                key.Close();
            }
        }

        [ComUnregisterFunction]
        public static void UnregisterBHO(Type type)
        {
            string guid = type.GUID.ToString("B");
            // BHO
            {
                RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(RegBHO, true);
                if (registryKey != null)
                    registryKey.DeleteSubKey(guid, false);
            }
            // Command
            {
                RegistryKey registryKey = Registry.LocalMachine.OpenSubKey(RegCmd, true);
                if (registryKey != null)
                    registryKey.DeleteSubKey(guid, false);
            }
        }
        #endregion
    }
}

Interop.cs

using System;
using System.Runtime.InteropServices;
namespace InternetExplorerExtension
{
    [ComVisible(true)]
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    [Guid("FC4801A3-2BA9-11CF-A229-00AA003D7352")]
    public interface IObjectWithSite
    {
        [PreserveSig]
        int SetSite([MarshalAs(UnmanagedType.IUnknown)]object site);
        [PreserveSig]
        int GetSite(ref Guid guid, [MarshalAs(UnmanagedType.IUnknown)]out IntPtr ppvSite);
    }


    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
    public struct OLECMDTEXT
    {
        public uint cmdtextf;
        public uint cwActual;
        public uint cwBuf;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 100)]
        public char rgwz;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct OLECMD
    {
        public uint cmdID;
        public uint cmdf;
    }

    [ComImport(), ComVisible(true),
    Guid("B722BCCB-4E68-101B-A2BC-00AA00404770"),
    InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)]
    public interface IOleCommandTarget
    {

        [return: MarshalAs(UnmanagedType.I4)]
        [PreserveSig]
        int QueryStatus(
            [In] IntPtr pguidCmdGroup,
            [In, MarshalAs(UnmanagedType.U4)] uint cCmds,
            [In, Out, MarshalAs(UnmanagedType.Struct)] ref OLECMD prgCmds,
            //This parameter must be IntPtr, as it can be null
            [In, Out] IntPtr pCmdText);

        [return: MarshalAs(UnmanagedType.I4)]
        [PreserveSig]
        int Exec(
            //[In] ref Guid pguidCmdGroup,
            //have to be IntPtr, since null values are unacceptable
            //and null is used as default group!
            [In] IntPtr pguidCmdGroup,
            [In, MarshalAs(UnmanagedType.U4)] uint nCmdID,
            [In, MarshalAs(UnmanagedType.U4)] uint nCmdexecopt,
            [In] IntPtr pvaIn,
            [In, Out] IntPtr pvaOut);
    }
}

and finally a form, that we will use to configure the options. In this form place a TextBox and an Ok Button. Set the DialogResult of the button to Ok. Place this code in the form code:

using System.Windows.Forms;
namespace InternetExplorerExtension
{
    public partial class HighlighterOptionsForm : Form
    {
        public HighlighterOptionsForm()
        {
            InitializeComponent();
        }

        public string InputText
        {
            get { return this.textBox1.Text; }
            set { this.textBox1.Text = value; }
        }
    }
}

In the project properties, do the following:

• Sign the assembly with a strong-key;
• In the Debug tab, set Start External Program to C:\Program Files (x86)\Internet Explorer\iexplore.exe
• In the Debug tab, set Command Line Arguments to http://msdn.microsoft.com/en-us/library/ms976373.aspx#bho_getintouch

• In the Build Events tab, set Post-build events command line to:

  "C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools\x64\gacutil.exe" /f /i $(TargetDir)$(TargetFileName) "C:\Windows\Microsoft.NET\Framework\v4.0.30319\RegAsm.exe" /unregister $(TargetDir)$(TargetFileName) "C:\Windows\Microsoft.NET\Framework\v4.0.30319\RegAsm.exe" $(TargetDir)$(TargetFileName)

Attention: as my computer is x64, there is a specific x64 inside the path of gacutil executable on my machine that may be different on yours.

How this addon works

It traverses all DOM tree, replacing the text, configured using the button, by itself with a yellow background. If you click on the yellowed texts, it calls a javascript function that was inserted on the page dynamically. The default word is 'browser', so that it matches a lot of them! EDIT: after changing the string to be highlighted, you must click the URL box and press Enter... F5 will not work, I think that it is because F5 is considered as 'navigation', and it would require to listen to navigate event (maybe). I'll try to fix that later.

Now, it is time to go. I am very tired. Feel free to ask questions... may be I will not be abled to answer since I am going on a trip... in 3 days I'm back, but I'll try to come here in the meantime.

Possible Duplicate:
Do you prefix your instance variable with 'this' in java ?

Simple question.

Is there a real difference between (generally, not only in constructors):

class ThisClass{
    private int a;
    public ThisClass(int a)
    {
        this.a = a;
    }
}

and

class ThisClass{
    private int _a;
    public ThisClass(int a)
    {
        _a = a;
    }
}

Other than coding convention?

Clarification - I didn't mean to the use of "this" as a reference to the object (such as passing it in a method) only to the use of it when assigning values in methods.

Technically, most languages don't care what the variable is called. This just changes the symbol lookup rules when compiling. The code to access this.a and _a in your examples should be the same, although the symbol tables will be different.

Locating to this.a in the symbol table may be slightly faster than locating _a during compilation. Any gain in lookup speed may be used in parsing this. so I wouldn't consider it an optimization. The optimization would only apply during compilation. a may be slightly more space efficient than _a in the symbol table.

When not being used for disambiguation this is likely a required syntax element. There are a number of other disambiguation techniques. IMHO none of the techniques is a clearly better than any others.

According to (c) ANSI ISO/IEC 14882:2003, page 127:

Linkage specifications nest. When linkage specifications nest, the innermost one determines the language. A linkage specification does not establish a scope. A linkage-specification shall occur only in namespace scope (3.3). In a linkage-specification, the specified language linkage applies to the function types of all function declarators, function names, and variable names introduced by the declaration(s).

extern "C" void f1(void(*pf)(int));
// the name f1 and its function type have C language
// linkage; pf is a pointer to a C function

extern "C" typedef void FUNC();
FUNC f2;
// the name f2 has C++ language linkage and the
// function's type has C language linkage

extern "C" FUNC f3;
// the name of function f3 and the function's type
// have C language linkage

void (*pf2)(FUNC*);
// the name of the variable pf2 has C++ linkage and
// the type of pf2 is pointer to C++ function that
// takes one parameter of type pointer to C function

What does all this mean? For example, what linkage does the f2() function have, C or C++ language linkage?

As pointed out by @Johannes Schaub, there is no real explanation of what this means in the Standard so it can be interpreted differently in different compilers.

Please explain the differences in the object file:

• a function's name with C language linkage and C++ language linkage.
• a function's type with C language linkage and C++ language linkage.

Language linkage is the term used for linkage between C++ and non-C++ code fragments. Typically, in a C++ program, all function names, function types and even variable names have the default C++ language linkage.

A C++ object code can be linked to another object code which is produced using some other source language (like C) using a predefined linkage specifier.

As you must be aware of the concept of name mangling, which encodes function names, function types and variable names so as to generate a unique name for them. This allows the linker to differentiate between common names (as in the case of function overloading). Name mangling is not desirable when linking C modules with libraries or object files compiled with a C++ compiler. To prevent name mangling for such cases, linkage specifiers are used. In this case, extern "C" is the linkage specifier. Let's take an example (c++ code mentioned here):

typedef int (*pfun)(int);  // line 1
extern "C" void foo(pfun); // line 2
extern "C" int g(int)      // line 3
...
foo( g ); // Error!        // line 5

Line 1 declares pfun to point to a C++ function, because it lacks a linkage specifier.

Line 2 therefore declares foo to be a C function that takes a pointer to a C++ function.

Line 5 attempts to call foo with a pointer to g, a C function, a type mis-match.

Diff in function name linkage:

Let's take two different files:

One with extern "c" linkage (file1.cpp):

#include <iostream>
using namespace std;

extern "C"
{
void foo (int a, int b)
{
    cout << "here";
}
}

int main ()
{
    foo (10,20);
    return 0;
}

One without extern "c" linkage (file2.cpp):

#include <iostream>
using namespace std;

void foo (int a, int b)
{
    cout << "here";
}

int main ()
{
    foo (10,20);
    return 0;
}

Now compile these two and check the objdump.

# g++ file1.cpp -o file1
# objdump -Dx file1

# g++ file2.cpp -o file2
# objdump -Dx file2

With extern "C" linkage, there is no name mangling for the function foo. So any program that is using it (assuming we make a shared lib out of it) can directly call foo (with helper functions like dlsym and dlopen) with out considering any name mangling effects.

0000000000400774 <foo>:
  400774:   55                      push   %rbp
  400775:   48 89 e5                mov    %rsp,%rbp
....
....
  400791:   c9                      leaveq 
  400792:   c3                      retq   

0000000000400793 <main>:
  400793:   55                      push   %rbp
  400794:   48 89 e5                mov    %rsp,%rbp
  400797:   be 14 00 00 00          mov    $0x14,%esi
  40079c:   bf 0a 00 00 00          mov    $0xa,%edi
  4007a1:   e8 ce ff ff ff          callq  400774 <foo>
  4007a6:   b8 00 00 00 00          mov    $0x0,%eax
  4007ab:   c9                      leaveq 

On the other hand, when no extern "C" is being used, func: foo is mangled with some predefined rules (known to compiler/linker being used) and so an application can not directly call it from it specifying the name as foo. You can however call it with the mangled name (_Z3fooii in this case) if you want, but nobody use it for the obvious reason.

0000000000400774 <_Z3fooii>:
  400774:   55                      push   %rbp
  400775:   48 89 e5                mov    %rsp,%rbp
 ...
...
  400791:   c9                      leaveq 
  400792:   c3                      retq   

0000000000400793 <main>:
  400793:   55                      push   %rbp
  400794:   48 89 e5                mov    %rsp,%rbp
  400797:   be 14 00 00 00          mov    $0x14,%esi
  40079c:   bf 0a 00 00 00          mov    $0xa,%edi
  4007a1:   e8 ce ff ff ff          callq  400774 <_Z3fooii>
  4007a6:   b8 00 00 00 00          mov    $0x0,%eax
  4007ab:   c9                      leaveq 
  4007ac:   c3                      retq   

This page is also a good read for this particular topic.

A nice and clearly explained article about calling convention: http://www.codeproject.com/KB/cpp/calling_conventions_demystified.aspx

I had my fair chance of getting through the python management of modules, and every time is a challenge: packaging is not what people do every day, and it becomes a burden to learn, and a burden to remember, even when you actually do it, since this happens normally once.

I would like to collect here the definitive overview of how import, package management and distribution works in python, so that this question becomes the definitive explanation for all the magic that happens under the hood. Although I understand the broad level of the question, these things are so intertwined that any focused answer will not solve the main problem: understand how all works, what is outdated, what is current, what are just alternatives for the same task, what are the quirks.

The list of keywords to refer to is the following, but this is just a sample out of the bunch. There's a lot more and you are welcome to add additional details.

• PyPI
• setuptools / Distribute
• distutils
• eggs
• egg-link
• pip
• zipimport
• site.py
• site-packages
• .pth files
• virtualenv
• handling of compiled modules in eggs (with and without installation via easy_install)
• use of get_data()
• pypm
• bento
• PEP 376
• the cheese shop
• eggsecutable

Linking to other answers is probably a good idea. As I said, this question is for the high-level overview.

For the most part, this is an attempt to look at the packaging/distribution side, not the mechanics of import. Unfortunately, packaging is the place where Python provides way more than one way to do it. I'm just trying to get the ball rolling, hopefully others will help fill what I miss or point out mistakes.

First of all there's some messy terminology here. A directory containing an __init__.py file is a package. However, most of what we're talking about here are specific versions of packages published on PyPI, one of it's mirrors, or in a vendor specific package management system like Debian's Apt, Redhat's Yum, Fink, Macports, Homebrew, or ActiveState's pypm.

These published packages are what folks are trying to call "Distributions" going forward in an attempt to use "Package" only as the Python language construct. You can see some of that usage in PEP-376 PEP-376.

Now, your list of keywords relate to several different aspects of the Python Ecosystem:

Finding and publishing python distributions:

• PyPI (aka the cheese shop)
• PyPI Mirrors
• Various package management tools / systems: apt, yum, fink, macports, homebrew
• pypm (ActiveState's alternative to PyPI)

The above are all services that provide a place to publish Python distributions in various formats. Some, like PyPI mirrors and apt / yum repositories can be run on your local machine or within your companies network but folks typically use the official ones. Most, if not all provide a tool (or multiple tools in the case of PyPI) to help find and download distributions.

Libraries used to create and install distributions:

• setuptools / Distribute
• distutils

Distutils is the standard infrastructure on which Python packages are compiled and built into distributions. There's a ton of functionality in distutils but most folks just know:

from distutils.core import setup

setup(name='Distutils',
      version='1.0',
      description='Python Distribution Utilities',
      author='Greg Ward',
      author_email='gward@python.net',
      url='http://www.python.org/sigs/distutils-sig/',
      packages=['distutils', 'distutils.command'],
 )

And to some extent that's a most of what you need. With the prior 9 lines of code you have enough information to install a pure Python package and also the minimal metadata required to publish that package a distribution on PyPI.

Setuptools provides the hooks necessary to support the Egg format and all of it's features and foibles. Distribute is an alternative to Setuptools that adds some features while trying to be mostly backwards compatible. I believe Distribute is going to be included in Python 3 as the successor to Distutil's from distutils.core import setup.

Both Setuptools and Distribute provide a custom version of the distutils setup command that does useful things like support the Egg format.

Python Distribution Formats:

• source
• eggs

Distributions are typically provided either as source archives (tarball or zipfile). The standard way to install a source distribution is by downloading and uncompressing the archive and then running the setup.py file inside.

For example, the following will download, build, and install the Pygments syntax highlighting library:

curl -O -G http://pypi.python.org/packages/source/P/Pygments/Pygments-1.4.tar.gz
tar -zxvf Pygments-1.4.tar.gz
cd Pygments-1.4
python setup.py build
sudo python setup.py install

Alternatively you can download the Egg file and install it. Typically this is accomplished by using easy_install or pip:

sudo easy_install pygments
or 
sudo pip install pygments

Eggs were inspired by Java's Jarfiles and they have quite a few features you should read about here

Python Package Formats:

• uncompressed directories
• zipimport (zip compressed directories)

A normal python package is just a directory containing an __init__.py file and an arbitrary number of additional modules or sub-packages. Python also has support for finding and loading source code within *.zip files as long as they are included on the PYTHONPATH (sys.path).

Installing Python Packages:

• easy_install: the original egg installation tool, depends on setuptools
• pip: currently the most popular way to install python packages. Similar to easy_install but more flexible and has some nice features like requirements files to help document dependencies and reproduce deployments.
• pypm, apt, yum, fink, etc

Environment Management / Automated Deployment:

• bento
• buildout
• virtualenv (and virtualenvwrapper)

The above tools are used to help automate and manage dependencies for a Python project. Basically they give you tools to describe what distributions your application requires and automate the installation of those specific versions of your dependencies.

Locations of Packages / Distributions:

• site-packages
• PYTHONPATH
• the current working directory (depends on your OS and environment settings)

By default, installing a python distribution is going to drop it into the site-packages directory. That directory is usually something like /usr/lib/pythonX.Y/site-packages.

A simple programmatic way to find your site-packages directory:

from distuils import sysconfig
print sysconfig.get_python_lib()

Ways to modify your PYTHONPATH:

Python's import statement will only find packages that are located in one of the directories included in your PYTHONPATH.

You can inspect and change your path from within Python by accessing:

import sys
print sys.path
sys.path.append("/home/myname/lib")

Besides that, you can set the PYTHONPATH environment variable like you would any other environment variable on your OS or you could use:

• .pth files: *.pth files located in directories that are already on your PYTHONPATH are read and each line of the *.pth file is added to your PYTHONPATH. Basically any time you would copy a package into a directory on your PYTHONPATH you could instead create a mypackages.pth. Read more about *.pth files: site module
• egg-link files: Internal structure of python eggs they are a cross platform alternative to symbolic links. Creating an egg link file is similar to creating a pth file.
• site.py modifications

To add the above /home/myname/lib to site-packages with a *.pth file you'd create a *.pth file. The name of the file doesn't matter but you should still probably choose something sensible.

Let's create myname.pth:

# myname.pth
/home/myname/lib

That's it. Drop that into sysconfig.get_python_lib() on your system or any other directory in your PYTHONPATH and /home/myname/lib will be added to the path.

I just came across a neat CSS trick. Check out the fiddle...

http://jsfiddle.net/duZAx/1/

This creates a little arrow/triangle-like effect, a "tooltip tail". This blows my mind! How does this work?!

---

CHALLENGE

Now, I pose a challenge to the Stack Overflow community! Is there a way to extend this CSS trick to create an effect as follows...

        

I'm not worried about the shadow yet, but this is an interesting problem :) Can this be done using only CSS?

---

UPDATE TO THE CHALLENGE

I figured out the initial challenge. Here's the fiddle...

http://jsfiddle.net/duZAx/7/

Now, I pose a new challenge. How do I exactly mimic the little picture above using pure CSS, including the shadow and having it cross-browser compatible?

---

UPDATE TO THE NEW CHALLENGE

Here's my solution after a combination of the answers below. I haven't tested it across multiple browsers, but it looks great in Chrome.

http://jsfiddle.net/MZXCj/8/

---

Thanks,
Hristo

Here's an example with a box-shadow, all latest version browsers should support this

http://jsfiddle.net/MZXCj/1/

HTML:

<div id="toolTip">
    <p>i can haz css tooltip</p>
    <div id="tailShadow"></div>
    <div id="tail1"></div>
    <div id="tail2"></div>
</div>

CSS:

body {font-family:Helvetica,Arial,sans-serif;}

#toolTip {
    position:relative;
}

#toolTip p {
    padding:10px;
    background-color:#f9f9f9;
    border:solid 1px #a0c7ff;
    -moz-border-radius:5px;-ie-border-radius:5px;-webkit-border-radius:5px;-o-border-radius:5px;border-radius:5px;
}

#tailShadow {
    position:absolute;
    bottom:-8px;
    left:28px;
    width:0;height:0;
    border:solid 2px #fff;
    box-shadow:0 0 10px 1px #555;
}

#tail1 {
    position:absolute;
    bottom:-20px;
    left:20px;
    width:0;height:0;
    border-color:#a0c7ff transparent transparent transparent;
    border-width:10px;
    border-style:solid;
}

#tail2 {
    position:absolute;
    bottom:-18px;
    left:20px;
    width:0;height:0;
    border-color:#f9f9f9 transparent transparent transparent;
    border-width:10px;
    border-style:solid;
}

Consider the following simple manipulation over a collection:

static List<int> x = new List<int>() { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
var result = x.Where(i => i % 2 == 0).Where(i => i > 5);

Now let's use Expressions. The following code is roughly equivalent:

static void UsingLambda() {
    Func<IEnumerable<int>, IEnumerable<int>> lambda = l => l.Where(i => i % 2 == 0).Where(i => i > 5);
    var t0 = DateTime.Now.Ticks;
    for (int j = 1; j < MAX; j++) 
        var sss = lambda(x).ToList();

    var tn = DateTime.Now.Ticks;
    Console.WriteLine("Using lambda: {0}", tn - t0);
}

But I want to build the expression on-the-fly, so here's a new test:

static void UsingCompiledExpression() {
    var f1 = (Expression<Func<IEnumerable<int>, IEnumerable<int>>>)(l => l.Where(i => i % 2 == 0));
    var f2 = (Expression<Func<IEnumerable<int>, IEnumerable<int>>>)(l => l.Where(i => i > 5));
    var argX = Expression.Parameter(typeof(IEnumerable<int>), "x");
    var f3 = Expression.Invoke(f2, Expression.Invoke(f1, argX));
    var f = Expression.Lambda<Func<IEnumerable<int>, IEnumerable<int>>>(f3, argX);

    var c3 = f.Compile();

    var t0 = DateTime.Now.Ticks;
    for (int j = 1; j < MAX; j++) 
        var sss = c3(x).ToList();

    var tn = DateTime.Now.Ticks;
    Console.WriteLine("Using lambda compiled: {0}", tn - t0);
}

Of course it isn't exactly like the above, so to be fair, I modify the first one slightly:

static void UsingLambdaCombined() {
    Func<IEnumerable<int>, IEnumerable<int>> f1 = l => l.Where(i => i % 2 == 0);
    Func<IEnumerable<int>, IEnumerable<int>> f2 = l => l.Where(i => i > 5);
    Func<IEnumerable<int>, IEnumerable<int>> lambdaCombined = l => f2(f1(l));
    var t0 = DateTime.Now.Ticks;
    for (int j = 1; j < MAX; j++) 
        var sss = lambdaCombined(x).ToList();

    var tn = DateTime.Now.Ticks;
    Console.WriteLine("Using lambda combined: {0}", tn - t0);
}

Now comes the results for MAX = 100000, VS2008, debugging ON:

Using lambda compiled: 23437500
Using lambda:           1250000
Using lambda combined:  1406250

And with debugging OFF:

Using lambda compiled: 21718750
Using lambda:            937500
Using lambda combined:  1093750

Surprise. The compiled expression is roughly 17x slower than the other alternatives. Now here comes the questions:

1. Am I comparing non-equivalent expressions?
2. Is there a mechanism to make .NET "optimize" the compiled expression?
3. How do I express the same chain call l.Where(i => i % 2 == 0).Where(i => i > 5); programatically?

---

Some more statistics. Visual Studio 2010, debugging ON, optimizations OFF:

Using lambda:           1093974
Using lambda compiled: 15315636
Using lambda combined:   781410

Debugging ON, optimizations ON:

Using lambda:            781305
Using lambda compiled: 15469839
Using lambda combined:   468783

Debugging OFF, optimizations ON:

Using lambda:            625020
Using lambda compiled: 14687970
Using lambda combined:   468765

New Surprise. Switching from VS2008 (C#3) to VS2010 (C#4), makes the UsingLambdaCombined faster than the native lambda.

---

Ok, I've found a way to improve the lambda compiled performance by more than an order of magnitude. Here's a tip; after running the profiler, 92% of the time is spent on:

System.Reflection.Emit.DynamicMethod.CreateDelegate(class System.Type, object)

Hmmmm... Why is it creating a new delegate in every iteration? I'm not sure, but the solution follows in a separate post.

Could it be that the inner lambdas are not being compiled?!? Here's a proof of concept:

static void UsingCompiledExpressionWithMethodCall() {
        var where = typeof(Enumerable).GetMember("Where").First() as System.Reflection.MethodInfo;
        where = where.MakeGenericMethod(typeof(int));
        var l = Expression.Parameter(typeof(IEnumerable<int>), "l");
        var arg0 = Expression.Parameter(typeof(int), "i");
        var lambda0 = Expression.Lambda<Func<int, bool>>(
            Expression.Equal(Expression.Modulo(arg0, Expression.Constant(2)),
                             Expression.Constant(0)), arg0).Compile();
        var c1 = Expression.Call(where, l, Expression.Constant(lambda0));
        var arg1 = Expression.Parameter(typeof(int), "i");
        var lambda1 = Expression.Lambda<Func<int, bool>>(Expression.GreaterThan(arg1, Expression.Constant(5)), arg1).Compile();
        var c2 = Expression.Call(where, c1, Expression.Constant(lambda1));

        var f = Expression.Lambda<Func<IEnumerable<int>, IEnumerable<int>>>(c2, l);

        var c3 = f.Compile();

        var t0 = DateTime.Now.Ticks;
        for (int j = 1; j < MAX; j++)
        {
            var sss = c3(x).ToList();
        }

        var tn = DateTime.Now.Ticks;
        Console.WriteLine("Using lambda compiled with MethodCall: {0}", tn - t0);
    }

And now the timings are:

Using lambda:                            625020
Using lambda compiled:                 14687970
Using lambda combined:                   468765
Using lambda compiled with MethodCall:   468765

Woot! Not only it is fast, it is faster than the native lambda. (Scratch head).

---

Of course the above code is simply too painful to write. Let's do some simple magic:

static void UsingCompiledConstantExpressions() {
    var f1 = (Func<IEnumerable<int>, IEnumerable<int>>)(l => l.Where(i => i % 2 == 0));
    var f2 = (Func<IEnumerable<int>, IEnumerable<int>>)(l => l.Where(i => i > 5));
    var argX = Expression.Parameter(typeof(IEnumerable<int>), "x");
    var f3 = Expression.Invoke(Expression.Constant(f2), Expression.Invoke(Expression.Constant(f1), argX));
    var f = Expression.Lambda<Func<IEnumerable<int>, IEnumerable<int>>>(f3, argX);

    var c3 = f.Compile();

    var t0 = DateTime.Now.Ticks;
    for (int j = 1; j < MAX; j++) {
        var sss = c3(x).ToList();
    }

    var tn = DateTime.Now.Ticks;
    Console.WriteLine("Using lambda compiled constant: {0}", tn - t0);
}

And some timings, VS2010, Optimizations ON, Debugging OFF:

Using lambda:                            781260
Using lambda compiled:                 14687970
Using lambda combined:                   468756
Using lambda compiled with MethodCall:   468756
Using lambda compiled constant:          468756

Now you could argue that I'm not generating the whole expression dynamically; just the chaining invocations. But in the above example I generate the whole expression. And the timings match. This is just a shortcut to write less code.

---

From my understanding, what is going on is that the .Compile() method does not propagate the compilations to inner lambdas, and thus the constant invocation of CreateDelegate. But to truly understand this, I would love to have a .NET guru comment a little about the internal stuff going on.

And why, oh why is this now faster than a native lambda!?

Below is a simple test fixture. It succeeds in Debug builds and fails in Release builds (VS2010, .NET4 solution, x64):

[TestFixture]
public sealed class Test
{
    [Test]
    public void TestChecker()
    {
        var checker = new Checker();
        Assert.That(checker.IsDateTime(DateTime.Now), Is.True);
    }
}

public class Checker
{
    public bool IsDateTime(object o)
    {
        return o is DateTime;
    }
}

It seems code optimization wreaks some havoc; if I disable it on the Release build, it works as well. That was rather puzzling to me. Below, I've used ILDASM to disassemble the 2 versions of the build:

Debug IL:

.method public hidebysig instance bool IsDateTime(object o) cil managed
{
  // Code size       15 (0xf)
  .maxstack  2
  .locals init (bool V_0)
  IL_0000:  nop
  IL_0001:  ldarg.1
  IL_0002:  isinst     [mscorlib]System.DateTime
  IL_0007:  ldnull
  IL_0008:  cgt.un
  IL_000a:  stloc.0
  IL_000b:  br.s       IL_000d
  IL_000d:  ldloc.0
  IL_000e:  ret
} // end of method Validator::IsValid

Release IL:

.method public hidebysig instance bool IsDateTime(object o) cil managed
{
  // Code size       10 (0xa)
  .maxstack  8
  IL_0000:  ldarg.1
  IL_0001:  isinst     [mscorlib]System.DateTime
  IL_0006:  ldnull
  IL_0007:  cgt.un
  IL_0009:  ret
} // end of method Validator::IsValid

It seems a store and load is optimized away. Targeting earlier versions of the .NET framework made the problem go away, but that may just be a fluke. I found this behaviour somewhat unnerving, can anybody explain why the compiler would think it safe to do an optimization that produces different observable behaviour?

Thanks in advance.

This bug already came up in this SO question by Jacob Stanley. Jacob has already reported the bug, and Microsoft has confirmed that it is indeed a bug in the CLR JIT. Microsoft had this to say:

This bug will be fixed in a future version of the runtime. I'm afraid it's too early to tell if that will be in a service pack or the next major release.

Thank you again for reporting the issue.

You sould be able to work around the bug by adding the following attribute to TestChecker():

[MethodImpl(MethodImplOptions.NoInlining)]

While browsing through gcc's current implementation of new C++11 headers, I stumbled upon "......" token. You can check, that the following code compiles fine [via ideone.com].

template <typename T>
struct X
{ /* ... */ };

template <typename T, typename ... U>
struct X<T(U......)> // this line is the important one
{ /* ... */ };

So, what is the meaning of this token?

edit: Looks like SO trimmed "......" in question title to "...", I did really mean "......" . :)

Every instance of that oddity is paired with a case of a regular single ellipsis.

  template<typename _Res, typename... _ArgTypes>
    struct _Weak_result_type_impl<_Res(_ArgTypes...)>
    { typedef _Res result_type; };

  template<typename _Res, typename... _ArgTypes>
    struct _Weak_result_type_impl<_Res(_ArgTypes......)>
    { typedef _Res result_type; };

  template<typename _Res, typename... _ArgTypes>
    struct _Weak_result_type_impl<_Res(_ArgTypes...) const>
    { typedef _Res result_type; };

  template<typename _Res, typename... _ArgTypes>
    struct _Weak_result_type_impl<_Res(_ArgTypes......) const>
    { typedef _Res result_type; };

My guess is that the double ellipsis is similar in meaning to _ArgTypes..., ..., i.e. a variadic template expansion followed by a C-style varargs list.

Here's a test supporting that theory… I think we have a new winner for worst pseudo-operator ever.

Edit: This does appear to be conformant. §8.3.5/3 describes one way to form the parameter list as

parameter-declaration-list_opt ..._opt

So the double-ellipsis is formed by a parameter-declaration-list ending with a parameter pack, followed by another ellipsis.

The comma is purely optional; §8.3.5/4 does say

Where syntactically correct and where “...” is not part of an abstract-declarator, “, ...” is synonymous with “...”.

This is within an abstract-declarator, [edit] but Johannes makes a good point that they are referring to an abstract-declarator within a parameter-declaration. I wonder why they didn't say "part of a parameter-declaration," and why that sentence isn't just an informative note…

Furthermore, va_begin() in <cstdarg> requires a parameter before the varargs list, so the prototype f(...) specifically allowed by C++ is useless. Cross-referencing with C99, it is illegal in plain C. So, this is most bizarre.

Usage note

By request, here is a demonstration of the double ellipsis:

#include <cstdio>
#include <string>

template< typename T >
T const &printf_helper( T const &x )
    { return x; }

char const *printf_helper( std::string const &x )
    { return x.c_str(); }

template< typename ... Req, typename ... Given >
int wrap_printf( int (*fn)( Req... ... ), Given ... args ) {
    return fn( printf_helper( args ) ... );
}

int main() {
    wrap_printf( &std::printf, "Hello %s\n", std::string( "world!" ) );
    wrap_printf( &std::fprintf, stderr, std::string( "Error %d" ), 5 );
}

I have the task of packaging and shipping a commercial application bundle, which will include:

1. a python library (developed by us)
2. some python programs depending on the library above
3. additional libraries not developed by us, but which are dependencies of our library.
4. a complete python installation (python 2.6)
5. additional stuff, libs and programs in other languages. Not a concern here, as they are not hooked into the above machinery, and the current shipping process works already.

The bundle is shipped to Linux, OSX and Windows. On Linux, it's distributed as a simple tar.gz. The user just unpacks the tar.gz and source a provided bash script in .bashrc, so that the environment is correctly set. On mac, it's a dmg. On windows, I have no idea. The windows guy is not here today, but what I see is that an exe is created somehow.

I will now explain in more detail the above points.

Our Python Library

We don't want to give out sources, so we want to provide only compiled python files. A better strategy to make them even more tamper-proof is welcome, even if it involves some deep hacking (e.g. I once saw magic done importing stuff from a .zip which was "corrupted" ad-hoc). The library at the moment does not have C level code or similar platform dependent code, but this is going to change soon. We will therefore have to provide platform-specific compiled .so together with the pyc.

Clearly, this library will be shipped in the package, together with the rest of our application. It will therefore be installed on the downloaded bundle. For this reason, it must be fully relocatable, and the user must in some way (either manually or via our env script) add the location of the untarred package to PYTHONPATH, so that the interpreter can find it.

Our Python Programs

We will ship applications in our bundle, and these applications will depend on our library. The code of these applications must be either visible by the user (so that he can learn how to use the library interface), or not visible (for those utilities we want to keep closed-source), so a double approach is called for.

Additional Libraries

Our library depends on 3rd party libraries we will have to ship, so that the user is up and running without any dependency hunting. Clearly, these libraries will be installed by us in the bundle, but we must hope these don't store the install path somewhere during the build, because that would make them non relocatable.

Our python

We will ship our version of python, which we assume the user will run in order to access our script. This is because we want to be sure of the python version running. Also, we may tinker a bit the executable or the standard library. We may have a concern about the interaction of this python with the standard python, and if the user wants a specific library on our python it will have to install it within our bundled package, and not on the standard place for libraries.

Request

I need to make my mind around this task. I've seen it done, but never done it personally, so I need your point of view. What I presented above is how I think things should work, according to how things are working right now, but it may be wrong. Any hint, quirk, suggestion, or strategy for a successful deployment is welcome. Given the complexity of the question, I already announce a high bounty on it, according to the best answer I can get.

This is not a complete answer but just a bunch of ideas. I wrote an installer for a client that incorporated some ideas that might be useful to you.

It was Linux only so I focussed on just that. We needed to ship specific custom versions of mySQL, lighttpd, python, memcached, a few 3rd party Python modules and some custom scripts. We needed to launch all these services without any problems and let the user control them using regular initscripts. It should work fine on a bunch of popular distros and therefore shouldn't rely on distro specific stuff.

What I did was as follows.

1. Created a 500MB (I'm don't recollect the size) file and formatted it as an ext3fs file system.
2. Mounted it at a point using a loopback device.
3. Ran deb-bootstrap on the mountpoint to create a custom Debian install.
4. Chrooted inside the partition and then ran a bunch of scripts which did an apt-get install on all our dependencies, installed all the eggs and other packages which were necessary for the app, installed the app itself in /opt (inside the chroot), installed supervisord (to do process management) and set things up. Now, this partition was a completely self contained Linux filesystem that contained the application and everything needed to run it. You could dump it anywhere, chroot inside it and launch the app. The only dependency it had with the outside world were the ports it would use for its services and the supervisord control socket. This was the main point. We were able to include exactly what we needed (compiled files, .pycs only etc.) for a few of the applications and didn't have to bother with any limitations in standard installation tools.
5. After this, we packaged a few extra scripts that would go into the external operating system. These were custom made for each distro that we would have to support. This part was distro specific. There were scripts that would go into /etc/init.d and some scripts that would setup the database and stuff at the beginning.
6. We then created an archive of the entire filesystem using makeself. It would checksum stuff and all that and provide a self extracting archive which if run would untar the whole thing into /opt on the host machine, chroot inside the directory and run a setup script that would ask the user a few questions like db username/password etc. and set things up. After that, it would fetch the scripts I mentioned in step 5 and put them on the host OS.

The initscripts would simply chroot into the partition and start supervisord. It would then take care of launching all the services we cared about. Shutting down the application was simply a matter of connecting to running supervisord and running a command. We wrapped this in the initscript so that the user experience was UNIX like.

Now, we'd give clients the self extracting .run file. They'd run it, get asked a few questions and it would create a directory under /opt which contained our app and all it's dependencies. The init scripts would be modified to start our app on bootup and things would work as expected.

I think step 4 gives you the freedom to install whatever you want, however you want so that things would work fine.