Best django questions in October 2010

Developing with Django+Celery without running `celeryd`?

8 votes

In development, it's a bit of a hassle to run the celeryd as well as the Django development server. Is it possible to, for example, ask celery to run tasks synchronously during development? Or something similar?

Yes you can do this by setting CELERY_ALWAYS_EAGER = True in your settings.
http://ask.github.com/celery/configuration.html#celery-always-eager

What's the meaning of '_' in python?

6 votes

When reading source code of Django, I find some statements:

class Field(object):  
    """Base class for all field types"""  
    __metaclass__ = LegacyConnection  

    # Generic field type description, usually overriden by subclasses
    def _description(self):
        return _(u'Field of type: %(field_type)s') % {
            'field_type': self.__class__.__name__
        }    
    description = property(_description) 

class AutoField(Field):
    description = _("Integer")

I know it set description as 'Integer', but don't understand the syntax: description = _("Integer").
Can some one help on it?

Please read up on Internationalization (i18n)

http://docs.djangoproject.com/en/dev/topics/i18n/

The _ is a commonly-used name for the function that translates strings to another language.

http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/#standard-translation

Also, read all of these related questions on SO:

http://stackoverflow.com/search?q=%5Bdjango%5D+i18n

How do you guys protect django admin site?

5 votes

I thought I might restrict it to show only on some IPs, but I have some freelance workers without static IPs that should be able to login to admin site. I rolled out a big project and I am looking for some ways to protect the admin site fom unwanted eyes.

If you are running it behind apache you can use one of its many modules for HTTP authentication (there are similar modules for other servers). This way the user can't even get to the login page without login in.

Another option would be to block all access from remote URL's and require users to use a VPN to access the admin pages. (I think this would be too big of a hassle)

We have a site where the admin interface is on a separate domain, it doesn't hide anything but keeps them separate.

Using a UUID as a primary key in Django models (generic relations impact)

5 votes

For a number of reasons^, I'd like to use a UUID as a primary key in some of my Django models. If I do so, will I still be able to use outside apps like "contrib.comments", "django-voting" or "django-tagging" which use generic relations via ContentType?

Using "django-voting" as an example, the Vote model looks like this:

class Vote(models.Model):
    user         = models.ForeignKey(User)
    content_type = models.ForeignKey(ContentType)
    object_id    = models.PositiveIntegerField()
    object       = generic.GenericForeignKey('content_type', 'object_id')
    vote         = models.SmallIntegerField(choices=SCORES)

This app seems to be assuming that the primary key for the model being voted on is an integer.

The built-in comments app seems to be capable of handling non-integer PKs, though:

class BaseCommentAbstractModel(models.Model):
    content_type   = models.ForeignKey(ContentType,
            verbose_name=_('content type'),
            related_name="content_type_set_for_%(class)s")
    object_pk      = models.TextField(_('object ID'))
    content_object = generic.GenericForeignKey(ct_field="content_type", fk_field="object_pk")

Is this "integer-PK-assumed" problem a common situation for third-party apps which would make using UUIDs a pain? Or, possibly, am I misreading this situation?

Is there a way to use UUIDs as primary keys in Django without causing too much trouble?


^ Some of the reasons: hiding object counts, preventing url "id crawling", using multiple servers to create non-conflicting objects, ...

A UUID primary key will cause problems not only with generic relations, but with efficiency in general: every foreign key will be significantly more expensive—both to store, and to join on—than a machine word.

However, nothing requires the UUID to be the primary key: just make it a secondary key, by supplementing your model with a uuid field with unique=True. Use the implicit primary key as normal (internal to your system), and use the UUID as your external identifier.

How to migrate a CSV file to Sqlite3 (or MySQL)? - Python

5 votes

I'm using Python in order to save the data row by row... but this is extremely slow!

The CSV contains 70million lines, and with my script I can just store 1thousand a second.


This is what my script looks like

reader = csv.reader(open('test_results.csv', 'r'))
for row in reader:
    TestResult(type=row[0], name=row[1], result=row[2]).save()

I reckon that for testing I might have to consider MySQL or PostgreSQL.

Any idea or tips? This is the first time I deal with such massive volumes of data. :)

For MySQL imports:

mysqlimport [options] db_name textfile1 [textfile2 ...]

For SQLite3 imports:

ref http://stackoverflow.com/questions/1045910/how-can-i-import-load-a-sql-or-csv-file-into-sqlite

Facebook Connect: capturing user data with django-profiles and django-socialregistration

5 votes

Either my google searching has completely left me or there's hardly any documentation/tutorials for django-socialregistration. Too bad, because it seems like a nice enough app. Through some trial-and-error, I have managed to get it mostly running on my site.

My question, using django-socialregistration how do I request permission for the facebook user's full name, current city and date of birth and store it in my UserProfile table (which is my AUTH_PROFILE_MODULE for django-profiles) in Django upon registration? Also, how do I post to the user's wall from Django once the connection is made?

Currently, when I click the "Connect with Facebook" button the facebook connection is made, a new Django user is created and the user is logged in with that Django account. However, no UserProfile is created and no facebook profile data is saved.

Any facebook connect gurus out there want to help the Django pony fly to Facebookland?

Setup:
- Django 1.2.1
- Python 2.5.2
- django-socialregistration 0.4.2
- django-registration 0.7
- django-profiles 0.2

alt text
"Kind sir, can you please help me find the magical Facebookland?"

In facebook_js.html you need to adjust the following line, by uncommenting items that you need to get from FB:

     FB.login(handleResponse/*,{perms:'publish_stream,sms,offline_access,email,read_stream,status_update,etc'}*/);

Then, in FacebookMiddleware you can extract that data from fb_user, like this:

     facebook.GraphAPI(fb_user['access_token']).get_object('me')

jquery template tags conflict with Django template!

5 votes

Today I'm trying to play with jquery-tmpl {{if}} & {{else}} statements.

<script id="mission-dialog" type="text/x-jquery-tmpl">
    <h3>${name}</h3>
    <p>${description}</p>
    <ul>
        {{each(i,cond) conditions.data}}
        <li>
            <img src="${cond.image}"/>
            <h4>${cond.name}</h4>
            <p class="status">${cond.status.value}/${cond.status.max}</p>
        </li>
        {{/each}}
    </ul>
</script>

But as you know {{ }} is reserved also for django template. So django will emit TemplateSyntaxError that it can't parse it.

How can I solve this problem?


updated:

I found a working <% raw %> custom tag (GPL) implementation from here.

http://www.holovaty.com/writing/django-two-phased-rendering/

Use the templatetag template tag to render the brackets:

{% templatetag openvariable %}each(i,cond) conditions.data{% templatetag closevariable %}

It's a bit fiddly, which is why a raw template tag has been proposed for Django 1.3.

Unit testing with django-celery?

5 votes

I am trying to come up with a testing methodology for our django-celery project. I have read the notes in the documentation, but it didn't give me a good idea of what to actually do. I am not worried about testing the tasks in the actual daemons, just the functionality of my code. Mainly I am wondering:

  1. How can we bypass task.delay() during the test (I tried setting CELERY_ALWAYS_EAGER = True but it made no difference)?
  2. How do we use the test settings that are recommended (if that is the best way) without actually changing our settings.py?
  3. Can we still use manage.py test or do we have to use a custom runner?

Overall any hints or tips for testing with celery would be very helpful.

Try setting BROKER_BACKEND=memory

(Thanks to asksol's comment. Now you can raise your accept rate ;)

Developing a URL Shortener

5 votes

I am trying to develop a URL shortener application for practice with Django. I do not understand how can I create unique strings for each long URL to use as a short URL. Something like other popular URL shorteners do. How can I do this? Is it possible to make all short urls of the same length?

  1. I do not understand how can I create unique strings for each long URL to use as a short URL. Something like other popular URL shorteners do.
    As sugerman has said, this is simple, you just create a hash table.

  2. How can I do this?
    There are dynamic ways to do this, but the simplest and most effective is to have a 2 field table in a database, which holds the hashkey and full url. Then your server, like Apache, would have the ability to redirect to the correct page.

  3. Is it possible to make all short urls of the same length?
    Yes, to a certain extent, however once you reach the maximum amount of keys, you would have to reuse/replace the short url IDs. When you set a fixed-length, then you're limiting the amount of possibilities.

My question to you:

I'm under the assumption that by URL shortener you are referring to something like jsFiddle or a pastebin in that they have something like http://jsfiddle.net/sdfj2/. Otherwise, we'd need some more clarification.

Real time apps: Socket.io vs Pusherapp

4 votes

What are the factors you would consider when choosing between Socket.io and Pusherapp?

http://Socket.io
http://pusherapp.com

Important considerations includes: integration w existing systems like rails/django, ease of setup, ease of development

Socket.io is using Ajax long polling which requires more resources. Pusherapp on the other hand is real push. Pusherapp has various libraries, they have gem for rails, wrappers for python and GAE. You can also try other services like:

  1. Pubnub
  2. Beaconpush
  3. Hookbox
  4. Pubsubhubbub (using XMPP instead of websockets)
  5. Kwwika

Form loses ability to send POST requests after 2 ajax updates

4 votes

I have an included object's form:

<form method="post" class="object_form" id="event-core-form" action="{% url save_event_core_data event.id %}" enctype="multipart/form-data">
    {{ form.as_p }}
    <p>
        <input class="object-submit" id="object-data-save" type="submit" value="Save data">
    </p>
</form>

After hitting 'submit' button I'm running this script, which submits my form via ajax, updates data and should return updated form that will be inserted back in it's place:

$("#object-data-save").livequery("click", function(e) {
    e.preventDefault();
    $(this).parents("form:first").ajaxSubmit({
        data: {"action": action},
        "success": function(data) {
            data = JSON.parse(data);
            $("#core-data").html(data["html"]);
            $("#message").show();
            $("#message").fadeIn(400).html('<span>'+data["message"]+'</span>');
            setTimeout(function(){
                $("#message").fadeOut("slow", function () {
                    $("#message").hide();
                });

            }, 1500);                
        }
    });
    return false;
});

and this runs the following function :

def event_core_data(request, event_id):
    template_name="management/core_event.html"    
    event = Event.objects.get(pk=event_id)
    form = EventForm()

    if request.method == "POST":
        form = EventForm(instance=event, data=request.POST)
        message = _("Error saving data")
        if form.is_valid():
            form.save()
            message = _(u"Data saved.")

        html =  render_to_string(template_name, RequestContext(request, {
            "form" : form,
            "event" : event,
        }))

        result = simplejson.dumps({
            "html" : html,
            "message" : message,
        }, cls = LazyEncoder)

        result = HttpResponse(result)
        logging.debug(result)
    else:
        form = EventForm(instance=event)
        result = ""
        try:
            result = render_to_string(template_name, RequestContext(request, {
                "form" : form,
                "event" : event,
            }))
        except:
            pass

    return result

After saving it once everything works as expected. But after third update my form is not inserted in the parent template. Instead I'm redirected to the edit function's url and the form is rendered as a raw html. Also, I've noticed in firebug, that when I'm being redirected - no POST is sent and a dummy 'alert' in my javascript is not fired. This is the function displaying initial state (if it's of any help):

def manage_event(request, event_id,):
    template_name = 'management/edit_event.html'

    try:
        event = Event.objects.get(pk=event_id)
    except DoesNotExist:
        url = reverse("manage_events")
        return HttpResponseRedirect(url)

    return render_to_response(template_name, RequestContext(request, {
        "core_data" : event_core_data(request, event_id),
        "event" : event,
    }))

EDIT

Here is a test link to this project, where you can see what's going on. 'event_core_data' returns the request.POST to console upon successfull update.

http://ntt.vipserv.org/manage/events/2

I'm also wondering why my date picker widgets disappear after submission. Are those things somehow connected together ?


EDIT 2

I've already tried using .post or .ajax instead of .ajaxSubmit but without any luck.

First of all, you're doing something slightly weird. You use a jQuery-plugin which is supposed to handle form submission over ajax and repopulate the fields. Still, on success, you replace all the HTML of your form with HTML from the server, negating its work.

This breaks your calendar/time widgets, as you initialize the widgets on page load, telling them to act on some page elements, which you later replace.

But this doesn't by itself break the form submission.

Firstly, you don't need a plugin for attaching events "live" if you stop replacing the form HTML. Secondly, you don't really need a plugin for that anyway as it seems the built-in live() method in jQuery should do the job (that is, if you actually do need this functionality). Thirdly, if you use plugins and they don't seem to be working properly, update to the latest version. The version you're using doesn't support the html() method in jQuery.

The livequery-plugin does its magic by overriding any jQuery-methods which might update the DOM. So when the programmer calls, f.ex, append(), it intercepts the call, calls append() for you, and then checks the document for new or disappeared elements matching your provided selector. The version you're using is not aware of html() and therefore does not intercept it.

So it works the first time as you initiate a DOM-check on page load. When that result is returned, the event is actually attached to the new submit-button because the calls to html() to set the new form and completion-message internally calls intercepted methods. Therefore, the second submission works as desired. But when the second call comes back, a jQuery cache is used internally, not calling any intercepted methods. So the event doesn't get attached to the submit-button, making it act as a regular form submission button.

To fix, stop using live-attaching for event-listening if there's not a non-apparent need for it. If there is, use the built in one or at least update your livequery plugin. Also, don't replace the whole form HTML. Again, if there's a non-apparent reason, re-initialize you calendar widget each time after setting the HTML.