Answer a question

Given the following code run from the Python interpreter:

import sys
sys.getdefaultencoding()
my_string = '\xc3\xa9'
my_string = unicode(my_string, 'utf-8')
my_string
print my_string

With Python 2.6.1 running on a mac, everything works fine:

$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> my_string = '\xc3\xa9'
>>> my_string = unicode(my_string, 'utf-8')
>>> my_string
u'\xe9'
>>> print my_string
é
>>> 

With Python 2.6.5 running on Ubuntu 10.04 LTS, it fails:

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> my_string = '\xc3\xa9'
>>> my_string = unicode(my_string, 'utf-8')
>>> my_string
u'\xe9'
>>> print my_string
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)
>>> 

Has something changed between Python 2.6.1 and 2.6.5 that requires different handling of unicode strings? Or does this have to do with something misconfigured in my (default Ubuntu server 10.04 LTS) linux environment?

Edit: Both environments have LANG=en_US.UTF-8

Answers

I can reproduce the error with the command:

$ PYTHONIOENCODING=ascii python -c'print "\xc3\xa9".decode("utf-8")'

Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0:\ ordinal not in range(128)

sys.getdefaultencoding() is 'ascii' and not very useful by default.

Try using your console encoding:

$ PYTHONIOENCODING=utf-8 python -c'print "\xc3\xa9".decode("utf-8")'
é

or

$ python -c'import locale; print "\xc3\xa9".decode("utf-8").encode(
> locale.getpreferredencoding())'
é

Check sys.stdout.encoding:

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding'
True UTF-8

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding' | cat
False None

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding' >/tmp/out
$ cat /tmp/out
False None

If sys.stdout.encoding is None try to use locale.getpreferredencoding() or set PYTHONIOENCODING as shown above. See http://wiki.python.org/moin/PrintFails

If the error occurs only in the interactive Python session then look at sys.displayhook().