agesys-dev team mailing list archive

[spencer rathbun <srathbun@xxxxxxxxxxxxxxxxx>]

Public bug reported:

When inserting data into a test Mysql db, I ran into the following
issue:

  File "C:\development\taxStatements\tax converter\taxes.py", line 60, in main
    db.insertRec(rec)
  File "C:\development\taxStatements\tax converter\dbInserter.py", line 1735, in insertRec
    self.insertManRec(rec, firstrec)
  File "C:\development\taxStatements\tax converter\dbInserter.py", line 1772, in insertManRec
    rec['Owner_City'], rec['Owner_State'], rec['Owner_Zip_Code'], rec['Owner_Country'], rec['Ow
ner_Attention_Line'] \
  File "C:\development\taxStatements\tax converter\dbInserter.py", line 1718, in get_or_create_
man
    instance = self.session.query(model).get((name, st, st_num, st_dir, st_suf, city, state, zi
p_code, country, attn_line))
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 775, in get
    return self._load_on_ident(key)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2512, in _load_on_ident
    return q.one()
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2184, in one
    ret = list(self)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2242, in _execute_and_inst
ances
    result = conn.execute(querycontext.statement, self._params)
  File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1449, in execute
    params)
  File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1584, in _execute_clause
element
    compiled_sql, distilled_params
  File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1691, in _execute_contex
t
    context)
  File "C:\Python27\lib\site-packages\sqlalchemy\dialects\mysql\oursql.py", line 96, in do_exec
ute
    cursor.execute(statement, parameters)
  File "cursor.pyx", line 122, in oursql.Cursor.execute (oursqlx\oursql.c:15747)
  File "statement.pyx", line 318, in oursql._Statement.execute (oursqlx\oursql.c:9241)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 13: ordinal not in range(1
28)

I'm using Python 2.7.2, Sqlalchemy 7.8, and oursql 9.3 on a Win 7 64bit
system. I'm reading in a data file in a simple for loop, doing no
encoding to the data. I convert each line into a dictionary, then create
a new object with the sqlalchemy orm. It converts that object into an
insert statement for the oursql library, which fails with this encoding
issue when I have non-ascii elements in my data.

Line 318:
http://bazaar.launchpad.net/~jerub/oursql/master/view/head:/oursqlx/statement.pyx#L318

Looking at the source, it seems that the cursor just assumes that it
will receive unicode objects, not strings. Since this check is inside of
the use_unicode if statement, disabling that option avoids the problem.
However, data is lost in this case.

This answer on SO, http://stackoverflow.com/a/11544648/724357, seems to
indicate that decoding the input strings with the proper encoding will
fix the issue. Decoding does solve this data issue.

Can the documentation be updated to reflect that use_unicode=0 may cause
data loss, and that use_unicode=1 expects a unicode object be passed in,
not a string?

** Affects: oursql
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Agesys
Team, which is subscribed to oursql.
https://bugs.launchpad.net/bugs/1033640

Title:
  ascii decode problem on execute

Status in oursql:
  New

Bug description:
  When inserting data into a test Mysql db, I ran into the following
  issue:

    File "C:\development\taxStatements\tax converter\taxes.py", line 60, in main
      db.insertRec(rec)
    File "C:\development\taxStatements\tax converter\dbInserter.py", line 1735, in insertRec
      self.insertManRec(rec, firstrec)
    File "C:\development\taxStatements\tax converter\dbInserter.py", line 1772, in insertManRec
      rec['Owner_City'], rec['Owner_State'], rec['Owner_Zip_Code'], rec['Owner_Country'], rec['Ow
  ner_Attention_Line'] \
    File "C:\development\taxStatements\tax converter\dbInserter.py", line 1718, in get_or_create_
  man
      instance = self.session.query(model).get((name, st, st_num, st_dir, st_suf, city, state, zi
  p_code, country, attn_line))
    File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 775, in get
      return self._load_on_ident(key)
    File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2512, in _load_on_ident
      return q.one()
    File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2184, in one
      ret = list(self)
    File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2227, in __iter__
      return self._execute_and_instances(context)
    File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2242, in _execute_and_inst
  ances
      result = conn.execute(querycontext.statement, self._params)
    File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1449, in execute
      params)
    File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1584, in _execute_clause
  element
      compiled_sql, distilled_params
    File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1691, in _execute_contex
  t
      context)
    File "C:\Python27\lib\site-packages\sqlalchemy\dialects\mysql\oursql.py", line 96, in do_exec
  ute
      cursor.execute(statement, parameters)
    File "cursor.pyx", line 122, in oursql.Cursor.execute (oursqlx\oursql.c:15747)
    File "statement.pyx", line 318, in oursql._Statement.execute (oursqlx\oursql.c:9241)
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 13: ordinal not in range(1
  28)

  I'm using Python 2.7.2, Sqlalchemy 7.8, and oursql 9.3 on a Win 7
  64bit system. I'm reading in a data file in a simple for loop, doing
  no encoding to the data. I convert each line into a dictionary, then
  create a new object with the sqlalchemy orm. It converts that object
  into an insert statement for the oursql library, which fails with this
  encoding issue when I have non-ascii elements in my data.

  Line 318:
  http://bazaar.launchpad.net/~jerub/oursql/master/view/head:/oursqlx/statement.pyx#L318

  Looking at the source, it seems that the cursor just assumes that it
  will receive unicode objects, not strings. Since this check is inside
  of the use_unicode if statement, disabling that option avoids the
  problem. However, data is lost in this case.

  This answer on SO, http://stackoverflow.com/a/11544648/724357, seems
  to indicate that decoding the input strings with the proper encoding
  will fix the issue. Decoding does solve this data issue.

  Can the documentation be updated to reflect that use_unicode=0 may
  cause data loss, and that use_unicode=1 expects a unicode object be
  passed in, not a string?

To manage notifications about this bug go to:
https://bugs.launchpad.net/oursql/+bug/1033640/+subscriptions