pyax team mailing list archive

[Mr Python <raychorn@xxxxxxxxxxx>]

Nope, rather small data set is being queried.

The problem is related to a "bug" in which gzip fails when it is used by a 
large number of os threads - in my case it fails around 400 threads all 
hitting pyax and then pyax hits gzip.

This is less a pyax bug than a Python bug but since pyax uses gzip this is a 
pyax problem too.

I am guessing Stackless Python would not have this problem because I am 
guessing gzip is hitting a C stack limit rather than a Python limit since I 
have 1 GB RAM available when gzip complains about the lack of RAM.

This apparent bug does not have a fix other than to reduce the number of 
threads being used however you might want to code a MemoryError trap to 
notice when this condition happens and there are a bunch of threads being 
used.

It took me a while to isolate the root cause however in the meantime I 
reported the bug as it appeared to be related to pyax then later learned 
this is a pyax second-hand bug via Python's gzip module.

--------------------------------------------------
From: "Kevin Shuk" <surf@xxxxxxxxxxxxx>
Sent: Friday, August 08, 2008 1:31 PM
To: <raychorn@xxxxxxxxxxx>
Subject: [Bug 256141] Re: Python gzip lib says "MemoryError: Can't 
allocatememory for compression object"

> I'm making the assumption that you're query is resulting in a very large
> data set - many fields, some quite large and perhaps many records.
>
> Run the query, only instead of returning fields, return count() to see how 
> many rows you're expecting back.
> How many fields are you requesting in your query and how big are they?
> Do you have a UNIX instance on which you can try this query in pyax? This 
> may be exacerbated by differences in Windows' memory management.
>
> The most immediate thing you can try is to reduce the number of fields
> you're returning - exclude any that are not necessary (especially large
> text fields).
>
> Also, if the query wants to return more than 200 rows, we can try
> reducing the query batch size to the minimum of 200 rows at a time. This
> will require releasing some code that's checked in, but not yet on the
> release branch.
>
> ** Changed in: pyax
>       Status: New => Triaged
>
> -- 
> Python gzip lib says "MemoryError: Can't allocate memory for compression 
> object"
> https://bugs.launchpad.net/bugs/256141
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Salesforce.com Apex API library for Python: Triaged
>
> Bug description:
> Traceback:
>
>  File "Z:\python projects\@lib\lib\pyax\connection.py", line 614, in query
>    query_result = self._callApex(self.svc.query, query_string)
>
>  File "Z:\python projects\@lib\lib\pyax\connection.py", line 158, in 
> _callApex
>    apex_result = apex_method(*args, **kw)
>
>  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 56, in query
>    return QueryRequest(self.context, self.sessionId, soql).post()
>
>  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 315, in post
>    data=self.makeEnvelope(),
>
>  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 273, in 
> makeEnvelope
>    ("s", _envNs), ("p", _partnerNs), ("o", _sobjectNs))
>
>  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 243, in __init__
>    XmlWriter.__init__(self, context.gzip_request)
>
>  File "Z:\python projects\@lib\lib\pyax\beatbox.py", line 173, in __init__
>    self.__gzip = gzip.GzipFile(mode='wb', fileobj=self.__buf)
>
>  File "C:\Python25\lib\gzip.py", line 120, in __init__
>    0)
>
> MemoryError: Can't allocate memory for compression object
>

-- 
Python gzip lib says "MemoryError: Can't allocate memory for compression object"
https://bugs.launchpad.net/bugs/256141
You received this bug notification because you are a member of pyax
Developers, which is the registrant for pyax.