UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21. Read more.
UPDATE! The community will be go into read-only on April 19, 8am Pacific in preparation for migration on April 21.Read more.
Absent Member.
Absent Member.
1632 views

Limit for <count> parameter in ReadCursorRequest?

Hi,
The ReadCursorRequest seems to have a hard upper limit for the value of the <count> parameter. I you try to retrieve, say, a million messages from the cursor, it returns a BAD PARAMETER error.
What is the upper limit of the <count> parameter, and can it be modified somehow?
Regards,
Maarten
Labels (1)
0 Likes
7 Replies
Absent Member.
Absent Member.

The number is computed, but it is under 4000.
You want to be careful and not ask for too many
messages. The performance degrades exponentially
as the size of the SOAP response buffer grows.
You will want to keep the size of the SOAP response
buffer under 64KB. Depending on the size of each
message, you usually want to ask for 200 or less
items at a time.

Preston

>>> On Thursday, June 16, 2011 at 4:36 AM,

mdirkse<mdirkse@no-mx.forums.novell.com>
wrote:

> Hi,
> The ReadCursorRequest seems to have a hard upper limit for the value of
> the <count> parameter. I you try to retrieve, say, a million messages
> from the cursor, it returns a BAD PARAMETER error.
> What is the upper limit of the <count> parameter, and can it be
> modified somehow?
> Regards,
> Maarten

0 Likes
Absent Member.
Absent Member.

Out of curiosity, what makes it exponential instead of linear? I assume
that means it cannot just be streamed per se but has to be buffered in
memory in some form.

OTOH asking for a million items is not bright, because even if Preston
can do that with 0 performance degradation, and in 0 time, YOU will have
probably buffer that response in most client soap stacks. Do you really
have 1 million x .5 (guess .5 kb per item) 500 MB RAM to spare PER
client thread?

I can tell you from experience that even going to 100 WILL cause some
POAs to give cursor timeout errors. Sometimes as low as 25 is needed for
some POAs not to be unhappy. (though that makes a fair performance diff)


On 6/16/2011 8:59 AM, Preston Stephenson wrote:
> The number is computed, but it is under 4000.
> You want to be careful and not ask for too many
> messages. The performance degrades exponentially
> as the size of the SOAP response buffer grows.
> You will want to keep the size of the SOAP response
> buffer under 64KB. Depending on the size of each
> message, you usually want to ask for 200 or less
> items at a time.
>
> Preston
>
>>>> On Thursday, June 16, 2011 at 4:36 AM,

> mdirkse<mdirkse@no-mx.forums.novell.com>
> wrote:
>
>> Hi,
>> The ReadCursorRequest seems to have a hard upper limit for the value of
>> the<count> parameter. I you try to retrieve, say, a million messages
>> from the cursor, it returns a BAD PARAMETER error.
>> What is the upper limit of the<count> parameter, and can it be
>> modified somehow?
>> Regards,
>> Maarten


0 Likes
Absent Member.
Absent Member.

Hi Michael,
I've only got 1 client thread, and do actually have half a gig of RAM for it to use, but I take your point that's it's not the most efficient way to handle things.
Were your timeouts on 100 messages on version 7 or version 8 POAs?
Regards,
Maarten
0 Likes
Absent Member.
Absent Member.

On 6/17/2011 5:06 AM, mdirkse wrote:
>
> Hi Michael,
> I've only got 1 client thread, and do actually have half a gig of RAM
> for it to use, but I take your point that's it's not the most efficient
> way to handle things.
> Were your timeouts on 100 messages on version 7 or version 8 POAs?
> Regards,
> Maarten
>
>

Both, on a variety of customers. RAM was not relevant. SOAP threads were
also one. Probably relevant was overall business of POA.

Trust me - it happens. Individual POAs do NEED tweaks on these settings,
unless you set down to about 25, which seems almost universally good
enough. As I said so - there was a noticeable performance diff between
25 and 100 most of the time.
0 Likes
Absent Member.
Absent Member.

I'm not sure why it is exponential. That is just what is observed.
The bottle neck seems to be converting the internal structures
into DOM before converting to the final XML string.

That said, we have made some changes for the Ascot release where
we bypass converting the data to DOM. We go straight from internal
data to XML. That seems to help performance. We still recommend
keeping the memory buffers small (less than 64 KB). Memory
fragmentation can cause difficult problems to identify.

Preston

>>> On Thursday, June 16, 2011 at 1:51 PM, Michael

Bell<mikebell@nospam.com>
wrote:
> Out of curiosity, what makes it exponential instead of linear? I assume
> that means it cannot just be streamed per se but has to be buffered in
> memory in some form.
>
> OTOH asking for a million items is not bright, because even if Preston
> can do that with 0 performance degradation, and in 0 time, YOU will have
> probably buffer that response in most client soap stacks. Do you really
> have 1 million x .5 (guess .5 kb per item) 500 MB RAM to spare PER
> client thread?
>
> I can tell you from experience that even going to 100 WILL cause some
> POAs to give cursor timeout errors. Sometimes as low as 25 is needed for
> some POAs not to be unhappy. (though that makes a fair performance diff)
>
>
> On 6/16/2011 8:59 AM, Preston Stephenson wrote:
>> The number is computed, but it is under 4000.
>> You want to be careful and not ask for too many
>> messages. The performance degrades exponentially
>> as the size of the SOAP response buffer grows.
>> You will want to keep the size of the SOAP response
>> buffer under 64KB. Depending on the size of each
>> message, you usually want to ask for 200 or less
>> items at a time.
>>
>> Preston
>>
>>>>> On Thursday, June 16, 2011 at 4:36 AM,

>> mdirkse<mdirkse@no‑mx.forums.novell.com>
>> wrote:
>>
>>> Hi,
>>> The ReadCursorRequest seems to have a hard upper limit for the value of
>>> the<count> parameter. I you try to retrieve, say, a million messages
>>> from the cursor, it returns a BAD PARAMETER error.
>>> What is the upper limit of the<count> parameter, and can it be
>>> modified somehow?
>>> Regards,
>>> Maarten

0 Likes
Absent Member.
Absent Member.

On 6/20/2011 4:02 AM, Preston Stephenson wrote:
> I'm not sure why it is exponential. That is just what is observed.
> The bottle neck seems to be converting the internal structures
> into DOM before converting to the final XML string.
>
> That said, we have made some changes for the Ascot release where
> we bypass converting the data to DOM. We go straight from internal
> data to XML. That seems to help performance. We still recommend
> keeping the memory buffers small (less than 64 KB). Memory
> fragmentation can cause difficult problems to identify.
>
> Preston


I would think the negative of going direct to xml is sending corrupted data?

0 Likes
Absent Member.
Absent Member.

Sorry, I'm a little unclear of what you mean by sending "corrupted data".

In the past, the data was converted to a string that was saved in
a DOM element with a tag. In converting to XML, the starting tag is
written to the stream, the string, then the ending tag.

That is the same process we are doing now. We are just bypassing the
separate writing the data to a string in DOM, then reading the data
in the DOM to write it to an XML string.

That said, we did work through some issues in making sure the data was
written out in a well formatted XML / SOAP string.

The cost clearly justifies the change. In stress tests that we ran, 75%
of all of the processing was in the DOM and XML processing. After the
change, the percentage dropped to 23%.

Preston


>>> On Monday, June 20, 2011 at 9:25 AM, Michael Bell<mikebell@nospam.com>

wrote:
> On 6/20/2011 4:02 AM, Preston Stephenson wrote:
>> I'm not sure why it is exponential. That is just what is observed.
>> The bottle neck seems to be converting the internal structures
>> into DOM before converting to the final XML string.
>>
>> That said, we have made some changes for the Ascot release where
>> we bypass converting the data to DOM. We go straight from internal
>> data to XML. That seems to help performance. We still recommend
>> keeping the memory buffers small (less than 64 KB). Memory
>> fragmentation can cause difficult problems to identify.
>>
>> Preston

>
> I would think the negative of going direct to xml is sending corrupted

data?
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.