Google Apps Driver "Serving Limit Exceeded"

[joe_fortier]

Heads Up and potential discussion

Our Google Apps Driver shutdown with a GA error "Serving Limit
Exceeded"

{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Serving Limit Exceeded",
"reason" : "servingLimitExceeded"
} ],
"message" : "Serving Limit Exceeded"
}</message>

I was able to restart the driver.

It appears to be related to new limits imposed on the admin user account
per
http://tinyurl.com/oed6sj9
But I've not verified.


--
joe_fortier
------------------------------------------------------------------------
joe_fortier's Profile: https://forums.netiq.com/member.php?userid=4309
View this thread: https://forums.netiq.com/showthread.php?t=54102

[dgersic]

On Thu, 20 Aug 2015 18:54:02 +0000, joe fortier wrote:

> Heads Up and potential discussion
>
> Our Google Apps Driver shutdown with a GA error "Serving Limit Exceeded"
>
> {
> "code" : 403,
> "errors" : [ {
> "domain" : "usageLimits",
> "message" : "Serving Limit Exceeded", "reason" : "servingLimitExceeded"
> } ],
> "message" : "Serving Limit Exceeded"
> }</message>
>
> I was able to restart the driver.
>
> It appears to be related to new limits imposed on the admin user account
> per
> http://tinyurl.com/oed6sj9
> But I've not verified.

I just had this hit me here. Restarted the driver, and it's off and
running again. I'm thinking maybe Google has a bug, and is somewhat
randomly imposing this limitation. But good luck getting anything from
them about it.



--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.

[celsolima]

That has happened to me twice now. The first time on Friday (08/21) and
again today (08/24). We are also under a Google Apps education license.


--
celsolima
------------------------------------------------------------------------
celsolima's Profile: https://forums.netiq.com/member.php?userid=260
View this thread: https://forums.netiq.com/showthread.php?t=54102

[burlandp]

We have seen this too, a couple of times over the past week.

As a messy workaround (in case anyone is interested) I have put a cron
job on the server to get dxcmd to start the driver every 10 mins.
If the driver is running, the command is ignored, if it has crashed, it
starts up.

eg:-
dxcmd -host 192.168.0.1 -user admin.org -password secret -start
'Gapps.Driver Set.org'

[brucetimberlake]

We are also seeing this with our Apps and Groups drivers. We have two
projects/IDs created for each of our Google drivers categories, and have
configured half of each set of drivers to use one or the other of the
IDs as an attempt at a workaround until the real cause can be
identified.

We found this: "If you don't send the userIp or quotaUser parameters,
then all calls are listed as coming directly from the server-side
machine itself, in which case calls can't be capped by user"
(https://developers.google.com/console/help/new/#cappingusage). That
sounds like Google may not recognize the quotaUser if it's not
explicitly sent with each API call, and is reverting to the default
15/second/user limit.

But clearly something seems to have changed on the Google side very
recently as we were running fine until sometime yesterday (Sept 3).


--
brucetimberlake
------------------------------------------------------------------------
brucetimberlake's Profile: https://forums.netiq.com/member.php?userid=1036
View this thread: https://forums.netiq.com/showthread.php?t=54102

[dgersic]

On Fri, 04 Sep 2015 18:14:02 +0000, brucetimberlake wrote:

> We are also seeing this with our Apps and Groups drivers. We have two
> projects/IDs created for each of our Google drivers categories, and have
> configured half of each set of drivers to use one or the other of the
> IDs as an attempt at a workaround until the real cause can be
> identified.
>
> We found this: "If you don't send the userIp or quotaUser parameters,
> then all calls are listed as coming directly from the server-side
> machine itself, in which case calls can't be capped by user"
> (https://developers.google.com/console/help/new/#cappingusage). That
> sounds like Google may not recognize the quotaUser if it's not
> explicitly sent with each API call, and is reverting to the default
> 15/second/user limit.
>
> But clearly something seems to have changed on the Google side very
> recently as we were running fine until sometime yesterday (Sept 3).

Have you (or anybody else) opened an SR with NetIQ yet?


--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.

[cpedersen]

On 9/4/15 9:00 PM, David Gersic wrote:
> Have you (or anybody else) opened an SR with NetIQ yet?

We know about this, that is as much as I can say right now.



Casper

[tomgreene]

We have two customers having this issue as well. We've used the cron
workaround for the moment.


--
tomgreene
------------------------------------------------------------------------
tomgreene's Profile: https://forums.netiq.com/member.php?userid=1528
View this thread: https://forums.netiq.com/showthread.php?t=54102

[dgersic]

On Sat, 05 Sep 2015 20:41:19 +0000, Casper Pedersen wrote:

> On 9/4/15 9:00 PM, David Gersic wrote:
>> Have you (or anybody else) opened an SR with NetIQ yet?
>
> We know about this, that is as much as I can say right now.

Ok, that's good to know.


--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.

[Marcus Tornberg]

Hi.

I am also seeing this issue with Google Apps for Education. In my case I
implemented the following rule in input transformation where I change
the level of the error from "fatal" (stop the driver) to "retry". This
might be an option to scheduling restart of the driver.

When I look at the Google Console the IDM project is nowhere near the
limits specified.

I'm looking forward to a fix.


Code:
--------------------
<rule>
<description>Workaround for Serving Limit Exceeded</description>
<comment xml:space="preserve">Change status from fatal to retry on Serving Limit Exceeded</comment>
<conditions>
<and>
<if-operation mode="nocase" op="equal">status</if-operation>
<if-xml-attr mode="nocase" name="level" op="equal">fatal</if-xml-attr>
<if-xpath op="true">description/text() = 'GoogleJsonResponseException: Serving Limit Exceeded'</if-xpath>
</and>
</conditions>
<actions>
<do-set-xml-attr expression="." name="level">
<arg-string>
<token-text xml:space="preserve">retry</token-text>
</arg-string>
</do-set-xml-attr>
</actions>
</rule>
--------------------


Best Regards
Marcus


--
marcus_jonsson
------------------------------------------------------------------------
marcus_jonsson's Profile: https://forums.netiq.com/member.php?userid=1157
View this thread: https://forums.netiq.com/showthread.php?t=54102

[dgersic]

On Wed, 09 Sep 2015 09:04:01 +0000, marcus jonsson wrote:

> Hi.
>
> I am also seeing this issue with Google Apps for Education. In my case I
> implemented the following rule in input transformation where I change
> the level of the error from "fatal" (stop the driver) to "retry". This
> might be an option to scheduling restart of the driver.
>
> When I look at the Google Console the IDM project is nowhere near the
> limits specified.
>
> I'm looking forward to a fix.
>
>
> Code:
> --------------------
> <rule>
> <description>Workaround for Serving Limit Exceeded</description>
> <comment xml:space="preserve">Change status from fatal to retry on
> Serving Limit Exceeded</comment> <conditions>
> <and>
> <if-operation mode="nocase" op="equal">status</if-
operation>
> <if-xml-attr mode="nocase" name="level"
> op="equal">fatal</if-xml-attr> <if-xpath
> op="true">description/text() =
'GoogleJsonResponseException:
> Serving Limit Exceeded'</if-xpath>
> </and>
> </conditions>
> <actions>
> <do-set-xml-attr expression="." name="level">
> <arg-string>
> <token-text xml:space="preserve">retry</
token-text>
> </arg-string>
> </do-set-xml-attr>
> </actions>
> </rule>
> --------------------
>
>
> Best Regards
> Marcus

Thanks Marcus. So with this rule in place, does the retry work?

I'm personally suspecting that this is a bug on Google's side of things,
not a driver shim issue, but this would be a nice work-around for it.



--
--------------------------------------------------------------------------
David Gersic dgersic_@_niu.edu
Knowledge Partner http://forums.microfocus.com

Please post questions in the forums. No support provided via email.
If you find this post helpful, please click on the star below.

[Marcus Tornberg]

Hi David.

Yes, the retry works. In my own implementation of the rule I also send
an e-mail notification to myself so I know how often the issue occurs.

The driver has had a big backlog and been working 24/7 for a month in
updating all user and groups, I have had the opportunity to see when
this issue occurs, and it seems to be perodic. The problem seems to be
from 17-19 hours (CET) where the driver gets the error, retries,
succesfully handles some requests and then the error occurs again. This
goes on for a few hours (mostly around 17-19) and then it works without
the error for days.

Because our driver has been working 24/7 as fast as it could I also
suspect the error to be on the Google side. For the fix, I would see
that NetIQ's driver returns retry instead of fatal when it get this
error from Google.

Best Regards
Marcus


--
marcus_jonsson
------------------------------------------------------------------------
marcus_jonsson's Profile: https://forums.netiq.com/member.php?userid=1157
View this thread: https://forums.netiq.com/showthread.php?t=54102