Recent 429 Error Handling

[pmularski]

Greetings Community!

Users have been reporting an increase in 429 errors while running existing/new scripts in the APIs. This post is to give some insight into the hows and whys of 429 errors, what we at Bullhorn are doing to help alleviate the impact, and what you can do on your end with your code.

There is built-in error handling specifically for 429s in our publicly-available open-source SDK as an indicator for baseline recommendations. The default HTTP-429 handling in the Bullhorn REST SDK is:
Wait 1 second and try again (Thread.sleep(1000);)
Attempts 3 times (API_RETRY variable)
Handled in this file, roughly lines 2071-2091 (few other subroutines are elsewhere in the file)
This can certainly be built out with more nuance in that, for example; Execute more tries with a shorter wait between, and then if still receiving fails, essentially suspend the transaction & logs on a database table for reinitiation in the next .5-1 hour span.

The root package for the SDK may be found here: https://github.com/bullhorn/sdk-rest

Here are a few frequently asked questions relating to infrastructure & cluster concerns for QoS limiting:

• Auto retry every second until successful adds additional load on our end , Retry is fine if this occurs rarely but in this case it’s happening almost every day, we had one today as well

In situations where this response is occurring, API responses would be typically delayed or time out. As a result, the time it takes to get the request in would be similar. In some cases the response time is better because the API recovers more quickly from issues.

• We were not aware of the new response code , is it possible to subscribe for any future changes in API

This response code has been part of the Bullhorn API for over a year, and was communicated broadly when API rate limiting was rolled out. When making changes that impact API behavior, Bullhorn does typically have a communication plan for API users.

• This week we saw capacity exceeded errors almost every day , our volumes are very low and are concerned when we move other LOBs

These issues can sometimes occur when simple bursts of traffic arrive in a narrow time window, and is not always a symptom that the overall capacity of the services are out of balance. In any scenario where this would persist beyond a couple minutes operational alerts will fire and our systems engineers will investigate and drive to RCA. Note that in many cases the RCA involves working with 3rd parties to improve the behavior of their integrations.

• What action is taken by Bullhorn internally for Server API capacity exceeded responses to avoid/minimize 429 response codes?

As mentioned above, any scenario lasting more than a few minutes will be thoroughly investigated by our operations team. We also new have a development team focused on overall performance and health improvements, which will be continually improving the overall performance and capacity of Bullhorn APIs

• Does Bullhorn auto scale and how often capacity planning is done on API servers?

Bullhorn has a lot of internal flexibility in how we scale and provision servers, and we're also continually investing in our ability to respond quickly. At the moment that scaling is done manually by our team. We constantly monitor and address load on every swimlane. As mentioned above however, sporadic instances of a 429 response do not necessarily indicate the need to scale if they are in response to a burst over a narrow timeframe.

-Bullhorn Support