Missing and reappearing records

[kschutten]

Dear fellow developers,

I wonder if anyone else has also experienced (dis)appearing records in Bullhorn?
I'm a developer for a Dutch Recruitment agency. I have a logged ticket already since march of this year with Bullhorn.

The problem we are facing is some API calls are missing records! Some records do not (always) appear in the API calls, end even worse not always in the GUI! (as Bullhorn also uses API calls in the background)! We noticed it at first with the entity Note but now also with entity Candidate.

For an example I call for Notes of a certain period of time (which has past today (until noon utc-4 of course)):

{Base_URL}/search/Note?query=dateLastModified:[20190625 TO 201906251200]&fields=*&sort=dateAdded,NoteId&start=0&count=200&BhRestToken={rest_token}

I get a total of 1774 notes (and of course the results for the first 200)
Seconds later I make the same call; Total: 1778!
I call again total 1774....
And so it's switching back and forward endlessly.
When I dig deeper I can even find the specific note(s) and when I look for the candidate to whom the note belongs I can see it also happening in the GUI too, which scares me the most. So when I load the candidates notes: I don't see the note! Refresh, It appears, refresh... It's gone!.... And ..... There it is again!

Sometimes I have 4x the same result, but the 5th is off again. Sometimes it's every call and other times everything is just in sync.

Bullhorn told me they have functionality running to replicate tables to speed up, and that's why the 2 calls can have a different outcome. They can and do re-sync the tables from time to time (every day or so), that's why we experience it only with recently changed/created records.

Can you please let me know if you have experienced similar symptoms? Or even try to call recent notes (a couple of times) on an active system?
Your reply will be much appreciated!

[ivandevpartner]

We had and still have that same behaviour.... Never found the source of the issue.
As far as I know this issue was present at least as far back as autumn 2017.
Unfortunately no real fix or workaround found.

[kschutten]

Hi ivandevpartner

Thanks for replying and letting it know! And although I feel sorry for others with the same issue, it's somehow good to know we are not alone on this one! Our recruiters are complaining since long to be misinformed from time to time, but only in March we found out the exact problem.

Until now we only experience it only on API entities where we have to use /search construction and the Lucene syntax, but that might be a coincidence.

The Bullhorn developers are in the dark about this issue, and with priority "not so much", I guess.
Therefore it would be really helpful to find more cases of this phenomenon.... So, if you are reading this, please try the API call(s) and reply!

[kkbarimah]

Hi kschutten,

This is Kingsley from Bullhorn Support. Do you have the ticket number of the Support ticket you submitted about this issue?

Thanks,
Kingsley

[kschutten]

Hi Kingsley,

Thanks for joining this topic!
Ticket 03614473

[kkbarimah]

Hello kschutten,

I've examined this ticket and it looks like there's been a lot of troubleshooting and research on this by our engineering team. I am also seeing that there are multiple reports of this issue and you are not alone on this. There should be an update on this ticket soon.

[kschutten]

Hi Kingsley,

Thank you for the reassurance! I sure do hope so, as I read from ivandevpartner it's almost 2 years since it was first noticed.
When creating the ticket in March it appeared that the issue was completely new to Bullhorn, but apparently it was not. I appreciate the honousty in confirming this issue was reported multiple times already.

I'll keep an eye on the progress and report back here (when there's something to report..... )!

[ivandevpartner]

Hello kschutten,

As I have posted before - this is an issue when pulling data from BH API.
It looks like it's affecting Notes specifically because the API endpoint used is "search".
What I think happens in my case when I try to pull all Notes in batches, is that when I order Notes by Id, it orders them as strings, so an Id of 2 is greater than 19999999.

There's some sort of "indexing" going on under the bonnet (most probably) and it (the index) is being restructured either continuously, or at certain times of the day. The query consults the "index" and returns the data sorted by Ids as strings (BTW sorting by dateAdded is almost the same as BH sorts those UNIX datetimes as strings as well) and at some point new data is added and removed from the Notes pile. Since a new Note with an ID 1999998 is going to be sorted BEFORE Note ID 2, it messes up all the batching and so the "start=...&count=200" becomes meaningless.

I am not sure how this would help you, but it might give some helpful insight into a possible workaround. I'd love to hear about one if you come up with it and are kind enough to share.

Ivan.

[kschutten]

Hi Ivan,

Thanks for taking the time!
We went great lengths with Bullhorn and although some people hate me by now, a solution is near!
Bullhorn is releasing it this month and rolling out to customers in batches.

This indeed happens only on "search" endpoints.
As you might know there are two different indexers for purposes of load balancing. One of them is always correct, the other not. That's why in a search you can find 100 notes, candidates or what ever, and when you run the search again; 99. But since the system decides which indexer to use for each request, you will see different results and don't know which indexer gave you. A serie of requests may look like: 100, 100, 100, 99, 99, 100, 99, 99, 100, 99, 99, 100, 100. The root cause is a defect in a low-level data transport which caused on of the indexers to miss events.

In the meantime; you can ask support for a "bootstrap" of the entity where this happens! (a kind of rebuild for the second indexer)

Cheers!

P.s. Best to sort in every request on dateAdded and then id (order=dateAdded,id)

[ivandevpartner]

Hello kschutten,

Is this "bootstrap" that rebuilds the index a request that one can make into the API or a separate support request that at some point might be answered by BH if they are kind enough? How would that fix the problem if after the index is recalculated new data arrives and messes up one of the indexes yet again? By the way, how long does that index stay correct?

I will definitely try to sort by two fields like you recommended. Thanks.

Ivan.

[kschutten]

Hi Ivan,

Unfortunately the latter; you have to ask BH support!
It's only a very temporary fix. In our case we see that these indexers are off once or twice a week! It happens far more often (4 notes a day on average) but there is also a nightly catch-up job that corrects a lot of these missed events (for our data warehouse load that is doable), and it does not correct all!

It hope this to be over soon..... It's been giving me a lot of headaches in the last year!
Good luck to you!

Cheers

[atibbles]

Hi All,

Just adding my experience with this issue too.

We are also having a similar issues with the Placement table, which unfortunately completely invalidates a key report that I am building throuogh the API. I look pretty stupid if I cannot even produce something with consistent numbers from one minute to the next.

Ticket 03944161 has been logged since 24th Feb 2020, with a couple of re-indexes in the meantime but the issue appears to be still unresolved with inconsistent results. I think our problem relates to a custom date field but not entirely sure that this is the problem.

Very disappointing (but not surprising) to hear that this has been an outstanding issue for so long.

[kschutten]

Hi atibbles,

The release has been deployed by Bullhorn, and is in production. It needs to be activated per client/environment and is now active for NPE environments. We really hope to see the final solution activated in our production environment in the next month. As we do not have the same amount of traffic on NPE, we can not test it properly yet.

I pushed it as hard as I possibly could. The good that came out of it was the acknowledgement for the bug en Bullhorn accepting it as a problem. The root cause lies somewhere deep in the architecture on a low-level data transport. I actually still don't know what triggers it.

Fingers crossed!

[MysteryA]

I've also noticed that the following API call always returns zero records. Perhaps this is part of the same issue:

search/Note?query=personReferenceType:Lead

[kschutten]

Dear fellow developers,

We seem to be past this issue.... 15 months after reporting and pushing it, this feels like a new era!
After the release was rolled out to production environment and the (indexer configuration) option was "set" for our Bullhorn ATS, we do see only consistent searches again!
We have to test bigger updates still, but it looks promising!