OSRM time not calculating routes for all observations

[Bryan Sayer]

We have been using osrmtime to calculate car routes for about 316 million observations. The first attempt at all observations resulted in exactly 75% of the observations being solved. Interestingly, it is the first 25% of observations that did not solve. I wonder if osrmtime takes the observations in reverse order.

When I check the resource usage during the run time there does appear to be an increasing usage of memory and pagefile during the run, causing me to wonder if the osrmtime is accumulating something in memory, that eventually fills it up and causing it to quit working fully.

I've tried doing the remaining 25% of observations four times, and it stops after about one or two days (depending on how many threads I allow it to use). So now I am going to split these 60 million records into six pieces of about 10 million each.

So the questions are:
1. Does osrmtime process the records in reverse order?
2. What type of run times are other people getting?
3. How can I check if I am using the current version of the osrm executable (not the ado files)?
4. Any advice on using osrmtime?
5. Is there a way to see what the problems are?

Thanks!

[Bryan Sayer]

I have discovered that the problem appears to be related to one thread on a CPU. I have four cores, so I have eight threads. I just tried about 5.2 million observations using eight threads (four is the default) and I have 12.5% of the records not solving. One million observations worked okay, but 5.2 million does not. Right now I am trying two threads on about 11 million observations.

So the problem appears to occur with some number of observations between 1 million and 5.2 million, and it appears to be one specific thread.Does anyone have an idea of how I can further track this down? Does it imply that one of my cores is defective (that seems strange since only one of the two threads is impacted)?