Reverse proxy instance health check (route_offline_thread)
Reverse proxy offline thread
Web Server has a thread named route_offline_thread which does the
health check of backend instances. Once this thread is started, it
finds out which of the backend instances are alive. To find out whether
a backend server is alive, it sends a OPTIONS HTTP request to the
backend instance. If the server responds back then the server is alive.
After the initial phase, the thread wakes up every 60 seconds to do
this health check. The thread doesn't send any "OPTIONS" request to
those instances which are already alive. It only tries to send
"OPTIONS" request to those instances which are offline. This way thread
doesn't impose any performance impact to the system and to backend
instances. If a backend instance is shutdown or crashed for whatever
reason, route_offline_thread will never know about it. Only way Web
Server rpp knows about the death/hang of the backend is when an
instance is chosen for serving a request. Rpp sends the request to
backend instance and if connection to the instance is either broken
then it will result in read/send failure. Web Server sets a response
timeout which can be configure using http-client-config
ObjectType function in obj.conf.
The above statement tells the rpp to close the connection to backend
instance if a connection (read) hangs for more than 400 seconds.
Default value is 300 seconds. If that happends then rpp marks the
instance as offline. After this time route_offline_thread will poll for
this offline instance every 60 seconds (which is not
configurable). If backend instance recovers from hang or
restarted and it responds back to "OPTIONS" HTTP request then
route_offline_thread mark the instance back to online. Connect timeout
for backend instance is 5 seconds (hardcoded). So if backend instance
is hanging (or too busy) and unable to handle connection for next 5
seconds then rpp will treat the instance as offline.