NodeJS + ExpressJS + ReactJS on Google GCP App Engine Restarts by Itself without any exception

Well I have posted same question in Stackoverflow as well. you can check and answer for this here

We have ReactJS + NodeJS backend app running in Google GCP App Engine, but what we are experiencing is server down time for few min randomly. We have info Logs and error logs in code almost every api and every major functions and all catch blocks. I have also put global exception catch block

process.on('uncaughtException', (error) => {
Logger.error('--- Exception -----', error);
});

I also have error handler middleware in expressjs.

but whenever server restarts by itself i don’t see any exception in logs, it doesn’t even appear in process.on(‘uncaughtException’ block also.

One observation is GCP App engine is auto scaling when there is more traffic, but app is going down whenever there more traffic despite auto scaling is happening.

I want to know how to handle this, how to debug this to identify where exactly the issue is, is the issue is at code level if so why none of the exceptions are caught? or the issue is with GCP?

our production server also goes down many times randomly.

I tried using longjohn and setting NODE_DEBUG environment variable to net to get some hint as suggested in other SO Post

Update: We Ran Jmeter Performance test to check whether load on server is the main reason, as Rafael Lemos rightly pointed in comment, in our case “The instance exceeds the maximum memory for its configured instance_class” may be the reason behind server going down, with lot of trail and error, when we made minimum no of instances = 2 we are not observing server going down.

However jmeter tests are all running fine but when we do jmeter test as well as manual testing at a time , we are observing Socket Hangup Exception, this crash happens immediately as soon as we start using front end app, where as jmeter tests access our backend app. Both backend and front end are running in same app engine.

Kindly find our observations below

Now question is Why Socket Hangup is coming, when I explored this, I found a solution in some other SO post, to handle socket hangup exception I have put below code in app.js but its not working

app.use((req, res, next) => {
  if(res.socket.destroyed) {
    console.log('----- socketIsDestroyed-- ', res.socket.destroyed);
    res.end();
  } else {
    console.log('----- socketNotDestroyed-- ', res.socket.destroyed);
    next();
  }
})

And in Google App Engine Sys logs I see following error log multiple times

GET 503  /readiness_check failReason:"app lameducked"

Leave a Reply

Your email address will not be published. Required fields are marked *


The reCAPTCHA verification period has expired. Please reload the page.