Uncategorized

NodeJS + ExpressJS + ReactJS on Google GCP App Engine Restarts by Itself without any exception

Well I have posted same question in Stackoverflow as well. you can check and answer for this here

We have ReactJS + NodeJS backend app running in Google GCP App Engine, but what we are experiencing is server down time for few min randomly. We have info Logs and error logs in code almost every api and every major functions and all catch blocks. I have also put global exception catch block

<code>process.on('uncaughtException', (error) => {
Logger.error('--- Exception -----', error);
});</code>

I also have error handler middleware in expressjs.

but whenever server restarts by itself i don’t see any exception in logs, it doesn’t even appear in process.on(‘uncaughtException’ block also.

One observation is GCP App engine is auto scaling when there is more traffic, but app is going down whenever there more traffic despite auto scaling is happening.

I want to know how to handle this, how to debug this to identify where exactly the issue is, is the issue is at code level if so why none of the exceptions are caught? or the issue is with GCP?

our production server also goes down many times randomly.

I tried using longjohn and setting NODE_DEBUG environment variable to net to get some hint as suggested in other SO Post

Update: We Ran Jmeter Performance test to check whether load on server is the main reason, as Rafael Lemos rightly pointed in comment, in our case “The instance exceeds the maximum memory for its configured instance_class” may be the reason behind server going down, with lot of trail and error, when we made minimum no of instances = 2 we are not observing server going down.

However jmeter tests are all running fine but when we do jmeter test as well as manual testing at a time , we are observing Socket Hangup Exception, this crash happens immediately as soon as we start using front end app, where as jmeter tests access our backend app. Both backend and front end are running in same app engine.

Kindly find our observations below

Now question is Why Socket Hangup is coming, when I explored this, I found a solution in some other SO post, to handle socket hangup exception I have put below code in app.js but its not working

<code>app.use((req, res, next) => {
  if(res.socket.destroyed) {
    console.log('----- socketIsDestroyed-- ', res.socket.destroyed);
    res.end();
  } else {
    console.log('----- socketNotDestroyed-- ', res.socket.destroyed);
    next();
  }
})</code>

And in Google App Engine Sys logs I see following error log multiple times

<code>GET 503  /readiness_check failReason:"app lameducked"</code>

Sequelize Database Entity Relation for Postgres SQL-Many To Many Relation

In case of Many To Many relation we can define relation using Sequelize belongsToMany method. But before moving on to Many-To-Many relation in sequelize, I suggest readers to go through my previous post on One-To-Many relation with sequelize.

Defining one-to-many relation in Sequelize

Now let us consider the below relation for understanding on how to define Many-to-many relation using sequelize. If you are developing a platform for school management system and considering the requirement one user is associated with many schools, this is true in case of teachers, where they will be teaching in multiple schools / colleges as guest lectures.

The database schema for Users and Schools Entities looks as follows.

Defining Many-To-Many Relation using Sequelize

Users Model

<code>
const UserModel = db.define(
  'users',
  {
    userId: {
      field: 'userId',
      type: sqlOperator.UUID,
      primaryKey: true,
      defaultValue: sqlOperator.UUIDV4
    },
    firstName: {
      field: 'firstName',
      type: sqlOperator.STRING(100),
      require: true
    },
    lastName: {
      field: 'lastName',
      type: sqlOperator.STRING(100),
    },
    email: {
      field: 'email',
      type: sqlOperator.STRING(255),
      defaultValue: null,
      allowNull: true,
      // unique: true,
      validate: {
        isEmail: true
      }
    },
    userName: {
      field: 'userName',
      type: sqlOperator.STRING(255),
      defaultValue: null,
      allowNull: true,
      // unique: true
    }
  },
  {
    tableName: 'users',
    timestamps: true,
    freezeTableName: true,
  }
);

UserModel.associate = async () => {
  //defining relation in user schools model
}

module.exports = UserModel;
</code>

Observe here that the relation between Users and Schools is defined through another intermediate table called User Schools hence the ralation is defined using belongsToMany using through keyword as given below.

UserSchools Model

<code>const SchoolModel = require('./SchoolModel');
const UserModel = require('./UserModel');

const UserSchoolModel = db.define(
  'user_schools',
  {
    userSchoolId: {
      field: 'userSchoolId',
      type: sqlOperator.UUID,
      primaryKey: true,
      defaultValue: sqlOperator.UUIDV4
    }
  },
  {
    tableName: 'user_schools',
    timestamps: true
  }
);

UserSchoolModel.associate = async () => {
  const UserModel = require("./UserModel");
  UserModel.belongsToMany(SchoolModel, {
    through: UserSchoolModel,
    foreignKey: "userUserId",
    as: 'userSchools'
  });
  SchoolModel.belongsToMany(UserModel, {
    through: UserSchoolModel,
    foreignKey: "schoolSchoolId",
    as: 'schoolUsers'
  });
};
module.exports = UserSchoolModel;
</code>

If you try to achieve this only using belongsTo and hasMany then you will face many sequelize errors while querying. You must use belongsToMany along with through keyword.

Be social and help to stop corona – Corona Relief

COVID-19 has affected almost every people, every sector and every nation. There are millions of people suffering with various problems. It is we who are strong financially, physically and mentally have to help people who are in need and be a corona warriors.

How can I help during COVID-19?

Join SociallyGood a trusted first ever marketplace for social service sector to donate thousands of non-profits who are serving millions of people who got affected by corona virus pandemic. Volunteer and donate through SociallyGood

Volunteer and Donate to trusted non-profits, find campaigns running for covid-19 here

Donate to Help during this COVID-19 Pandemic

We all know how badly Corona Virus has affected millions of people. Daily wage workers and all other businesses are hit badly. Its time to help those who are in need, there are thousands of Non Profits working for COVID-19 situation.

Sewa USA is running Fund raiser program as Sur Surgam, donate for Sewa USA

Sewa Bharati is asking for donations which they use to support daily wake workers, donate for this

You can help health care heros, they are Corona Warriors who are helping people by keeping their life in risk, Donate for Doctors, nurses during this COVID-19 pandemic

Volunteering Opportunities for COVID-19

COVID-19 pandemic has made our life miserable, almost every country is facing hardship due to corona virus. Corona Warriors like doctors, nurses and many millions of social workers are helping us to save us from Corona.

This is the time where everyone has to serve the society to help people and to stop pandemic. There are several Non-Profits working towards this goal. You can find several volunteering opportunities in a Social Service Marketplace called SociallyGood, a trusted platform for Social Service Sector. Here are some of the volunteering opportunities for COVID-19

Some of the trending campaigns which may help you to serve the society are here

Volunteering Campaigns for COVID-19

People are getting frustrated, depressed and feeling lonely during the lockdown due to COVID-19, there are many ways to help people for mental health support during COVID-19

Click here to work for this cause

Food distribution to Migrant workers during COVID-19 Pandemic

Click here to donate for COVID-19

Platform Architecture In Computer Science – Micro Services Approach

We have seen how to design platform architecture with multiple apps, we understood pros and cons of that approach in my previous article, Read it from below link

Read – Platform Architecture with Multiple Apps

For applications like Amazon, Netflix, Linked in the backend systems deals with billions of requests per second. It is not possible to handle such a huge traffic by having one consumer app or one seller app. The solution to such a huge systems is micro service architecture.

Probably Amazon has mastered in Micro Service Architecture which runs over more than 500 micro services, similarly Netflix has over 500 micro services and their API Gateway handles over 2 billion requests daily – Source

So dividing Amazon kind of apps into multiple micro services may have following services

Amazon Kind Of Platforms Micro Services

  1. User & Identity Management Micro Service
  2. Product Management Micro Service
  3. Order Management Micro Service
  4. Payment Management Micro Service
  5. Delivery Management Micro Service
  6. Data Mining Micro Service
  7. AI Micro Service

The Micro Service Architecture Looks as follows for Amazon Kind of Platforms

Advantage of Micro Service Architecture

Highly Scalable: Systems of this kind are highly scalable, capable to handle billions of requests every day

Heterogenous Technology: Another major advantage is we can develop each micro service using different technology whichever is suitable for it. Example Order Management you can develop using nodejs and MongoDB, however for data mining management, you can use python and MongoDB. For payment management you can use Java with Postgres SQL

Easy to Build and Maintain: Your development team might have spread across the world, it is easy to maintain and build any service any time

Ever Evolving: Every micro service can evolve at any great extent without affecting other systems

After knowing all of these the question arises, is Micro Service architecture is the best? Should Every App has to adopt to it ?

The answer is NO : Architecture of every platform greatly varies depends upon the needs of the platform. Micro Services approach is suitable for highly scalable systems, however for smaller platforms having multiple apps under same platform is suitable.

To understand even better Read Case Studies of Different Platform Architectures

Platform Architecture in Computer Science: where do API Gateway & GraphQL fit

In our previous article we have seen why and how to design the platform as multiple apps running in independent server with their own database.

Read in Previous Article

API Gateway and Need of API Gateway in Platform Architecture

But for any platform the most important Managing the API Calls, what I mean here is there should be something in between clients and various services which will receive incoming requests, invokes various apis and return the appropriate results.

Also any backend platform should keep track of Api calls, rate limiting – it is defining the rate at which consumer can access apis, limit the number of api calls per second per user or per app, throttling – is controlling usage of apis for consumers for a given period of time, securing apis through api keys and user authentication.

in platform architecture API Gateway is like front facing for incoming client requests. Apigee is one very popular api gateway, however it is suitable for enterprise and big applications, for small apps and startups where one can’t afford Apigee, they can go for other API Gateways like Google Cloud End Points but it has very limited features OR Kong API gateway.

Boosting your Client Apps Performance and Stability by Adopting GraphQL

GraphQL is a query language for APIs, it is like giving control to clients/apps to fetch what exactly they want rather than decided by server.

Ask for What you need and Get exactly that 🙂

Send a GraphQL query to your API and get exactly what you want

<code>//GraphQL Query to API
{
  product: {
    name,
    images,
    price
  }
}

//This will be input to your API and you will get only that in //response quickly

//Response
{
  product: {
    name: 'Micro Max G6',
    images: &#91;],
    price: '6599 Rs'
  }
}</code>

Get Multiple Information in Single Request

GraphQL will make your apps lightning speed which even out perform greatly in slow network speed as well, because GraphQL is powerful and returns multiple information in single request, which apps typically get it from different api calls.

There are multiple bunch of benefits of GraphQL and its worth to consider using GraphQL while architecting your Platform. Well where GraphQL fits in your platform architecture ? See the updated architecture diagram

Continue Reading About Micro-Service Platform Architecture

Life Beyond Structure, Analysis, program and computers

I found this wonderful video and found the meaning of life, you may be a programmer, a civil engineer, a car maker, a researcher, a teacher or a student whoever you may be, but one must listen to this talk by Prof. Devdas Menon, Department of Civil Engineering, IIT Madras

Some of the very interesting point taken out from this video are as follows

George Calvin’s famouse quote in 80’s
The paradox of our time and history is that


We tall the buildings but short the temper,
wide our free ways but narrowed our view point
we spend more but we have less
we buy more but we enjoy less
we have bigger houses but smaller family
we have more knowledge but less judgement
we have more experts but yet more problems
we have more medicines but less wellness
we multiplied our positions but reduced our values
we talk too much , drive too fast , stay up too late, getup too tired, we have reached moon and back to earth but trouble crossing a road to meet a neighbour,
we have conquered outer space but not inner space
we have done larger things but not better
we have learnt to rush but not to wait
we have fast food but slow digestion
a big men but small character
too many young couple but more divorces
more entertainment but less happiness

MASLOW’s Hierarchy Of Needs

Matthieu Ricard’s Quote

A French Scientist, who turned as tibetan monk

We all tried consciously or unconsciously, competently or clumsily , Passionately or calmly, adventurously or routinely to be Happier, Yet we more often confuse the genuine happiness with merely seeking enjoyable sensation. Happiness is state of inner fulfilment but not the gratification of inexhaustible desire for outer things. This is traditional wisdom.

Kabir Das Said it beautifully in 15th century in one of his famous doha as

कस्तूरी कुंडल बसे,मृग ढूँढत बन माही। ज्योज्यो घट– घट राम है,दुनिया देखें नाही ।।

Listen to this nice video by Prof. Devdas Menon, Department of Civil Engineering, IIT Madras