COVID-19 pandemic has made our life miserable, almost every country is facing hardship due to corona virus. Corona Warriors like doctors, nurses and many millions of social workers are helping us to save us from Corona.
This is the time where everyone has to serve the society to help people and to stop pandemic. There are several Non-Profits working towards this goal. You can find several volunteering opportunities in a Social Service Marketplace called SociallyGood, a trusted platform for Social Service Sector. Here are some of the volunteering opportunities for COVID-19
People are getting frustrated, depressed and feeling lonely during the lockdown due to COVID-19, there are many ways to help people for mental health support during COVID-19
Despite all of these inputs, I am still saying platform architecture depends upon various needs of the platform, hence there is nothing like one standard way, it will be always evolving. Hence let us look into some of the references for Highly Scalable Platform Architectures to understand even better
Netflix architecture looks very complex and it is micro-service architecture
Some highlights of Netflix Architecture, Hundred’s of microservices and one giant service. thousand’s of servers spread across the world, thousand’s of Content Delivery Networks, Netflix uses their own CDN called Open Connect
100s of microservices
1000s of daily production changes
10,000s of instances running in AWS cloud
10, 000, 000, 000 hours of streamed
This is how it looks 🙂
Reference: How Netflix works: the (hugely simplified) complex stuff that happens every time you hit Play
We have seen how to design platform architecture with multiple apps, we understood pros and cons of that approach in my previous article, Read it from below link
For applications like Amazon, Netflix, Linked in the backend systems deals with billions of requests per second. It is not possible to handle such a huge traffic by having one consumer app or one seller app. The solution to such a huge systems is micro service architecture.
Probably Amazon has mastered in Micro Service Architecture which runs over more than 500 micro services, similarly Netflix has over 500 micro services and their API Gateway handles over 2 billion requests daily – Source
So dividing Amazon kind of apps into multiple micro services may have following services
Amazon Kind Of Platforms Micro Services
User & Identity Management Micro Service
Product Management Micro Service
Order Management Micro Service
Payment Management Micro Service
Delivery Management Micro Service
Data Mining Micro Service
AI Micro Service
The Micro Service Architecture Looks as follows for Amazon Kind of Platforms
Advantage of Micro Service Architecture
Highly Scalable: Systems of this kind are highly scalable, capable to handle billions of requests every day
Heterogenous Technology: Another major advantage is we can develop each micro service using different technology whichever is suitable for it. Example Order Management you can develop using nodejs and MongoDB, however for data mining management, you can use python and MongoDB. For payment management you can use Java with Postgres SQL
Easy to Build and Maintain: Your development team might have spread across the world, it is easy to maintain and build any service any time
Ever Evolving: Every micro service can evolve at any great extent without affecting other systems
After knowing all of these the question arises, is Micro Service architecture is the best? Should Every App has to adopt to it ?
The answer is NO : Architecture of every platform greatly varies depends upon the needs of the platform. Micro Services approach is suitable for highly scalable systems, however for smaller platforms having multiple apps under same platform is suitable.
API Gateway and Need of API Gateway in Platform Architecture
But for any platform the most important Managing the API Calls, what I mean here is there should be something in between clients and various services which will receive incoming requests, invokes various apis and return the appropriate results.
Also any backend platform should keep track of Api calls, rate limiting – it is defining the rate at which consumer can access apis, limit the number of api calls per second per user or per app, throttling – is controlling usage of apis for consumers for a given period of time, securing apis through api keys and user authentication.
in platform architecture API Gateway is like front facing for incoming client requests. Apigee is one very popular api gateway, however it is suitable for enterprise and big applications, for small apps and startups where one can’t afford Apigee, they can go for other API Gateways like Google Cloud End Points but it has very limited features OR Kong API gateway.
Boosting your Client Apps Performance and Stability by Adopting GraphQL
GraphQL is a query language for APIs, it is like giving control to clients/apps to fetch what exactly they want rather than decided by server.
Ask for What you need and Get exactly that 🙂
Send a GraphQL query to your API and get exactly what you want
<code>//GraphQL Query to API
{
product: {
name,
images,
price
}
}
//This will be input to your API and you will get only that in //response quickly
//Response
{
product: {
name: 'Micro Max G6',
images: [],
price: '6599 Rs'
}
}</code>
Get Multiple Information in Single Request
GraphQL will make your apps lightning speed which even out perform greatly in slow network speed as well, because GraphQL is powerful and returns multiple information in single request, which apps typically get it from different api calls.
There are multiple bunch of benefits of GraphQL and its worth to consider using GraphQL while architecting your Platform. Well where GraphQL fits in your platform architecture ? See the updated architecture diagram
There are many ways to design the platform architecture, however every platform has its own unique requirements. The architecture of every platform can vary totally to meet the needs of respective platform.
There are many case studies to look into platform architecture which I will cover in different article, we can look into linked-in, ebay, Amazon and netflix architectures to get better understanding on Platform architecture.
For now in this article we will understand first approach that Having Multiple Apps with its own independent Database and Backend. For our understanding purpose we will consider designing a platform for an Amazon Kind of Online Shopping Platform
Platform Architecture for Amazon Kind of Online Shopping Platform with Multiple Apps Approach
If you consider Amazon kind of Online Shopping Platform, we first have to identify which are the different huge modules which are candidates for making it as a separate app altogether
Sellers or Vendors who sell the products
Seller is the one who has his shop and sell products in amazon
He should be able to add his products
He should be able to manage discounts, offers for his products
He should be able to dispatch his product
He should be able to manage his revenue/profit and other finance aspects
Buyers
The one who buy products in Amazon
He should be able to manage his Cart
He should be able to manage his delivery addresses
He should be able to do payment using his choice of payment mode
Product Dispatch Management
Tracking the product while dispatching from one place to another
Every city has a collection centers and products has to be managed for timely delivery
Products Delivery Management
Once product is reached the destination city it has to be picked by delivery person
Assign delivery person
Deliver to customer and update the status
receive cash in case is Cash-On-Delivery
Amazon Finance Management
This is kind of Super Admin Functionality
Managing refunds
managing failed transactions
Transferring money to Seller
I listed only few functionalities however there will be many more under each of these main features.
Why Can’t be its one super big Monolithic Application ? Rather than multiple Apps ?
To answer this question we have to think of following important aspects which we covered in our first Article Which are as follows
Scalability : For Amazon Kind of Applications, millions of requests will be coming every second. There are millions of users buying and millions of sellers selling and millions of products listed, millions of transactions happening. It is highly impossible to handle everything in one huge monolithic server, because every server has some upper limits with respect to hardware and capacity, one huge database to scale it to that level we have to divide it into multiple pieces and manage it independently.
Maintainability: It is very hard to maintain one huge monolithic app for such a huge volume.
Resilient: Any major update to particular part of the platform should not affect to other parts of the platform
Development: For amazon kind of huge platforms, they have huge team spread across the world and each teach is enhancing their own module by doing lot of research within their small team. It is not possible to manage development activity effectively if its one huge monolithic app
Now let us see how the Architecture Diagram looks: You can use various apps for designing Architecture Diagram
In the above diagram, each app is one server and it has its own database. There are third party services used commonly across many apps, like
Elastic Search Service : For faster product search, Elastic Database is must for better searching capability.
Cloudinary is another service used for multi media file handling
Authentication service is like firebase, auth0 for user authentication
Many more such services will be used
Advantages of Multi Application Platform Architecture
Scalability: In this architecture the platform is scalable to handle high traffic, because during peak time like Amazon big billion day, more and more people will buy products, the traffic will be too high for Consumer App. So in that case if its independent server by itself, we can increase the capacity of that server alone to handle millions of requests, however other apps may not need so much so at this stage.
A platform architecture is an abstract high level design of the platform that answers to some of the key important factors of any platform like scalability, resilience, maintainability. It should cover almost all important big modules, multiple apps, all the servers, databases, caching, third party solutions and etc.
A platform should be ever evolvable, should be able to adopt to the new changes and needs of the platform and it helps to meet all technical and operational requirements with focus on optimising performance and security.
Desirable Properties of Platform Architecture
High level of Abstraction – It should be simple enough and comprehensive enough to get high level of understanding
Resilient – Each major module (app) should not be get affected so much due to one defective app
Scalable – Architecture should consider the scalability of the platform “Scalability” refers to the number of users, sessions, transactions, and operations that can be accommodated by the entire system
Maintainable – Should be able to make changes to any part of the system without breaking or damaging too much to other parts
Ever Enhancing – it should be able to evolve every-time when there is a need of change
Important Aspects to focus on while designing platform architecture.
How many servers we need to maintain, is it per app level or every major functionality
How caching is handled in platform
API Gateways for API security, metering, throttling
Searching functionality within the platform
Users and Identity Management
Security
Databases
File Servers
Third Party Services if any
Data query and Manipulation for faster data access like using GraphQL
Communication between each apps or each servers using event queue, event bus or intermediate app whichever way it is
Handling batch processes using cronjobs or cloud functions
Postgres Sequelize Exception while executing Find Query
Error: Invalid value { noticeTo: { ‘$eq’: ’02faac43-a777-4bb8-902b-da0eebbcd70e’ } } at Object.escape (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/sql-string.js:65:11) at PostgresQueryGenerator.escape (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:986:22) at _joinKeyValue.value.map.item (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:2486:69) at Array.map () at PostgresQueryGenerator._whereParseSingleValueObject (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:2486:52) at PostgresQueryGenerator.whereItemQuery (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:2268:19) at Utils.getComplexKeys.forEach.prop (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:2173:25) at Array.forEach () at PostgresQueryGenerator.whereItemsQuery (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:2171:35) at PostgresQueryGenerator.getWhereConditions (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:2583:19) at PostgresQueryGenerator.selectQuery (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/dialects/abstract/query-generator.js:1315:28) at QueryInterface.select (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/query-interface.js:1122:27) at Promise.try.then.then.then (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/sequelize/lib/model.js:1759:34) at tryCatcher (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromise0 (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/promise.js:649:10) at Promise._settlePromises (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/promise.js:729:18) at _drainQueueStep (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/async.js:93:12) at _drainQueue (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/Users/rama/Data/Development/Social/vk_proj/source/server/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues as _onImmediate at runCallback (timers.js:705:18) at tryOnImmediate (timers.js:676:5) at processImmediate (timers.js:658:5) at process.topLevelDomainCallback (domain.js:126:23)
Solution for Postgres Sequelize Invalid value Error:
Very strange is the query written above is as per the standards of sequelize library, however its throwing a strange exception. I fixed by changing the query as follows
NOTE: instead of using $or and $eq, I am using sequelize op ( operator )
Update: As per latest sequelize library v4.x, For better security Sequelize recommends dropping alias operators $ (e.g $and, $or …) – Reference
This kind of issue usually occur when we try to compare mismatched data types. You cannot compare an integer with a varchar. PostgreSQL is strict and does not do any magic typecasting for you. I’m guessing SQLServer does typecasting automagically (which is a bad thing).
The above code throws operator does not exist: integer = character varying exception, because we are trying to compare integer with string. To fix this issue change as follows
<code>const phone = "9998886660"; //this will fix the issue
UserModel.count({
where: {
phone: phone
}
});</code>