One of the important aspect of any platform is performance, to increase the performance to satisfy end users with quick access one can not escape from in memory cache. Database operations are always costly, executing complex queries to meet real time needs will make the entire platform slow. For faster access, in-memory cache is very much needed. People use Radis and Kafka.
Caching in platform Architecture
Now interesting question is where do Caching service fits in my architecture? How do we represent it in platform architecture and where do we integrate caching?
Typically we should have a cache manager within the app which manages caching and database. For any request first we have to check whether required data is in cache ? if not then only hit Database else return from cache.
In case of data update or new insertions , the cache manager has to take care of updating the cache and then make the updated data available for further read operations.
The updated platform architecture looks as follows now
Redis or Kafka are well known services for in-memory cache, but both have their own purpose, so we have to decide upon what to use when ?
File Server for file storage in platform architecture
I thought of covering this also in the same article as it is a small but very important to consider. Because lot of developers have confusion regarding where to store files ? many people will create a folder in their application server and store all files there itself, which is a very wrong for ay real-time applications.
It is recommended to store files in file servers like Amazon AWS S3 bucket and Google Storage, if its publicly sharable multimedia file like image or video then go for Cloudinary kind of CDN services.
Why not to store files in application server?
- Application servers like amazon EC2 or Google App engine are not made for file storage, their purpose is to run your app
- Application servers are compute engines where you can install any software whichever you need for you app to run in cloud, where as file servers are just providing the storage to store huge files
- File servers have different qualities altogether they can take you files backup, create replica if needed, they are capable to transfer huge data fast enough, they are cheaper than app servers
- in case of app servers, for some reason if something goes wrong, u may need to create new server quickly and release your app again, if your all file data is also stored in the same server you may loose it permanently, but file server are more reliable for that matter
- This is what amazon claims -> Amazon S3 is designed for 99.999999999% (11 9’s) of data durability because it automatically creates and stores copies of all S3 objects across multiple systems.
- File servers also provide other services like securing files through authorised access, encrypt the files
- Manage and access control – you can create multiple folders and access each folder for specific service so that other services won’t get access to it
- File servers like Amazon S3 provide services like querying on the data stored in files, all such nice features you won’t get it in app servers because they are not meant for it