System design for millions of users (Part 2)

Designing a system that supports millions of users isn't a small challenge, it requires continuous improvement. Following system design for millions of users (Part 1), we will continue to explore ways to build systems that scale to millions of users.

System design for millions of users (Part 2)


System design for millions of users (Part 2)

Designing a system that supports millions of users isn't a small challenge, it requires continuous improvement. Following system design for millions of users (Part 1), we will continue to explore ways to build systems that scale to millions of users.



1. CDN

A CDN is a geographically dispersed network of servers used to deliver static content. Static content includes images, videos, CSS files, JS files, and more.


Dynamic content caching is a new concept and is beyond the scope of this article so you can cache HTML pages based on request path, query string, cookie, and request header.



Workflow of CDN




  • When the user gets image.png using the URL of the image. The domain name of the URL is provided by the CDN. The last two image URLs are used to indicate the status of the image URLs on the Amazon site and Akamai CDN.
  • If image.png is not in the cache of the CDN server, request the file from the native webserver or online storage such as Amazon S3.
  • The origin returns an image.png to the CDN server that contains an HTTP header Time-to-Live that describes the retention period of the cached image.
  • The CDN caches the image and returns it to User A. The image remains in the cache until the TTL expires.
  • User B sends a request to the same photo.
  • If the TTL has not expired, the image will be returned from the cache.




2. Stateless web tier

To scale out the web tier, you need to transform the web tiers state. This is a challenge for long-term in-memory session data such as SQL and NoSQL. Each web server in the cluster has access to database state data, which is called the stateless web tier.




3. Stateful architecture

Some differences between stateful and stateless servers are that stateful servers store client data (states) from one request to the next. The stateless server does not need to remember its state.



User A's session data and user image are also stored on Server 1. To authenticate User A, an HTTP request must be sent to Server 1. If the request is sent to another server, such as Server 2, the request will receive an error. That's because A's session isn't there. Similarly, the HTTP request to authenticate User B should be sent to Server 2 and User C should be sent to Server 3.




4. Stateless architecture


In the stateless architecture, HTTP requests from users can be sent from shared storage to any web server that loads state data. State data is stored on a data share that is external to the webserver. Stateless systems are simpler, more powerful, and extensible.



Move session data from the web tier and store it in persistent storage. Data sharers include RDBMS, Redis, NoSQL, and so on. NoSQL is the easiest choice for extensibility. Auto-scaling refers to the process of automatically adding or removing web servers based on traffic. Autoscaling works after the state data is cleared from the webserver.


The website will be gradually expanded to attract many overseas users. Supporting multiple data centers is essential to improve availability and provide a better experience in all regions.




5. Datacenter

The following figure is an example of two data centers.



In normal operation, users are geographically routed and geoDNS is routed in English. geoDNS is a DNS service that allows domain names to be resolved to IP addresses based on the user's location. If the data center goes down, it forwards all traffic to the active data center.


Technical challenges to tackle when setting up a multi-datacenter

  • Traffic navigation

Effective tools are needed to drive traffic to the right data center. GeoDNS allows you to direct traffic to your nearest data center based on your location.


  • Data synchronization

Users in different domains can use different local caches or databases. In the event of a failover, traffic can be forwarded to a data center where data is not available. A common strategy is to replicate data across multiple data centers.


  • Testing and deployment

When setting up a multi-data center, it's important to test your website/app in different locations. Automated deployment tools are important for maintaining service consistency across all data centers.




6. Message Queue

Message Queuing is a persistent in-memory component that supports asynchronous communication and acts as a buffer and delivers requests asynchronously. The basic architecture of message queues is very simple.


An input service, called a publisher or producer, composes a message and publishes it to a message queue. Other services or servers, called subscribers or consumers, connect to the queue and perform the actions defined in the message.



Separation makes Message Queuing a great architecture for building scalable and reliable applications. Message Queuing allows publishers to create messages in the queue so that non-existent subscribers can process the messages later. Subscribers can read the message even if there is no publisher.


In the image below, the webserver is uploading the image processing to the message queue. Image processing workers receive work from the message queue and perform image customization tasks asynchronously.


Publishers and subscribers can scale independently. As the queue grows in size, more workers are added to reduce processing time. However, in most cases, if the queue is empty, the number of workers can decrease.



Millions of users

System expansion is an infinite loop. We can learn something new every time repeat. New strategies for expanding to millions of users require more tweaking. For example, you may need to optimize your system and divide it into more uniform sub-services. All the techniques learned in this lesson provide a good foundation for solving new problems. 

  • Stateless web architecture
  • Make a backup everywhere
  • Cash as much as possible
  • Support for multi-data centers
  • Save static resources to your CDN
  • Data scaling using sharding
  • Split the hierarchy into multiple devices
  • Monitor the system and use automation tools



If you are considering offshore development, please feel free to contact us.

※Here is our contact information.

Account Manager: Quan (Japanese/English available)

Phone number: (+84) 2462 900 388

Email: contact@hachinet.com

Please feel free to contact us for consultation/application by phone.

Click here for more information ▶