Comments
yourfanat wrote: I am using another tool for Oracle developers - dbForge Studio for Oracle. This IDE has lots of usefull features, among them: oracle designer, code competion and formatter, query builder, debugger, profiler, erxport/import, reports and many others. The latest version supports Oracle 12C. More information here.
Cloud Expo on Google News
SYS-CON.TV

2008 West
DIAMOND SPONSOR:
Data Direct
SOA, WOA and Cloud Computing: The New Frontier for Data Services
PLATINUM SPONSORS:
Red Hat
The Opening of Virtualization
GOLD SPONSORS:
Appsense
User Environment Management – The Third Layer of the Desktop
Cordys
Cloud Computing for Business Agility
EMC
CMIS: A Multi-Vendor Proposal for a Service-Based Content Management Interoperability Standard
Freedom OSS
Practical SOA” Max Yankelevich
Intel
Architecting an Enterprise Service Router (ESR) – A Cost-Effective Way to Scale SOA Across the Enterprise
Sensedia
Return on Assests: Bringing Visibility to your SOA Strategy
Symantec
Managing Hybrid Endpoint Environments
VMWare
Game-Changing Technology for Enterprise Clouds and Applications
Click For 2008 West
Event Webcasts

2008 West
PLATINUM SPONSORS:
Appcelerator
Get ‘Rich’ Quick: Rapid Prototyping for RIA with ZERO Server Code
Keynote Systems
Designing for and Managing Performance in the New Frontier of Rich Internet Applications
GOLD SPONSORS:
ICEsoft
How Can AJAX Improve Homeland Security?
Isomorphic
Beyond Widgets: What a RIA Platform Should Offer
Oracle
REAs: Rich Enterprise Applications
Click For 2008 Event Webcasts
Sharding for Scale | @DevOpsSummit #BigData #DevOps #Microservices
Where you decide to shard for scalability impacts the complexity of the entire architecture

Sharding for Scale: In the App or in the Network?

Sharding has become a popular means of achieving scalability in application architectures in which read/write data separation is not only possible, but desirable to achieve new heights of concurrency. The premise is that by splitting up read and write duties, it is possible to get better overall performance at the cost of a slight delay in consistency. That is, it takes a bit of time to replicate changes initiated by a "write" to the read-only master database. It's eventually consistent, and it's generally considered an acceptable trade off when searching for higher and higher scalability.

While the most well-known cases of read/write separation and sharding are based on geography - east coast versus west coast, for example - there are other cases where localized sharding has also been put into play with great success. Generally these types of architectures base their sharding decisions on user names, splitting them up between databases based on statistical analysis of occurrences. This achieves greater scalability at the data layer by better distributing the rate at which writes (which are generally speaking a blocking action) occur, and thus achieving greater scale and concurrency for only a slight period of inconsistency.

The mechanism is, in theory, quite simple and is loosely based on an algorithmic principle taught in most computer science algorithm classes: hashing. Basically when a request comes in, the application looks at some piece of data - like the user name - and based on that data decides which of X databases to send it to. How that division is determined is not as relevant (to this discussion, anyway) as the action itself. It's like registration at an event or in college where you're split up based on the first letter of your last name. You remember, every one whose name starts with A-G go in this line, you others go over there, in those lines.

That's sharding. And it's most commonly implemented in the application, where the connection to a database is created and used to manage the data that is the lifeblood of every application and business today.

Now, I told you all that to share another approach to sharding; one that takes advantage of programmability in the network (data path programmability, to be precise).

Shard in the Network or the App?
In the first scenario, in which sharding occurs in the application, there's almost certainly (I'd be willing to bet real live money) a load balancing service in front. It's distributing requests to a pool (cluster) of application instances, each of which individually decides which database to talk to given the data available. If we insert some intelligence - some programmability - into that load balancing service, we move the sharding decision in front of the application.

Now when a request comes in, the load balancing service examines the data available and decides to which application instance the request should be sent. The data is likely the same - a user identity - but may be something more applicable to the service, say a product name or number extricated from the URL of a RESTful API fronting a microservice.

sharding - network versus application

Basically what you're doing is taking the block of code responsible for sharding from inside the application and moving it into the network.

In the diagram to your left (or at least it was on the left when I wrote this) illustrates. The reason the "apps" in the "network" illustration are colored is to highlight that each of them is dedicated to a specific database. The code - the app itself - is all the same. There's no difference except for the configuration that tells it "you are dedicated to database A-G".

This is starkly different from the "in the application" sharding example in which all instances are exactly the same, including the configuration, as each one may be at any time talking to any one of the databases, depending on the data received.

Now, I believe it's obvious (because I colored all the database connections) that when you shard in the application, the complexity of the network is pretty high, as well as the load on each database as it has to maintain connections with each and every application instance. Operational Axiom #2 tells us "as load increases, performance decreases" so it's likely we're seeing an impact on performance (in a negative way) from a sharding in the application architectural approach.

Conversely, the network complexity in the sharding in the network approach is fairly low (and straight forward) and actually simplifies the entire architecture. The load on the databases themselves remains lower because there is only one instance (or pool of instances) with which it needs to manage connections.

The negative of the "in the network" approach is that you have another component (service) that must be managed - that means application lifecycle management applies -  and there are likely separate configurations necessary for each of the pools (clusters) responsible for scaling out each application instance (because each pool only talks to one of the databases). But this negative also means that the code responsible for sharding is localized, it is itself a "network microservice" that is small and isolated, meaning it can be tweaked independently of the application code. That's a positive, especially if there might be a need to increase the level of sharding or change its core mechanism required to scale the application. That's one of the reasons microservices are growing more popular; the localization and isolation ensure the ability to change without disruptive impact on other services or applications.

Taking advantage of programmability in the network to achieve new levels of scalability while simplifying your architecture is another reason programmability in the network is an invaluable tool in your architectural toolbox.

About Lori MacVittie
Lori MacVittie is responsible for education and evangelism of application services available across F5’s entire product suite. Her role includes authorship of technical materials and participation in a number of community-based forums and industry standards organizations, among other efforts. MacVittie has extensive programming experience as an application architect, as well as network and systems development and administration expertise. Prior to joining F5, MacVittie was an award-winning Senior Technology Editor at Network Computing Magazine, where she conducted product research and evaluation focused on integration with application and network architectures, and authored articles on a variety of topics aimed at IT professionals. Her most recent area of focus included SOA-related products and architectures. She holds a B.S. in Information and Computing Science from the University of Wisconsin at Green Bay, and an M.S. in Computer Science from Nova Southeastern University.

Latest AJAXWorld RIA Stories
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structu...
"We're developing a software that is based on the cloud environment and we are providing those services to corporations and the general public," explained Seungmin Kim, CEO/CTO of SM Systems Inc., in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the ...
"Our strategy is to focus on the hyperscale providers - AWS, Azure, and Google. Over the last year we saw that a lot of developers need to learn how to do their job in the cloud and we see this DevOps movement that we are catering to with our content," stated Alessandro Fasan, He...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around qualit...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct ...
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
Click to Add our RSS Feeds to the Service of Your Choice:
Google Reader or Homepage Add to My Yahoo! Subscribe with Bloglines Subscribe in NewsGator Online
myFeedster Add to My AOL Subscribe in Rojo Add 'Hugg' to Newsburst from CNET News.com Kinja Digest View Additional SYS-CON Feeds
Publish Your Article! Please send it to editorial(at)sys-con.com!

Advertise on this site! Contact advertising(at)sys-con.com! 201 802-3021


SYS-CON Featured Whitepapers
ADS BY GOOGLE