Remember back in 2010, in-memory computing was the “thing.” Flash forward and today, every traditional database vendor provides some kind of in memory capabilities in their disk based database. It’s clear that in memory database is out of the hype stage and has become a necessary technology for today’s business to achieve competitive advantage.
Being in the business for 17 years with 3 years of focus on in-memory, I see all kinds of questions from customers, prospects and analysts. Based on numerous conversations, here are the top five key things I believe a customer should consider when selecting an in-memory database. These are not in priority order.
An In-Memory Database should
1) Do both transactions and analytics on a single copy of data: Separate databases for transactional and analytical applications will require additional IT systems to move data from the transactional database to the analytical database. This also increases the time it takes to see the results from the time the actual transactions occurred. Unless the database can do both transactions and analytics in one database instance, it is not possible to get real-time insights as transactions happen. Traditional disk based database systems with an in-memory option keeps data in row format for transactions and keep a subset of that data (setup by the DBA) in columnar format for analytics. This approach is better than separate database systems for transactions and analytics but still does not provide real-time results and requires more system resources and DBA wizardry.
2) Keep all the data in memory unless otherwise specified: Data must be in memory to get results in a predictable time. The time it takes to get results for unplanned queries really depends on the number of tables in memory / on disk that are required to complete the query. Columnar store, advanced compression techniques, the ability to process both transactions and analytics from single copy of data allows in-memory databases to keep more data (if not all) in memory. In-memory columnar databases do not require indexes and materialized views to reduce I/Os from the disk to speed up results. This saves more memory and helps to keep more data in memory. Unless all the data is in memory it is not possible to drill down into granular details from high level dashboards to get a complete picture of what is really happening in the business. Having the complete data set in memory also helps to predict the future more accurately and quickly. Real-time insights for unplanned queries and ability to quickly go to the details are not possible with a database that keeps data in disk by default and only if instructed loads data into memory for processing.
3) Process more business logic and advanced analytics close to the data: When traditional databases were originally created in 1970s, due to computing constraints and to meet application requirements dedicated specialized systems were created. Few such systems are web servers, predictive analytic systems, text analytics systems, geo spatial processing systems and rule engines. Due to this, current IT landscapes significantly increase data latency, data duplication, IT landscape complexity and add layers of resources to manage and build applications on these systems. A modern in-memory database should be able to execute most, if not all, of the business logic and advanced analytics inside the database on a single copy of data used for both transactions and analytics - so that results are always fast and real-time. This helps businesses become more agile in leveraging opportunities and reacting to the potential risks.
4) Provide tools to develop applications faster for any device and integrates well with open source software and other traditional databases: Today’s developers are comfortable with open source application development tools such as Eclipse, and languages such as SQL, JSON, JavaScript and JavaScript-based UI frameworks that deliver HTML5 UIs. Open source software technologies such as Hadoop and Apache Spark provide excellent solutions to solve some business problems. An in-memory database should provide seamless integration with traditional databases and key open source innovations. Developers should not be deciding whether to keep a table in memory or not for every table used in an application. Developers should not be having to decide whether to keep a table in memory or not for every table used in an application. Developers should not be testing their old applications after dropping OLTP indexes because they keep the table in memory. An In-memory database should also provide built in data visualization tools.
5) Give choice in deployment options: Organizations expect to consume software through a combination of on-premise and via the cloud. A Database that can be accessed from a public cloud will significantly reduce initial capital expenditure and the time it takes to get started. Once an organization decides to go to production, a managed cloud service can provide the security, support and service levels organizations expect for enterprise-grade applications. Organizations should be able to run an in-memory database on-premise using commodity servers such as x86 and should fit into the new paradigm of a Software Defined Data Center.
An In-memory database may require an initial investment and requires data migration, just as any database does, but in addition should have a proven track record of simplifying IT landscapes and providing recurring cost savings. This is in addition to the value organizations expect – becomes an agile asset and empowers developers to build applications that can deliver competitive advantage.
In summary, a traditional database that depends on disk but gives incremental performance with in-memory enhancements may look good in the short term as this gives the ability to run applications as is. But the fact is that it requires more administration, hand-tooling and developer work to select the tables to keep in memory and optimize applications on an ongoing basis. These databases utilize more system resources to keep less data in memory and to synchronize multiple copies defeats the purpose of the true in-memory database that I have defined above.
Those are my five. Do you have yours? What would you add to my list?
See how SAP HANA can help @ Innovation & Me
About SAP HANA
Projected Cost Analysis of the SAP HANA Platform by Forrester
