Friday, Jan 19, 2018
HomeTopicsBig DataDirector of Product QA and DB Design of SMC3

Director of Product QA and DB Design of SMC3

What are the most important considerations for DBAs and architects to consider as they approach big data?

AC: DBAs are constantly looking for ways to provide the best performance, better replication, and the best future for their company.  Reshaping a company’s architecture is no small task, but staying up-to-date about your options keeps DBAs and developers on the forefront of company innovation. Regarding big data, scalability and high availability are essential.

How has SMC3 managed their data to ensure uptime and reliability for their customers?

AC: Fulfilling the high demand placed on our data is of utmost importance to us. Thousands of shippers access our data each day in order to compare transportation pricing options and optimize their LTL (less than shipments, and we see millions of transactions per month. If this information becomes unavailable, even for a short time, our customers could lose millions in revenue. So we did exhaustive research and found that while proprietary solutions offered some benefits, there were superior advantages to be gained from open source alternatives. At the core of our business is a rock-solid architecture that ensures our data is available.

What made you start to consider open source for your database?

AC: The fact that open source was free to use and vendor neutral made the decision to develop with it easy – it cost us nothing but our own time to investigate, and we ultimately found that it offered performance on par with the big proprietary database companies. Another benefit was that with open source options, we know that even if every company that offered support disappeared, we’d still be able to maintain it ourselves, and could always rely on the community for contributions and support.  Now we use MariaDB, the open source database, and are very happy with it.

What recommendations do you have for organizations needing to scale and manage costs?

AC: Evaluate your options to ensure that your database choice can scale while maintaining high availability and reliability. Conduct tests based on your actual usage of products so you get an accurate picture of performance. In our case, we ran our general query log for 30 minutes on a production server, and captured every database command. We then repeated this activity, running the same commands on each of the different database products – both proprietary and open source – to get full transparency into each option’s capabilities.

What are benefits of open source databases?

AC: We’ve found great results with MariaDB as it offered shorter run times, and even a 30-40% speed increase for some applications versus some proprietary databases. Besides being a reliable open source option, MariaDB also featured key performance enhancements not found elsewhere. It offered better replication with easier to read Global Transaction IDs, and allowed multiple threads per schema – providing value to our business. Since our migration to MariaDB, which took just two days, the database has been stable, secure and trouble-free, and the open source code has kept administration simple. We found open source to be cost-effective, while also providing the tailored support necessary for easily handling of operational questions and making bug fixes simple.