Improving Search Performance: Travolic’s Successful Migration of Places Data to Elasticsearch
Introduction
Travolic is a metasearch engine that aggregates flight prices from multiple airlines and travel agencies, providing users with a convenient and reliable platform to compare and book their travel arrangements. However, as the number of users and the volume of data stored in the system continued to grow, we were faced with the challenge of optimizing search performance while ensuring that data accuracy remained a top priority. To overcome this challenge, we decided to migrate our search places data, which encompasses information on airports, cities, and stations, from MongoDB to Elasticsearch. This article aims to delve into the migration process and highlight the impact it had on Travolic’s overall performance.
Background
At Travolic, the places data plays a crucial role in the search engine by enabling users to discover the closest airports and relevant travel details. Even though MongoDB was functioning satisfactorily, Travolic was exploring avenues to enhance the search performance, especially with respect to response time.
Travolic’s team recognized Elasticsearch as a promising solution due to its exceptional speed and language capabilities. Elasticsearch is an open-source search engine that is constructed on top of the Apache Lucene search library. The platform is specifically created to handle enormous quantities of data, which is precisely what Travolic needs for their constantly expanding data requirements. Furthermore, Elasticsearch offers an extensive range of language support, which is essential for a metasearch engine platform that serves a global audience.
Migration Process
The migration process involved moving the places data from MongoDB to Elasticsearch. The team worked closely with Elasticsearch experts to ensure that the migration was carried out seamlessly and without any disruptions to the platform’s operations.
To begin the migration process, the team designed an Elasticsearch schema that aligned with the current MongoDB schema by mapping fields from the MongoDB collection to their corresponding fields in Elasticsearch. Following schema definition, the team leveraged Elasticsearch’s bulk API to transfer the data into Elasticsearch.
While migrating the places data from MongoDB to Elasticsearch, the team faced certain data accuracy issues. Specifically, some of the data in the MongoDB collection was not normalized correctly, which resulted in inconsistencies when it was loaded into Elasticsearch. Consequently, the team had to manually clean up the data and ensure that it was correctly normalized before loading it into Elasticsearch.
A further obstacle encountered by the team during the migration process concerned nested documents. Although MongoDB supports these structures, Elasticsearch does not. Consequently, the team had to flatten the nested data and store them as individual fields in Elasticsearch.
Impact on Performance
Travolic’s search performance was significantly impacted by the migration to Elasticsearch. Before the migration, search queries had an average response time of approximately 2–5second. However, following the migration, the average response time improved to between 50-80 milliseconds.
One of the main factors contributing to the improved performance was the speed of Elasticsearch. Elasticsearch is designed to handle large amounts of data and can quickly index and search data, making it an ideal solution for a platform like Travolic that requires fast access to travel-related data.
Another factor that contributed to the improved performance was the language support offered by Elasticsearch. Elasticsearch has built-in support for many languages, including English, Spanish, French, and German. This is particularly important for a platform like Travolic, which serves customers from all around the world and needs to support a variety of languages.
Post-Migration
Once the migration from MongoDB to Elasticsearch was completed, a new challenge arose for the team at Travolic: how to keep the data in sync between MongoDB and Elasticsearch. Specifically, they needed to ensure that updates or deletions made to documents in the airports and cities collections in MongoDB were also reflected in the corresponding documents in the places index in Elasticsearch.
To address this issue, the team decided to use a message queue system, specifically RabbitMQ, to handle the synchronization process between MongoDB and Elasticsearch.
They created a service that would publish an event with the updates or deletes as they occurred in MongoDB. Another service was then created to consume these events and apply the changes to the corresponding documents in the places index in Elasticsearch.
By using RabbitMQ to handle this synchronization process, the team at Travolic was able to ensure that any changes made to the data in MongoDB would be quickly reflected in the corresponding documents in Elasticsearch, without the need for manual intervention or updating. This helped to maintain the integrity and accuracy of the data across the platform, ensuring that users could always access the most up-to-date information.
Conclusion
Travolic’s migration from MongoDB to Elasticsearch has been a huge success. The migration process went smoothly, and it significantly improved the platform’s performance, reducing the average response time for search queries from 2–5 second to between 50-80 milliseconds. The migration to Elasticsearch has also set Travolic up for future growth and expansion, as Elasticsearch’s capabilities make it a scalable solution to handle the platform’s growing data needs.
