Price Points by Omnia Retail

Introduction Omnia processes millions of retailer’s products information each day to deliver the best prices according to their needs. To gain advantage from machine learning techniques Omnia began a new project to...

Introduction Omnia processes millions of retailer’s products information each day to deliver the best prices according to their needs. To gain advantage from machine learning techniques Omnia began a new project to backup and utilize its data to improve its business analytics and explore machine learning. The project has two main objectives: Backup data (about 38 GB per day, with expected growth in the coming months); Ability to create multiple ETL (Extract, transform, load) processes to make data available for diverse purposes a. This project ETL process is to transform unstructured data into business report data. Alongside that, we add the following technical requirements: Progress traceability and monitor the backup and the ETL; Facility to recover if something goes wrong in the process; Ability to scale the speed linearly to the amount of data; Flexibility to create multiple distinct ETL processes. As for constraints: Internet speed from our datacenter to Azure (about 50 MB/s); Couchbase version (3.0.0), cluster size and specs – to retrieve that for backup.