The Business Need
A smooth information flow stands as the prime objective of any supply chain management. Keeping the supplier and the product end user up-to-date on their supplies and orders was the requirement of a key player in the rewards solutions provider. While most of their SCM workflow was automated and a unified view was presented in their integrated rewards supplier portal, crucial pieces like order placement and tracking could not be moved into their portal. They needed a workaround for providing order details from a large number of supplier websites in real-time. Owning an elaborate crawl manager platform - Mobito, our experts at Mobius took up the large-scale crawl project with ease.
Mobius had to collect the product feeds from 45 websites on a daily basis and 4 websites on weekly basis. The collected information was then to be mapped under the client category structure and uploaded into their vendor database tool.
Challenges we faced
Identifying rules for extracting valuable data instances which are often presented in complex structures from web pages consumed a lot of time.
The sites were from various geographical locations in different languages. Extraction from these sites posed a huge challenge as it could be accomplished by handling multiple character-set and encoding types. Simultaneous requests to different supplier websites were handled through throttling mechanism, website blockages and crack captcha requests automatically.
How we solved the problem
Mobito – Mobius Cloud-based Web Extraction Platform built on cloud infrastructures like AWS, Azure & GCP, was configured as “Platform as a Service” (PaaS), which fully automated the order management process. The platform was integrated with the client’s portal through REST API Endpoints. The request and response would be in XML format.
Data scraping experts from Mobius developed an army of Bots atop Mobito to crawl and fetch order details like order status, tracking details from multiple supplier portals. By specifying the websites and meta category names, Mobius created site-specific crawlers to fetch all the order details and create feeds as per client’s conventions, thrice a day to ensure no updates are overlooked or missed. Errors within the supplier websites were reported at once to the client via email.
The orders that were needed to be monitored and fulfilled were pulled from the portal periodically. The collected details would be pushed back to the rewards portal. Daily reports, trends, and stats could be viewed through the platform dashboard. The real-time information thus sourced from the supplier sites was populated in the client API. Between every run, all the bots were monitored to check for any issues in their functionality with a 24/7 support facility.
Results
Mobius’s automated process utilized the best of Cloud optimizations and processed millions of request in Auto scale mode. The solution helped the client to maintain their commitment to provide periodic order fulfillment status updates to their end-users, which would otherwise be time-consuming on a real-time basis. All the details were fetched automatically by handling complex navigations, thereby extracting only precise content and reducing the downstream processing requirements.
The missing pieces in the customer’s order management process were easily positioned and integrated seamlessly with the client’s portal. Since the arduous task of updating the order details was automated by Mobito, efficiency at the client’s workplace shot up as they were able to focus more on adding value to their customer offerings.