The Business Need
One of the world’s largest information providers had been providing competitive insights to companies in the global petrochemical market encompassing fast-growing energy and fertilizer division. This was achieved by delivering trusted pricing data and high-value news, analysis and independent consulting services thereby enabling customers to make well-informed trading and planning decisions. They needed technical expertise to collect and aggregate data from 62 different operator sites related to gas commodity, on a real-time basis.
Challenges we faced
The major challenge was to develop source-specific crawlers as different operators provide data in different measurement units, like million cubic meters (MCM) - a volume measurement and kilowatt-hours (KWH) - an energy value measurement.
They required technical expertise to aggregate natural gas commodity data like production, demand and unit capacity, amount of gas stored, outflow, import and export from the Transmission System Operators (TSO), Storage System Operators (SSO) and LNG System Operators(LSO), on a real-time basis.
Operators update the data on their websites on a daily basis or intraday basis, at different times of the day. The client wanted the process to query operator websites for new data regularly.
How we solved the problem
Mobius process incorporated an automated solution to scrape and aggregate the data on an hourly basis. Regular checks were performed to ensure all data were being captured and normalized. The normalized data were then validated i.e. checked for zero values or significant day-on-day fluctuations. Mobius process also retained all the historical data for a requested period, for diagnostics and refreshing purpose. The automated process scraped various online data on an hourly basis and published the latest data to the client’s SFTP in two different measurement units – MCM and MWH.