Big Data, Big Problem



Every company, every organization relies on research: about competing companies and related industries; about new product opportunities and technologies; about market dynamics and new geographic markets; about regulatory context and policy debates…whether they pay an outside firm to do custom research for them, invest time and money in producing research themselves, or use whatever relevant information they can find or purchase off the shelf. Yet the conventional business research process is time-consuming, wasteful, requires highly-skilled operators performing largely manual, handcraft work, produces inconsistent results, relies primarily on off-the-shelf technology (web searches, phone calls), and often requires high-priced subscriptions to primary source material.

No wonder so many organizations outsource the research and analysis of primary market data to costly consulting, market and investment research firms who create and maintain industry databases manually. Yet even after paying high prices for manually aggregated and analyzed information, customization costs even more, and typically has a long lead time.

In all the years of business analytics, business intelligence and big data automation and now enterprise 2.0 sharing and content curation, very little IT has been applied to the problem of bringing unstructured data and information from outside the enterprise into the enterprise in a structured way until very recently. Sourcing, structuring and transforming unstructured external business data and information remains one of the big unsolved knowledge and workflow problems in business.