Sherlock Holmes is known for his ability to make clever deductions from the analysis of a heap of seemingly insignificant details. That unique skill allows him to solve crimes that no one else can solve. Today, companies employ similarly clever -- if not always quite so dapper or well known -- investigators to extract information and synthesize solutions to corporate problems. Known as business intelligence (BI), the field is staffed by people with outstanding analytical skills who either design or apply computer tools for analyzing large data sets related to business operations. It's an information technology (IT) job, but some people come to it with a scientific background.

Big data

Sherlock Holmes has an advantage over modern big-data investigators: The quantity of data he has to sift through is manageable, at least for a mind like his. Today's problem-solvers must extract tiny clues from huge data sets, which grow so rapidly that ever more advanced data-mining tools are needed to handle them. Companies from most of the largest sectors of the economy, including retail, manufacturing, pharmaceuticals, and finance, are keen to make the best possible use of these data. That desire has led to the emergence of companies that specialize in extracting a strategic advantage from hyperbytes of data.

Back in the early 1970s, the first BI companies, such as Cognos, pioneered the design of software and computer tools able to transform raw business operations data into analyzable information -- information that BI analysts could then use to answer companies' business-related questions. Seeking a slice of the BI market, larger IT companies acquired many of these small BI companies. (IBM acquired Cognos; SAP acquired Business Objects.) The work was technical, and BI companies usually did the analysis for client companies. Software giants Microsoft, Oracle, and SAS developed their own BI expertise.

Lately, a new generation of BI companies has emerged, among them California-based Cloudera. These companies began to provide client businesses with easier-to-use tools, allowing them to query their own data and analyze the resulting information, obtaining answers without the assistance of outside BI analysts. Other Cloudera-type BI companies have also emerged, including Splunk, Tableau, EMC's Greenplum, MicroStrategy, QlikTech, Information Builders, TIBCO Spotfire, and Yahoo!'s recent spinoff, HortonWorks.

Peter Lancett (CREDIT: Kathleen Tyler/IBM)
Peter Lancett entered business intelligence with a 1-year work placement in sales operations at IBM in London during his undergraduate studies. Read a profile of Lancett here.

This expansion means new opportunities for scientists (and others) interested in the field. A June 2011 McKinsey Global Institute report predicts that over the next 7 years, in the United States alone, there will be "a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts to analyze big data and make decisions based on their findings." These projections cover talent employed by all organizations dealing with BI information (not just BI companies). The McKinsey report is based on the assumption that companies from all sectors of the U.S. economy will fully adopt big-data techniques by 2018. If that happens, the report estimates, the demand for skilled employees will exceed the available workforce by as much as 60%.

It is early days, but already there are signs of the expected expansion. "New positions are springing up all over the place," says Paul Miller, an analyst and cloud computing specialist at the Web 2.0 technology news portal GigaOM. Miller adds that "people who can work with the background, scary [technical] stuff but also on the business end" are likely to be in demand.

A new profession, dubbed "data scientist in residence," is emerging. Venture capital firm Greylock Partners recently hired D. J. Patil, the former chief scientist of business social networking site LinkedIn (who holds a Ph.D. in applied mathematics) to serve as an in-house data specialist. Patil's role is to employ his knowledge of BI applications to help non-BI companies financed by Greylock Partners exploit the potential of big data for their businesses.

Technical savvy and communication skills

Stephen Gold (CREDIT: Kathleen Tyler/IBM)

BI companies employ a broad range of technical specialists, including software developers, test team coordinators, quality assurance engineers, analysts, IT architects, configuration managers, and project managers, says Vikki Sly, U.K.-based global recruitment director at the BI company QlikTech, which is headquartered in Radnor, Pennsylvania. For all of these jobs, "analytical aptitude is essential," says Stephen Gold, director of worldwide marketing and global education at IBM in Chicago, Illinois.

A scientific background can be helpful, too. One of the key skills in BI, as in academic research, is the ability to ask the right question, says Raif Majeed, who leads a quality assurance engineering subteam in Seattle at Tableau, a BI company that was founded by two Stanford University computer scientists. "My thesis advisor, Prof Marcia Baker at the University of Washington [UW, Seattle], instilled in me the importance of knowing what specific question you are trying to answer," writes Majeed, who did a Ph.D. at UW Seattle in atmospheric sciences, in an e-mail. "The nature of the data, the processes behind data collection, and the scope of the analysis differ between business intelligence and physical science, but the questions are surprisingly similar. What is the trend? What are the outliers? Why? What is this telling us?"

Raif Majeed (CREDIT: Raif Majeed)

Christian Marcazzo found his molecular biology and biochemistry degree at the University of California, Berkeley -- and his "ability to apply the scientific methodology and approach to problem solving" -- essential when he took a BI position as senior director for life science marketing at TIBCO Spotfire in London, he says. An understanding of the kinds of problems client companies need to solve is important, too. Marcazzo's prior experience working in companies providing bioinformatics and genomics software to pharmaceutical and biotech companies proved valuable in helping his BI clients.

Christian Marcazzo (CREDIT: Epicore Software Corp.)

A statistically intensive social science degree, or a social science degree combined with a module in statistics, would also be sufficient preparation for many jobs because companies usually offer applicants on-the-job training, Miller says. Lund, Sweden–based Anders Samuelson, director of the project management office at QlikTech, advises studying economics in addition to computer science to better understand how to shape products to meet customers' expectations.

According to Sly, QlikTech also pays attention to a candidate's customer orientation, leadership skills, and ability to work in a team. Increasingly, BI companies "bring people who have other skills to be able to communicate to the business side of operations," Miller says. Now TIBCO Spotfire's sales director for the life sciences in the Europe, Middle East, and Africa region, Marcazzo spends most of his time talking to client companies in the life sciences, manufacturing, and consumer goods sectors to identify problems his BI tools can help them solve. This entails presenting complex BI information to less data-savvy management people. "Where scientists [working in BI] sometimes struggle is that they fail to convey the relevance of what they do to solve a problem by getting too deep into the details," Marcazzo says.

Increasingly, BI companies are getting involved in the training of potential employees before they even knock on the door. For example, IBM is collaborating with Yale University, Fordham University, the University of Connecticut, and DePaul University in the United States. In Europe, the company is partnering with the University of Ulster in Northern Ireland; the Institut d'Administration des Entreprises, Aix-en-Provence; and the École des Dirigeants & Créateurs d'Entreprise, Paris, to provide certification programs in analytical skills.

Finding a way in

Joseph Misiti (CREDIT: Joseph Misiti)
A user of social media, Joseph Misiti saw an opportunity to apply big-data algorithms to social media data. He launched Social Q, in March 2011. Read a profile of Misiti here.

A good starting point for scientists interested in entering the field is the job boards of GigaOM and other technology news portals like TechCrunch, VentureBeat, and ReadWriteWeb. Miller also recommends attending some of the industry's big events, such as GigaOM's Structure:Data and publisher O'Reilly Media's Strata Conference, both held annually in the United States.

But the biggest opportunities in the field may be yet to come. "As more and more people expand their online presence, whether through social networking, mobile apps, et cetera, the data sets are going to continue to become larger and larger," says Joseph Misiti, chief technology officer and co-founder of Social Q, a New York City–based start-up exploiting social media for market research. "New datasets will evolve and businesses (and jobs) will form with the sole purpose of providing sophisticated analytics on top of these datasets," he writes in an e-mail.

For scientists with a technical background, business intelligence is one more industry career field they can enter. "Given the current economy and challenges that organizations are facing, they are in desperate need to evolve their business thinking around the use of analytics," IBM's Gold says. An attractive aspect of a BI job is being able to see the outcome of decisions made by businesses, Sly adds. "People get an opportunity to see the success of their work."

Sabine Louët is a freelance writer based in Dublin.