Digital Landfill

Here's a SlideShare version -- 

And here's a YouTube version -- 

With the explosion of content in the enterprise, findability has become a major concern for most organizations. Tens of millions of documents can be scattered between multiple file systems, databases and content management applications. Each application has a different access method and no common interface exists for a user who needs a piece of knowledge to get their job done. Putting all of this content in a common location and providing an interface to find this data is the job of enterprise search.

[Note: Are you a Service Company or VAR in the Document Management Space? If so, have you made your plans to attend the Document Management Service Providers Executive Forum November 4-6 in Nashville? Make your plans NOW; seats are limited.]

8 Keys to Findability

1 -- Data in Many Types of Files.

Much of the content in organizations has grown over time through multiple different systems. Every merger or acquisition introduces additional types of files and storage locations. Many of the applications to open this data are proprietary and may not be available to all users. Data, however, resides in all of these different file types and must be accessible. The ability to identify relevant files and provide a peek into the content regardless of format removes a burden from users that often leaves data unused.

2 -- Data Locked in Systems.

As organizations grow, so do the number of applications that they support. Each of these applications, whether purchased or developed internally, stores corporate data and - therefore - knowledge. This is often a fishing expedition, requiring knowledge of many different systems and interfaces.

New users can take months or years to learn all of these applications and understand the data that they contain. While helping a client build a customer service system, I learned that their representatives had to open as many as 11 applications to answer a single customer question. Not only did this cause a dramatic increase in the time spent on the task, but often led to incorrect answers.

Providing a single interface into the data from many systems eliminates this problem. Even though the information is in a single application, it still needs to provide sufficient context and relevant details so the user can take action. This approach moves users from a simple search box to a search-based application and continues companies down the path of deriving value from enterprise search.

3 -- Not all Text is the Same.

Much of the functionality of search comes from the ability to handle unstructured data. An application that can help users find a subset of documents based on a keyword search can be valuable, but the ability of the search application to interpret the text and provide context around specific elements or entities provides the findability that produces even greater value.

The ability to find products, people, locations and many other entities allows for the documents to be meaningfully categorized. This categorization is different for each company and - sometimes - for each user. Consider the value of searching for a term and being presented with thousands of possible matches and then being able to further refine the search based on a product and customer list that is created specific to the current results.

4 -- Users Can’t Spel so Gud.

Somewhere between thought and my keyboard is a gremlin because what I think is not always what appears! This is certainly a common problem and one that must be accounted for in any search effort. Many times data in files is overlooked because a keyword is misspelled. The ability for a search engine to find near matches and common mistakes provides an option to expand a search and return documents that would otherwise get overlooked.

This capability is also useful in a search interface. Checking terms in a query against a spelling dictionary and a list of words common to the corpus and making suggestions provides another tool that improves the ability to quickly find the correct information.

5 -- Word Forms should not be Important.

Spelling is certainly helpful but lemmatization - the ability to group together the different inflected forms of a word so they can be analyzed as a single item - has the ability to transform a search experience. Wouldn’t it make sense that if a search is performed for “run,” the word “ran” may also be relevant? This is a simple example, but the ability to expand a search to include multiple forms of the requested terms means that users don’t have to spend time performing multiple searches or miss data that may be exactly what they need.

6 -- Relevancy is not the same for Everyone.

Turning search terms into relevant results ensures that users can find the documents they are seeking effectively. What is relevant to one company or one person may not be relevant to another. Tuning for relevance can therefore turn a mediocre search application into a fantastic one. Some applications are just fine with basic word matching. The more we understand about documents and their contents, however, the easier it is to adjust weighting. For example, the ability to recognize a document by type, author, creation date or contained concepts allows the application to place a higher weight on specific data fields and more accurately predict the results a user is seeking. A document on the first page of results should be very relevant to the entered search terms, but keep in mind the ability to refine the search based on domain specific data.

7 -- Security is Paramount.

We restrict access to documents in many different ways. Often the decision of authorization is left up to a specific application. Application programmers go to great lengths to ensure that data is only viewed by those who are authorized to do so. A viable search application therefore must be able to understand and respect the authorization rules given to the content that is indexed. This means that, even though a user finds a relevant match, if they have not been granted permission to the document, they should not be shown any part of the document or even be made aware that it exists.

8 -- Effective Search Experience.

All of the above points are great and will produce a fantastic index, but that index is irrelevant without an application that provides an intuitive, friendly experience. Users need to understand not only how to perform a search, but how to filter their results and how to take action to be productive with those results. The average number of search terms is about 1.7 words, which is not a lot when searching across millions of documents. Therefore, a conversation type of experience where users can get feedback from the results and refine their search makes for the most effective search results. This solves the findability problem, but the final step is allowing users to take action on a document through preview, common tasks, and other functionality provided directly by the application.

These eight items are critical to consider when implementing an enterprise search solution. Keep in mind however, that most often this needs to be an iterative process. The goal of findability requires regular reviews of user activity and tuning to ensure that the application is still effective. The core, however, is a good search engine and some domain knowledge about the content available to an enterprise. While these projects can be difficult, achieving findability for your knowledge store will make your users productive. And that is certain to make management happy.

---

Dan Gilroy is a Project Manager with RDA Corporation headquartered in Maryland. Dan is located in Charlotte, NC and has over 15 years of experience developing business solutions utilizing Microsoft technologies. Managing cross-functional project teams ranging in size from 5 to 30, Dan has a depth of management experience in a large variety of industries and corporate environments. He has leveraged a variety of methodologies and technologies to successfully and efficiently solve business problems for a long list of satisfied clients. Dan’s technical specialty is projects that leverage SharePoint and FAST to provide business value through collaboration.

Enterprise Content Management has become a critical strategic business process, due in part to increasingly stringent legislation and aggressive litigation. To meet this challenge, some organizations take a do-it-yourself approach, letting their in-house IT department build an ECM system from scratch or using ostensibly cheap tools. Inevitably, this only complicates matters and leads to greater expense with an inordinate amount of time spent on research and development, testing and debugging, and on-going support. When ECM is viewed as a technology system and managed as an IT project rather than as a management-led information governance initiative, the resulting implementation may lack the necessary security features and functionality to ensure regulatory compliance and discovery protection.

[Note: Are you a Service Company or VAR in the Document Management Space? If so, have you made your plans to attend the Document Management Service Providers Executive Forum November 4-6 in Nashville? Make your plans NOW; seats are limited.]

8 Things to Consider Before Developing an In-House ECM Solution

1 -- Beyond IT.

ECM implementations are often mistakenly viewed as simply Information Technology projects that will have little impact on the business until the finished application is installed and ready for use. Of course, IT does have a major role to play in any ECM deployment. Typical tasks include configuring application and database servers, setting up backup and recovery devices and schedules, integrating the application with user account services, installing client software on user’s workstations, training end users and administrators, and so forth.

This will inevitably consume a significant portion of the IT department’s time and may, for even limited deployments, require the sole focus of one or more IT team members for the duration of the project.

A successful ECM implementation satisfies the needs not just of Information Technology but of the business as a whole. To do this, it must include the active participation of other departments within the organization to address issues of planning, research, and business process analysis and alignment.

2 -- Research & Planning.

Before turning on a server or writing even a single line of code, a successful ECM development effort requires substantial and time-consuming research into applicable legal requirements, business processes, and technologies. Yet, some organizations focus only on the last—and least important—element, technology. This occurs when ECM is viewed and implemented as an IT project, rather than as a business strategy.

3 -- Time & Resources.

An informal survey of IT employees revealed that few if any were able to devote quality time to researching new technologies within the scope of their day-to-day activities of administering and maintaining existing systems or training and supporting users. For most, time for research and education had to be stolen from other activities in one or two minute increments, or conducted after hours on their own time. It is thus not surprising that many in-house ECM projects suffer from delays, scope creep, cost overruns, or even abandonment and complete failure.

A successful project must bring together multiple departments within the company, even those which may not yet be putting the ECM system into use. Including other departments in the planning and definition phase, including those not targeted for the initial rollout, makes it easier to extend the system to those departments later without additional development and customization.

4 -- Open Source vs. Commercial Applications.

Determining an organization’s license requirements for commercial applications, analyzing license options and negotiating volume purchases can be a complex and time-consuming task, such that larger corporations may employ full-time Contract Managers to navigate these agreements.

Some organizations attempt to side-step this process using free and open-source alternatives. While such options exist, for everything from the database and application server to the IDE used to write and debug code, this course is not without cost. Such a solution invariably takes longer to implement—and therefore costs more in the long run—than an off-the-shelf pre-developed and tested system.

5 -- Application & Database Licenses.

Some document management systems are designed to give users direct access to the underlying database used to store document metadata and therefore require a database access license for each user.

For example, an installation of Microsoft Office SharePoint Server 2007 (MOSS) requires a license for each SharePoint server, as well as client access licenses (CALs) for each device or user connected to the servers. Additionally, an equal number of server and user CALs must be purchased to allow those users to access the requisite Microsoft SQL Server. It should be noted that these MS SQL Server licenses and CALs cost almost as much as the MOSS licenses, immediately doubling the apparent license cost of a SharePoint system.

6 -- Testing & Debugging.

ECM systems created in-house tend to be in a constant state of flux, undergoing continual cycles of development and customization. The resulting testing and debugging may take as long as the development itself.

7 -- Training.

User acceptance is crucial to a successful ECM implementation. If users are not comfortable with the interface or don’t understand how to carry out their jobs, they will invariably find ways to circumvent the system or return to their old ways of working. To this end, sufficient training—for system administrators and for end users—is essential.
Unfortunately, ECM systems developed in-house tend to be works-in-progress, with long cycles of development, testing, and debugging. This means that end user training must likewise be ongoing, leading to reduced efficiency and increased frustration.

8 -- Ongoing Support.

It’s almost inevitable that an individual crucial to an in-house ECM development will eventually move on, either to another position in the same company, or to an outside firm. JDA Professional Services, an IT staffing agency, comments that “78 percent of all IT workers will leave their company before their fifth anniversary. IT workers with eight to ten years experience—the category of employee that previously was most likely to stay with a company through retirement—are now changing jobs more frequently than ever.”

When this happens, an organization may be left without adequate support for an application developed in-house. While knowledge transfer programs and succession planning can mitigate the risk, ongoing support for custom applications is an issue that should be considered before an in-house development project is allowed to grow into a mission-critical application like ECM.

---

Kevin D’Arcy is the Vice-President of Sales and Marketing for Hybrid Document Systems Inc. He frequently works with organizations to consult and recommend on document management initiatives across Canada. He is active in many industry groups, most notably AIIM and IIMDA, for whom he currently holds a position on the executive board. Kevin differentiates himself on developing unique solutions to document management challenges.