© 2009 ness technologies – proprietary and confidential author : omer yfhar documentum...

Author : Omer yfhar

Documentum infrastructure 2009

בס"ד

www.ness.comOmer.yfhar@ness.com

Search by text

Improve regular query

מה אפשר לעשות איתו

processes ""Save-to-Searchable

The roll of index partition- Partition overview

table mechanism dmi_queue_item

process Index agent

What is fulltext query

Troubleshooting , administering and DRP

advanced possibilities of spreading

תפריט ראשי

Properties or content of doc is changed

Event is made in dmi_queue_item

Index agent pull events records from dmi_queue_item

FAST index the object

The object is searchable

process"Save-to-Searchable"

www.ness.com

How does it works?“Save-to-Searchable”: a high-level overview

Index Agent

Object’s Content

Object’s meta-data

Object’s dmi_queue_item event

DocProcessor

(1) Save

Indexer

index5Omer.yfhar@ness.com

www.ness.com

How does it works? “Save-to-Searchable”: a high-level overview

Index Agent

Object’s Content

DocProcessor

(1) Save

Indexer

(2) Index Agent queries dmi_queue_item

(3) meta-data pulled and formatted into “dftxml”

(4) Content is pulled by IndexAgent in Getfile and staged to a temp area

6Omer.yfhar@ness.com

www.ness.com

Index Agent

Object’s Content

DocProcessor

(1) Save

Indexer

(5) “message-based” representation of dftxml sent to DocProcessor which will get content from “staged” area

www.ness.com

Index Agent

Object’s Content

DocProcessor

(1) Save

Indexer

(6) Doc Processor writes out “fixml” file to disk & deletes “staged” data

www.ness.com

Index Agent

Object’s Content

DocProcessor

(1) Save

Indexer

(7) Indexer picks up fixml and writes status about it and adds / updates the index

www.ness.com

Index Agent

Object’s Content

DocProcessor

(1) Save

Indexer

(7) Indexer picks up fixml and writes status about it and adds / updates the index

Once the data is in the index it is now “searchable” !

www.ness.com

How does it works? Partition overview

in order to hurry the process of index there are three index partitions (by default)

The new index docs are going to the first and small partition. After some time they will merge to the last and gig partition

This kind of mechanism allow to search docs that are gust imported to documentum with no time

Omer.yfhar@ness.com 11

www.ness.com

How does it works? Partition overview

Omer.yfhar@ness.com

Additionally every index partition have an offline and online copyThe index process is made against the offline partition and the search against the online. After the index is finished the offline will merge to the online partition.enables search on 100% from the existing documents without a disturbance of the action of index

online copy

offline copy

All the event of the types which register for index are sent to this table

- from DA it's possible to define kinds of specific documents will sent to index

- the default events are:

dm_save\dm_destroy\dm_checkin\dm_readonlysave\dm_movecontent

- to un register event (API):

unregister,c,<r_object_id_of_dm_sysobject_type>,<event>,dm_fulltext_index_user

- maintenance is made by the dm_QueueMgt administrative method

Dmi_queue_item

Pull the records from dmi_queue_item table

- logs are saved in %documentum\log

- demands a management of a JAVA memory

JvmMs 1024

JvmMX 1024

JvmSs 128

- it's possible to stop from DA or API

API command:

apply,c,null,FTINDEX_AGENT_ADMIN,NAME,S,<full text index name>,AGENT_INSTANCE_NAME,S,<agent instance name>,ACTION,S,shutdown (or start)

Index Agent

Fulltext query

-the query is accomplished to against FAST only

-select list values & security are pulled from database

-search & where are converted to FullText query

Query process standard & database query VS fulltext

document contain + whereFulltext

Select + document contain + where

Result

Select value & security

Standard query

- The where & select will generate against the RDBMS

- A document contain query in addition enable(NOFTDQL)

- More standard query options:

- A enable(TRY_FTDQL_FIRST) hint can be add to search document contain

(when FAST will reach his timeout or wildcard search limits)

- There is option to configure inside the application that for a specific where clause it will add the enable(NOFTDQL) hint

Query process standard & database query VS fulltext query

Converting a database query to a fulltext query

- Generate a database query against the FAST ant not against the RDBMS

- At the end of query add the enable(FTDQL) hint

there are some restrictions depending on the docbase version (can not sum, group by …)

why should I to do such a thing???

www.ness.comOmer.yfhar@ness.com 18

that this can improve the time of the answer

in dozens of percents

Reduce I\O demand

Reduce the errors in the dmi_queue_item table

Improving Save To Searchable time

improving a user's experience for search of documents

Nctrl command

Stopping the Fast machine

Data verification examination in the index by FtIntegrity Tool

Disaster recovery

advanced possibilities of deployment

Troubleshooting and administering

Reduce i\o demand

- Disable the auto FIXML defragmentation and schedule it to weekend

- Splitting the big partition into two partition (if there are more then 10 M docs)

Reduce timeout \ merge \ duplicate errors in the dmi_queue_item

- JAVA memory management of Index Agent

- Increase timeout value of Index Agent

- Reduce i\o demand

Adding more Doc Processor engine

- Reduce the time of index

- A large CPU & RAM consumer

Improvement save to searchable

Disable the auto defragmentation & split the large

partition

Adding a doc processor

JAVA memory management of

Index Agent

Improving user experience for search

- Reduce i\o demand

- Increase wildcard + timeout restriction for search (only for strong hosts machine)

- Think when to use \ convert fulltext query or standard query

Nctrl command

- A CMD interface to administrate the Fast. Can be found in:

%documentum\fulltext\IndexServer\bin

- nctrl sysstatus > show the server process status

- nctrl stop > stop the Fast services completely

nctrl stop indexer > stop the index engine (search is possible)

nctrl stop qrserver search-1 > stop the search engine

nctrl add procserver > add another index engine

Web management console

- http://localhost:16000 > matching engine > control panel

WEB interfacea present situation, logsAnd advanced management possibilities

An indicate that the index system is ready after initialize fnet: engine up

An indicate that the partitions are ready after initialize

Shutting down the FAST machine

- To prevent a data corruption, make sure all index partition are in Idle state

- The best way to achieve this is to stop the Index Agent few minute before

- From CMD run the nctrl stop command. This will stop the Fast & service in the right order

- The services are:

fdispatch.exe fsearchctrl.exe

fidxjob.exe fsearchctrl.exe

findex.exe search.exe

frtsobj.exe

What can I do in system update & any other tasks that require me to shutdown the server?

Schedule a task which:1. Iapi command to stop Index Agent2. Sleep command for 15-30 minute3. Nctrl stop command to stop the Fast

Disaster recovery

- Backup of the Fixml files (%documentum\data\fulltext\fixml\set_*) only set_

- require stopping the index engine (nctrl stop indexer)

- The best way to achieve it is by script

- Reindex from the Fixml files

- Very simple and easy

Restoring Fast from Fixml files

1. Stop Index Agent2. Nctrl stop to stop FAST3. If needed, delete %Documentum\data\fulltext\fixml4. Restore %Documentum\data\fulltext\fixml5. Delete all inside %documentum\data\fulltext\data6. Start index server (service)7. After start8. Start cmd in d:\documentum\fuultext\indexserver\bin9. Type: rtsadmin <hostname> 16099 webcluster 0 0

resetindex10.When all done, start index agent

Round Robin (basic) distribution

- Parallel index and search mechanism

- Distribute the index & search equally across all the nodes

- All the node members must be online

- Adding or removing of a node require to reindex (rebalance the FIXML)

advanced possibilities of departure multinode

Direct Routing distribution

- Parallel index and search mechanism

- Each node is responsible for index specific filestore\s

- good idea if you designate a filestore per type or archive

- Nodes can be add, dropped, or stopped without interfering the other nodes

advanced possibilities of departure multimode

Two separated, independent index systems

- To separate Index Agents

- For each FAST(index agent) a unique event is made in the dmi_queue_item table

- Can be installed with

multimode configuration

advanced possibilities of departure high availability

High availability – multimode direct routing configuration

Dmi_queue_item

Index Agent_1 Index Agent_2

Thanks

all taken from EMC solutions

© 2009 ness technologies – proprietary and confidential author : omer yfhar documentum...

Documents

emc documentum contentservicesforsharepoint...emc®...

vocea pumnalului. trilogia pe taramul haosului - patrick...

documentum e integración continua

alex w. ness - transferencia de espÃritus - alex w. ness[1]

bren ness el 0001

בס"ד משימות הערכה בתושב"ע

penrose consciou ness

ness | portfolio abril

hgu ness 2009 paper

integració sap - documentum

webdav documentum 6.5

documentum portlets 사용설명서 · 목차 1장...

ness report

embedded ness

130 ness ziona

mad ness case

1 מבוא למערכות מידע סמסטר ב', תש"ע...

cap13 van ness

ness Échangeur thermique

catálogo ness'12