© 2009 ness technologies – proprietary and confidential author : omer yfhar documentum...
Post on 20-Dec-2015
214 views
TRANSCRIPT
©2009 Ness Technologies – Proprietary and Confidential
Author : Omer yfhar
Documentum infrastructure 2009
FAST
בס"ד
processes ""Save-to-Searchable
The roll of index partition- Partition overview
table mechanism dmi_queue_item
process Index agent
What is fulltext query
Troubleshooting , administering and DRP
advanced possibilities of spreading
3
תפריט ראשי
Properties or content of doc is changed
Event is made in dmi_queue_item
Index agent pull events records from dmi_queue_item
FAST index the object
The object is searchable
4
process"Save-to-Searchable"
www.ness.com
How does it works?“Save-to-Searchable”: a high-level overview
Index Agent
Object’s Content
Object’s meta-data
Object’s dmi_queue_item event
DocProcessor
(1) Save
Indexer
fixml
www.ness.com
How does it works? “Save-to-Searchable”: a high-level overview
Index Agent
Object’s Content
Object’s meta-data
Object’s dmi_queue_item event
DocProcessor
(1) Save
Indexer
fixml
index
(2) Index Agent queries dmi_queue_item
(3) meta-data pulled and formatted into “dftxml”
(4) Content is pulled by IndexAgent in Getfile and staged to a temp area
www.ness.com
How does it works? “Save-to-Searchable”: a high-level overview
Index Agent
Object’s Content
Object’s meta-data
Object’s dmi_queue_item event
DocProcessor
(1) Save
Indexer
fixml
index
(2) Index Agent queries dmi_queue_item
(3) meta-data pulled and formatted into “dftxml”
(4) Content is pulled by IndexAgent in Getfile and staged to a temp area
(5) “message-based” representation of dftxml sent to DocProcessor which will get content from “staged” area
www.ness.com
How does it works? “Save-to-Searchable”: a high-level overview
Index Agent
Object’s Content
Object’s meta-data
Object’s dmi_queue_item event
DocProcessor
(1) Save
Indexer
fixml
index
(2) Index Agent queries dmi_queue_item
(3) meta-data pulled and formatted into “dftxml”
(4) Content is pulled by IndexAgent in Getfile and staged to a temp area
(5) “message-based” representation of dftxml sent to DocProcessor which will get content from “staged” area
(6) Doc Processor writes out “fixml” file to disk & deletes “staged” data
www.ness.com
(6) Doc Processor writes out “fixml” file to disk & deletes “staged” data
How does it works? “Save-to-Searchable”: a high-level overview
Index Agent
Object’s Content
Object’s meta-data
Object’s dmi_queue_item event
DocProcessor
(1) Save
Indexer
fixml
index
(2) Index Agent queries dmi_queue_item
(3) meta-data pulled and formatted into “dftxml”
(4) Content is pulled by IndexAgent in Getfile and staged to a temp area
(5) “message-based” representation of dftxml sent to DocProcessor which will get content from “staged” area
(7) Indexer picks up fixml and writes status about it and adds / updates the index
www.ness.com
(6) Doc Processor writes out “fixml” file to disk & deletes “staged” data
How does it works? “Save-to-Searchable”: a high-level overview
Index Agent
Object’s Content
Object’s meta-data
Object’s dmi_queue_item event
DocProcessor
(1) Save
Indexer
fixml
index
(2) Index Agent queries dmi_queue_item
(3) meta-data pulled and formatted into “dftxml”
(4) Content is pulled by IndexAgent in Getfile and staged to a temp area
(5) “message-based” representation of dftxml sent to DocProcessor which will get content from “staged” area
(7) Indexer picks up fixml and writes status about it and adds / updates the index
Once the data is in the index it is now “searchable” !
www.ness.com
How does it works? Partition overview
in order to hurry the process of index there are three index partitions (by default)
The new index docs are going to the first and small partition. After some time they will merge to the last and gig partition
This kind of mechanism allow to search docs that are gust imported to documentum with no time
www.ness.com
How does it works? Partition overview
Additionally every index partition have an offline and online copyThe index process is made against the offline partition and the search against the online. After the index is finished the offline will merge to the online partition.enables search on 100% from the existing documents without a disturbance of the action of index
online copy
offline copy
12
All the event of the types which register for index are sent to this table
- from DA it's possible to define kinds of specific documents will sent to index
- the default events are:
dm_save\dm_destroy\dm_checkin\dm_readonlysave\dm_movecontent
- to un register event (API):
unregister,c,<r_object_id_of_dm_sysobject_type>,<event>,dm_fulltext_index_user
- maintenance is made by the dm_QueueMgt administrative method
13
Dmi_queue_item
Pull the records from dmi_queue_item table
- logs are saved in %documentum\log
- demands a management of a JAVA memory
JvmMs 1024
JvmMX 1024
JvmSs 128
- it's possible to stop from DA or API
API command:
apply,c,null,FTINDEX_AGENT_ADMIN,NAME,S,<full text index name>,AGENT_INSTANCE_NAME,S,<agent instance name>,ACTION,S,shutdown (or start)
14
Index Agent
Fulltext query
-the query is accomplished to against FAST only
-select list values & security are pulled from database
-search & where are converted to FullText query
15
Query process standard & database query VS fulltext
query
document contain + whereFulltext
RDBMS
Select + document contain + where
Result
Select value & security
Standard query
- The where & select will generate against the RDBMS
- A document contain query in addition enable(NOFTDQL)
- More standard query options:
- A enable(TRY_FTDQL_FIRST) hint can be add to search document contain
(when FAST will reach his timeout or wildcard search limits)
- There is option to configure inside the application that for a specific where clause it will add the enable(NOFTDQL) hint
16
Query process standard & database query VS fulltext query
Converting a database query to a fulltext query
- Generate a database query against the FAST ant not against the RDBMS
- At the end of query add the enable(FTDQL) hint
there are some restrictions depending on the docbase version (can not sum, group by …)
17
Query process standard & database query VS fulltext
query
why should I to do such a thing???
Query process standard & database query VS fulltext
query
that this can improve the time of the answer
in dozens of percents
Reduce I\O demand
Reduce the errors in the dmi_queue_item table
Improving Save To Searchable time
improving a user's experience for search of documents
Nctrl command
Stopping the Fast machine
Data verification examination in the index by FtIntegrity Tool
Disaster recovery
advanced possibilities of deployment
19
Troubleshooting and administering
Reduce i\o demand
- Disable the auto FIXML defragmentation and schedule it to weekend
- Splitting the big partition into two partition (if there are more then 10 M docs)
Reduce timeout \ merge \ duplicate errors in the dmi_queue_item
- JAVA memory management of Index Agent
- Increase timeout value of Index Agent
- Reduce i\o demand
Adding more Doc Processor engine
- Reduce the time of index
- A large CPU & RAM consumer
20
Troubleshooting and administering
Troubleshooting and administering
Improvement save to searchable
Disable the auto defragmentation & split the large
partition
Adding a doc processor
JAVA memory management of
Index Agent
Improving user experience for search
- Reduce i\o demand
- Increase wildcard + timeout restriction for search (only for strong hosts machine)
- Think when to use \ convert fulltext query or standard query
22
Troubleshooting and administering
Nctrl command
- A CMD interface to administrate the Fast. Can be found in:
%documentum\fulltext\IndexServer\bin
- nctrl sysstatus > show the server process status
- nctrl stop > stop the Fast services completely
nctrl stop indexer > stop the index engine (search is possible)
nctrl stop qrserver search-1 > stop the search engine
nctrl add procserver > add another index engine
Web management console
- http://localhost:16000 > matching engine > control panel
23
Troubleshooting and administering
Troubleshooting and administering
WEB interfacea present situation, logsAnd advanced management possibilities
Troubleshooting and administering
An indicate that the index system is ready after initialize fnet: engine up
Troubleshooting and administering
An indicate that the partitions are ready after initialize
Shutting down the FAST machine
- To prevent a data corruption, make sure all index partition are in Idle state
- The best way to achieve this is to stop the Index Agent few minute before
- From CMD run the nctrl stop command. This will stop the Fast & service in the right order
- The services are:
fdispatch.exe fsearchctrl.exe
fidxjob.exe fsearchctrl.exe
findex.exe search.exe
frtsobj.exe
27
Troubleshooting and administering
Troubleshooting and administering
What can I do in system update & any other tasks that require me to shutdown the server?
Schedule a task which:1. Iapi command to stop Index Agent2. Sleep command for 15-30 minute3. Nctrl stop command to stop the Fast
Disaster recovery
- Backup of the Fixml files (%documentum\data\fulltext\fixml\set_*) only set_
- require stopping the index engine (nctrl stop indexer)
- The best way to achieve it is by script
- Reindex from the Fixml files
- Very simple and easy
29
Troubleshooting and administering
30
Troubleshooting and administering
Restoring Fast from Fixml files
1. Stop Index Agent2. Nctrl stop to stop FAST3. If needed, delete %Documentum\data\fulltext\fixml4. Restore %Documentum\data\fulltext\fixml5. Delete all inside %documentum\data\fulltext\data6. Start index server (service)7. After start8. Start cmd in d:\documentum\fuultext\indexserver\bin9. Type: rtsadmin <hostname> 16099 webcluster 0 0
resetindex10.When all done, start index agent
Round Robin (basic) distribution
- Parallel index and search mechanism
- Distribute the index & search equally across all the nodes
- All the node members must be online
- Adding or removing of a node require to reindex (rebalance the FIXML)
31
advanced possibilities of departure multinode
Direct Routing distribution
- Parallel index and search mechanism
- Each node is responsible for index specific filestore\s
- good idea if you designate a filestore per type or archive
- Nodes can be add, dropped, or stopped without interfering the other nodes
32
advanced possibilities of departure multimode
Two separated, independent index systems
- To separate Index Agents
- For each FAST(index agent) a unique event is made in the dmi_queue_item table
- Can be installed with
multimode configuration
33
advanced possibilities of departure high availability
34
High availability – multimode direct routing configuration
Dmi_queue_item
Index Agent_1 Index Agent_2
Thanks
all taken from EMC solutions
35