8/4/2019 p128-ceri_1982

1/9

HORIZONTAL DATA PARTITIONING IN DATABASE DESIGN

S.Ceri - M.Negri - G.PeLagatti

Istituto di Elettrotecnica ed ElettronicaPolitecnico di Milan0P.zza Leonardo da Vinci,32Mi lano, Italy

AbstractIn this paper the problem of horizontallypartitioning data on a set of resources isconsidered.The main optimization parameter is thenumber of accesses performed by theapplication programs to different portionsof data. The concepts which are requiredfor the determination of relevant portionsof data are defined and a methodology fordetermining the access parameters isproposed.The formulation of the generalpartitioning problem in 3 specificapplication enviroments is shown:distributed database design, filepartitioning on a primary and a secondarymemory and dlta distribution on differentdevices.It is shown that the analytical models ofthese specific app lications use theparameters of the general formulation .

1. Introduct ionThe problem of optimally partitioning dataover a set of facilities is faced inseveral cases in information system design/1,21. Consider for example the design ofPermiss ion to copy without fee all or part of this material is grantedprovided that the copies are not made or distributed for directcommercial advantage, the ACM copyright notice and the title of thepublication and its date appear, and notice is given that copying is bypermission of the Association for Comput ing Machinery. To copyothetise, or to republish , requires a fee and/or specific permission.

@ 1982 ACM O-89791-073-7/82/006/0128 $00.75

a distributed information system /5-9/, orthe allocation of data on differentphysical areas or devices /IO-13/, or thepartitioning of a file into a primary anda secondary memory /14-151: in all thesecases, data must be partitionea intosubsets and the subsets are associateawith the nodes, physical areas, devices orfiles respectively. Despite the seeminglyvery different application environments,all these cases share a common problemconsisting in the determination ofrelevant portions of data and of how theseportions are accessed by the applicationsrunning in the system. In all theseapplication environments, the use of aresource depends on the data which aremanaged by the resource; therefore thedetermination of the access pattern of theapplications is requi rea. The importanceof this problem will encrease with thecurrent technological trend touardsdistributed systems.The determination of the access pattern ofthe applications to different portions ofdata is not an easy task, because itrequires to understand both the loyicaland the statistical properties of the dataand of the app lications; this problem istherefore interdisciplinary in nature.Curiously, not much research has beendevoted to this problem. For instance,several works exist on file allocation indistributed databases 13-91, but few ofthem deal with file fragmentation /9,10/,and none of them with the characterization

128

8/4/2019 p128-ceri_1982

2/9

of access patterns to possible fragments.In this paper, the set of data to bepartitioned is considered to be a fileconsisting of homogeneous records. Thebasic assumption of this paper is that theprobabi lity that a record is accessed in agiven time period depends on some propertyof the record itself and therefore all therecords, which have the same propertieshave also the same probability to beaccessed. The determination of whichproperties of the records are relevantwith respect to its probability to beaccessed is therefore an important aspectof the data partitioning problem.Given the above assumptio n, section 2 ofthis paper defines the logical andstatistical concepts which are needed inorder to characterize the access patternof transactions to data; section 3 showshow these concepts can be applied from amethodological viewpoint and presents aresult which allows to reduce the numberof required parameters; section 4 showshow the parameters which characterize theaccess pattern to the records are used inthe formulation of the optimalpartitioning problem for a set of designeniironments.

2. Basic definitions

2.1 PredicatesA file F is a set of homogene ous records;records are subdivide d into attributes . Apredicate xi is a boolean function definedon F; each predicate can be evaluated oneach record. The subset of the records ofF for which xi is true is represented by. Therefore:

8/4/2019 p128-ceri_1982

3/9

natural form in one of them, and negatedin the other one), having a differentratio between the accesses to the mintermfragment and its cardinality; therefore:

xk is relevant < -->3Yi,YjEY(X) Ia(yi)/n(yi) # a(yj)/n(yj)A(yiVxk q yjV-xk)

A complete set X of predicates is minimalif and only if all its elements arerelevant.Let X be a minimal complete set ofpredicates, with m=IXI; any booleanexpression E of them is itself apredicate, and CB> represents the subsetof records for which B is true, called acandidate fragment of F. The number ofdifferent candidate fragments that can bedefined is 2m (two fragments are differentif they are different sets; equiva 1boolean expressions represent there fthe same fragment).The general problem of characterizingdishomogeneity of accesses of transact i

entore

theons

to the records of a file is now properlyformulated as the problem of determiningthe accesses of transactions to any of itscandidate fragments. This problem isequivalent to the determination ofaccesses to minterm fragments, because,once that accesses to each mintermfragment are knoun, it is possible todetermine the accesses to any candidatefragment by expressing the correspondingboolean expression as the disjunctio n ofminterm predicates, and then summing theaccesses which are made to thecorresponding minterm fragments.

2.3 Implications bet ween predicatesImplications betueen predicates arerelevant to the determination of theaccesses to candidate fragments.Implications exist not only between simplepredicates but also between generalboolean expressions of simple predicates;for instance:DEPT='dl'h (STATUS='mgr'VQUAL='Progr') ->

-> JOB='instructor'ASALARY > 'SU.UUU'If- an implication exists between twoexpressions B k 1and i3 , then they have torespect the following obvious rules on thenumber of accesses to the corresponainycandidate fragments:(Bk -> l3l) -> a(Bkh B1) = a(Bk)

a(BkV BL) = a(t3')The existence of an implication betweentwo expressions has the consequence thatsome minterm fragments can ue void. Letmint(Bk) represent the set of mintermpredicates which imply d . . :ie

mint(Ek) = C yi I yi -> tlk).Then, given an implication I: B k -> ul,all the minimal fragments expressed by theminterms belonging to the set:

V(I) = mint(Bk)Amint(~tl L)are void. In fact, let yicV(l) be such dminterm. Suppose that a record 0 6 iYi)exists, then the expression tithe expression B1 is false fo: ~~e'r~:o:~~thus contraddictiny the implication.Thus, the implications are usea fordetermining and eliminatiny from theanalysis those minterm frayments which arevoid.

2.4 Access probabilitiesThe statistical behaviour of the accessesto the records of F is described referringto the model of figure 1. The file F isrepresented as a set , uhose elements arethe records of F; the set is partitioneainto disjoint minterm frayments, asresulting from the definition of acomplete and minimal set of s i m p I epredicates, and the analysis of all theimplications between them.A second set, called request set K,represents all the accesses to records ofF made by application proyrams in a yivenperiod of time.Accesses in the request set are mapped tothe corresponding records of the file F bya function f : R -> F; in fact, eacnaccess in the set R refers to a particularrecord of F, while the same record of t

130

8/4/2019 p128-ceri_1982

4/9

8/4/2019 p128-ceri_1982

5/9

data set.This assumption requires the conservation,in the request set, of proportionalitiesexisting between subsets of records in thedata set. Let us take the predicate x, asa reference, and consider the subdivisionof the subset of accesses to records forwhich x , holds into two further subsets inwhich the minterm predicates: x hx1 2'x, A -x2 hold respectively. Then, thisassumption requires that these two subsetsare in the same proportion as thecorresponding minterm fragments in thedata set. The assumption is formulated asfoilows:p(x,Ax2(f(r))) / p(x,A-x2(f(r))) qP(X,AX2(0)) / p(x,"-x2(o))BY applying simple algebrictransformations, we obtain:p(x,"x2(f(r)))=p(x,(f(r))) p(x2(o)/x,(o))==(a(x,,)/A)(n(x,Ax2)l n(x,)),where p(x/y) is the well-known expressionof the probability of event x conditionedby the event y.Note that if the symmetrical assumptionholds taking the predicate x2 as areference, then the following facts alsohold:I) the ratios between accesses to mintermfragments and their cardinality are equal,i.e.p(yi(f(r))) / p(yj(f(r))) == p(q(o)) I P(Yj(0))2) neither x1 nor x2 are relevantaccording to the definition in section 2,and in fact accesses are homogeneouslydistributed in the data set.A graphical representation of the twoassumptions is shown in figure 2.

3. Methodo logical problemsThe concepts which have been defined inthe previous sections suggest to performthe following steps when determining theaccess pattern of applications to data:I) Classification of transactions.2) Specification of a complete and

(possibly) minimal set of predicates.

3) Determination of implications betueenpredicates.

4) Analysis of statistical assumptions.5) Determination of accesses to minterm

fragments.6) Integration of access requirements of

different classes.The major problems which arise inperforming the above steps are analyses in1211; this paper adresses only a few ofthem.In performing the desiyn,care must betaken of the fact that relevant predicatescannot always be deduced from the text oftransactions.Consider for example a bank transactionconsisting in the request of the currentamount of the bank account of a customer.The transaction needs for theidentification of the customer thespecification of either (a) his name, or(b) his office number and account number.Assume that the bank has a distributeddata base system, and that each office ofthe bank can issue the transaction, with adifferent frequency. In order to describeadequately the access properties oftransaction activations from a particularoffice, say office 1, one of the relevantpredicates is clearly:

x : bank-office-numb er = 1In fact, while each customer can use anyoffice for making the transaction, theprobability that at office 1 thetransaction is issued by a customer whosebank account number is also 1 is very-high, as each customer uses his officemore often than any other office. In case(a) the predicate cannot be deauced by thetext of the transaction, while case (b)uses the predicate explicitely as aselection criteria. In case (a), thedesigner should investigate on the generalfeatures of the transaction andcharacterize the common properties of theconsidered transaction activations, inorder to determine the appropriatepredicate.

132

8/4/2019 p128-ceri_1982

6/9

Referring to step 5 of the design, themost obvious algorithm for determinin g theaccesses to minterm predicates is thefollowin g: Let Xk be a set of k simplepredicates, Y(Xk) be the associated set ofminterms predicates, as defined in section2.2. Let xk+, be a simple predicate whichdoes not belong to Xk, and letxk+,xk x k+l' The accesses to mintermfragments 2 3of all the sets X , X I...., Xncan be progressively computed using theassumptions introduced in section 2.5; ifno property applies,the number of accessesto the new minterm fragment 'YiAxk+l'being considered is explicitelydetermined. This algorithm allows todetermine the accesses to minterms, butits application is rather he avy; in orderto complete, it is necessery to analyse3- 2n- 1 intermediate minterms. Thecomputation of this number is given in1211. It is possible, however, to follow adifferent algorithm, which consists inanalysing for each set X k only the mintermx Ax h....Ax

' k2 k instead of the complete setY(X ); the determination of accesses tothe other minterm fragments is given thenby the solution of a linear system ofequations. This algorithm requires theanalysis of statistical assumptions onlyfor n2 - n - 1 pairs, and the solution ofa system of n2 equations, in the n2variables corresponding to accesses tominterms. The equations are of the form:a(Bk) = > a(yi)

yibmint(Bk)where a(yi) are unknown, and atBk) areextimated; in facts, there are:(I) the general equation expressing that

the accesses to all minterms are A(2) n equations expressing the accesses to

simple predicates(3) 2 - n - 1 equations, each expressingthe accesses to the conjunc tions ofpredicates in the natural form; eachequation is produced by one of theabove pairs.

In /21/ the proof is given that this

2algorithm requires to analyze n -n-l pairsof predicates and the associated equationsare linearly independent.During step 6 of the desiyn, in order tointegrate the access requirements otdifferent classes, all the simplepredicates produced for each class aremerged into one set S. The global mintermfragments wj that are generated using thesimple predicates of S are a finerdecomposition of F than those determineuby the minimal fragments of each sinyleclass; the accesses to a minterm yi of agiven class are homogeneously distributeaon the global minterms w. such that3W. --> Yi ; therefore:I n(wj)

a(wj)=a(yi)-------n(yi)

4. Formulation of optimal partitioningproblems for a set of design environments .In this section, the parameters whichcharacterize the access pa ttern oftransactions are used in the formulationof the optimal partitioning problem torthe following design environments: optimalfragmentation and allocation ot adistributed database, horizontalpartitionin g of a file into a primary anua secondary memory, allocation of a fileon several devices for assigniny a yivendistributio n of work load. The main yoalof this section is not to proviae anadiscuss complex optimization models, outto show that the problem formulations forthese applications require exactly thoseconcepts and parameters which have beenintroduced in the previous sections. Theemphasis is therefore on the problemformulation, not on its solution.4.1 Optimal fragmentation and allocationof a distributed database.Recent researches on distribute d databases/17-19/ have shown that it is generallyconvenient to partition a file into

133

8/4/2019 p128-ceri_1982

7/9

fragments, which are the smallest units ofallocation. Clearly, it is not possible tosolve the fragmentation problemindependently before the allocationproblem, because the optimal fragmentationcan be determined only with respect to theoptimal allocation of fragments.The mixed fragmentation and allocationproblem is treated using as goal function(to be minimized) the number of remoteaccesses of transactions to the records ofthe file; the criterion is sufficientlyhigh level and general, does not requirethe extimation of many tedious parametersfor modelling the distributed resources,and seems to be sufficiently effective inreal cases.The global minterm fragments introduced insection 3 are the proper units ofallocation, since they contain recordswhich are homogeneous ly accessed by alltransactions; transactions must be dividedinto classes according to their activationnodes. Let a(wi/nj) be the number ofaccesses to the global minterm fragment wifrom the transactions which are activatedat node n.,3 and let rij and uij representthe fractions of these accesses which areretrievals and updates respectively, withr.. t u.. = 1.3 3Two distinct allocation problems areconsidered: the non-redundant problem, inwhich exactly one copy of each mintermfragment is allocated, and the redundantproblem, in which at least one copy ofeach minterm fragment is allocated. Inboth cases, the final partitioning of thefile into fragments is obtained bycollecting in the same fragment all thoseminterm fragments whose resultingallocation is on the same node.In the non-redundant problem, theminimization of the number of remoteaccesses of the transaction to the recordsof the file requires that each record beallocated on the node where it is accessedmore often; therefore:

8/4/2019 p128-ceri_1982

8/9

maximizing the total value of thecontained goods; clearly, the maximumweight corresponds to the capacity C ofthe fast memory, and the value ismaximized when most of the accesses are torecords allocated in the fast memory.Again, global minterms introduced insection 3 are the proper units ofallocation; the value vj associated toeach minterm w. isI given by the ratioaCwj)/nCwj). The problem has a linearinteger formulation by associating to eachglobal minterm a binary variable zj:max ) zj vj

l all j 1under the constraint:> zj n(wj)

8/4/2019 p128-ceri_1982

9/9

Sec. 5, Florida, 1980. for Hierarchical Structures", ACM - 3 ,vol. 2, no. 1, March 1977.

3. W. W. Chu: "Optimal File Allocation ina Multiple Computer System", IEEE - T&vol. C - 18, no. IO, 1969.

4. K. P. Eswaran: "Placement of Records ina File and File Allocation in a ComputerNetwork", Proc. IFIP Congress , North--Holland, 1974.

5. s. Mahmoud, J. S. Riordon: OptimalAllocation of Resources in DistributedInformation Networks", ACM - TODS , vol.1, no. 1, 1976.

6. H. L. Morgan, J. D. Levin: "OptimalProgram and Data Location in ComputerNetworks", CACM , vol. 20, no. 5, 1977.7. P. P. s. Chen, J. Akoka: "OptimalDesign of Distributed InformationSystems", IEEE - x, vol. SE-6, no. 12,Dec. 1980.

a. s. Ceri, G. Martella, G. Pelagatti:"Optimal File Allocation on a Network ofMinicomputers", Proc. Int. Conf. on- - - -Databases , Heyden Pub., Aberdeen, July1980.

9. S. Ceri, S. B. Navathe, G. Wiederhold :"Optimal Design of Distributed Databases",Working paper, Stanford University, 1981(Submitted for publicatio n).

10. J. A. Hoffer: "An Integer ProgrammingFormulation of Computer Database DesignProblems", Information Science , vol. 11,July 1976.

Il. J. A. Hoffer, D. G. Severance: "TheUse of Cluster Analysis in PhysicalDatabase Design", Proc. First Int. Conf.- --on Very Large Data Bases---- , Framingham,1975.

13. S. T. March, D. b. Severance: "TheDetermination of Efficient KecoruSegmentation and Blockiny Factors forShared Data Files", ACM - 3 , vol. L,no. 3, Sept. 1977.

14. M. J. Eisner, D. ~1. Severance:"Mathematical Techniques for EfficientRecord Segmentation in Larye ShdreUDatabases", JACM , vol. 23, no. 4, Octooer1976.

15. M. Hammer, B. Niamir: "A HeuristicApproach to Attribute Partitioniny", rroc.ACM - SItiMOD , 1979.

16. S. K. Chany, W. H. Chenw: "AMethodology for Structured DatabaseDecomposition", IEEE -,TSE vol. St-6,no. 2, Mar. 1980.17. J. B. Rothnie et al: "Introduction toa System for Distributed Databases tsuu-1)ACM - TODS , vol. 5 no. 1, Mar. lY8U.

18. B. G. Lindsay, P. ti. SeLinger et al:Notes on Distributed Databases", HJL571(33471), IBM Res. Lab., San Jose, Jul.1979.

19. S. Ceri, G.Pelagatti: "The Allocationof Operations in Distriouted DatauaseAccess", IEEE - E , Feb 1982.

20. D. J. Luenberger: "Introduction toLinear and Nonlinear Programminy",Addison-Wesley, 1973.

21. s. Ceri, M. Neyri, b. Pelayatti:"Prob terns in Horizontal aatapartitioning", Internal Report, AEEtrlvn.82-5, Apri 1 1982.

12. M. Schkolnick: "A Clustering Algorithm

136