s03 hybrid app_and_gae_datastore_v1.0
DESCRIPTION
Google BigTable and GAE DatastoreTRANSCRIPT
![Page 1: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/1.jpg)
Smart TV
http://www.softwareinlife.com
Hybrid Appand
GAE DataStore
대표이사 장선진@jangsunjin
- Mobile Phone: (+82) 010-4585-1770- E-Mail: [email protected]
![Page 2: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/2.jpg)
MapReduce
• MapReduce is a patented software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers.
http://www.cs.berkeley.edu/~ballard/cs267.sp11/hw0/results/htmls/Muzaffar.html
![Page 3: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/3.jpg)
CAP and ACID• CAP
– 정합성 (Consistency). The client perceives that a set of opera-tions has occurred all at once.
– 가용성 (Availability). Every operation must terminate in an in-tended response.
– 단절내성 (Partition tolerance). Operations will complete, even if individual components are unavailable.
• ACID– 원자성 (Atomicity). All of the operations in the transaction will
complete, or none will.– 일관성 (Consistency). The database will be in a consistent state
when the transaction begins and ends.– 고립성 (Isolation). The transaction will behave as if it is the only
operation being performed upon the database.– 지속성 (Durability). Upon completion of the transaction, the op-
eration will not be reversed.http://queue.acm.org/detail.cfm?id=1394128
![Page 4: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/4.jpg)
Google BigTable System
Tablet Server #1
MapReduce Worker #1
GFS #1
Tablet Server #2
MapReduce Worker #2
GFS #2
BigTable Master Server
Chubby Server
![Page 5: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/5.jpg)
Google BigTable Architecture
GAE Application
DataNucleus ORM(JDO or JPA Implementation)
Low Level API
Datastore
GFS(Google File System)
Big
Table
Meta 0 tablet
Meta 1 tablet Meta 1 tablet Meta 1 tablet
Tab 1 Tab 2 Tab 3 Tab 4 Tab n
![Page 6: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/6.jpg)
Google BigTable• Dynamic control over
Data Layout and Format• Not support a full Rela-
tional Model• Dynamic control over
serving data from mem-ory or disk
• Column oriented• The map is indexed by
– A row key– A column name– A timestamp
BigTable
Key
TimeStamp / Version
Columns
Index
![Page 7: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/7.jpg)
LDAP(Lightweight Directory Access Protocol)
• The Lightweight Directory Access Protocol (LDAP) is an application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) net-work.
• Directory services may provide any organized set of records, often with a hierarchical structure, such as a corporate electronic mail directory. Similarly, a telephone directory is a list of sub-scribers with an address and a phone number.
![Page 8: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/8.jpg)
Google App Engine Product Roadmap• Features on Deck
– App Engine out of Preview– SSL access on custom domains– Full-text search over Datastore– Support for Python 2.7– Support for running MapReduce jobs across App En-
gine datasets– Bulk Datastore Import and Export tool– Improved monitoring and alerting of application serving– Logging system improvements to remove limits on size and
storage– Integration with Google Storage for Developer
![Page 9: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/9.jpg)
DataStore 의 제약사항• Write 가 느림• 검색 가능한 Text 사이즈 : 500 자• Entity 최대 사이즈 : 1Mb• 최대 Batch (save/delete) : 500 Entity• SQL 을 사용할 수 없음• 데이터베이스의 제약조건을 적용할 수 없음• Aggregation 함수를 지원하지 않음 (count, avg)• 쿼리가 최대 조회 레코드 : 2000• 쿼리에서 Equal 연산 이외의 조건은 1 개만 추가 가능• 트랜잭션의 단위는 Entity Group 에 한정 됨
![Page 10: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/10.jpg)
Datastore API and ORM API
Data-store API
Java ORM API
(JDO, JPA)
![Page 11: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/11.jpg)
Datastore API(Low Level API)• 모든 Entity 는 Kind 가 지정• Entity 는 하나이상의 Property 를 포함• 특정 Kind 에 포함된 Entity 는 동일한 Property 를 갖지 않음• Entity 의 Property 는 이름은 동일하지만 데이터 유형이 다를수 있음• Schemaless 의 특성• ID 는 지정하지 않을 경우 자동 할당 됨
Object-Oriented RDBMS Datastore
Class Table Kind
Object Record Entity
Attribute Column Property
![Page 12: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/12.jpg)
Entity• App Engine Datastore 에서 관리하는 객체는 Entity• Entity 는 하나의 Key 를 포함
– 모든 Entity 중 유일한 구분자• Key 의 구성
– Path– Parent Entity 의 key– Entity 의 Kind– Entity 에 할당된 이름
• App 이 할당한 값• Datastore 가 지정한 numeric ID
• 지원 데이터 타임– integers, floating point values, strings, dates, binary data 등
• 각 엔티티는 하나이상의 Property 를 포함– 하나의 Property 에 복수의 데이터 타입 저장 가능– 하나의 Property 에는 복수의 값이 저장 가능– 복수의 값들의 데이터 유형은 다를 수 있음
![Page 13: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/13.jpg)
Entity Model• Settings
– Kind : Table– Key : Primary Key– Entity Group : Partitioning– Property : column
Kind Person
key /Person:you
Entity Group /Person:you
Name helloworld
age 25
![Page 14: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/14.jpg)
Entity Example(Create)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
Entity employee = new Entity("Employee");employee.setProperty("firstName", "Antonio");employee.setProperty("lastName", "Salieri");Date hireDate = new Date();employee.setProperty("hireDate", hireDate);employee.setProperty("attendedHrTraining", true);
datastore.put(employee);
![Page 15: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/15.jpg)
Entity Example(Scheme Free)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
Entity employee = new Entity("Employee");employee.setProperty("firstName", “Sun-Jin");
datastore.put(employee);
![Page 16: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/16.jpg)
Entity Group• Entity Group 는 부모 Entity 를 갖는 Entity 의 Group
• 하나의 트랜잭션에서 여러 Entity 를 조회 , 생성 , 수정 , 삭제 할 수 가능하지만 그 범위는 단일 Entity Group 에 국한
• 부모 Entity 를 가지 않은 Entity 를 Root Entity 라고 함
• Entity Group 은 하나의 Root Entity 와 Child Entity 로 구성
• 하나의 Entity Group 은 동일한 분산 네트웍 서버에 저장됨 (Tablet 서버 )
![Page 17: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/17.jpg)
Entity Group
Root Entity(xxxx)
Child Entity(xxxxyyyy)
Child Entity(xxxxyyyyzzzz)
Entity Group
![Page 18: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/18.jpg)
Entity Example(Read)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
• Key rootKey = KeyFactory.createKey("Employee", “CEO");
• Entity employee = datastore.get(rootKey);
![Page 19: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/19.jpg)
Entity Example(Update)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
• Key rootKey = KeyFactory.createKey("Employee", “CEO");
• Entity employee = datastore.get(rootKey);
employee.setProperty("firstName", “Sun-Jin");employee.setProperty("lastName", “Jang");
datastore.put(employee);
![Page 20: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/20.jpg)
Entity Example(Delete)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
• Key rootKey = KeyFactory.createKey("Employee", “CEO");
• Entity employee = datastore.get(rootKey);
datastore.delete(employee);
![Page 21: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/21.jpg)
Entity Example(Query)•
// ...DatastoreService datastore = DatastoreServiceFactory.getDatastore-Service();
• Query q = new Query(”employee") • q.addFilter("lastName", Query.FilterOperator.EQUAL,
lastNameParam); • q.addFilter("height", Query.FilterOperator.LESS_THAN, maxHeight-
Param);
• PreparedQuery pq = datastore.prepare(q);
• for (Entity result : pq.asIterable()) { • String firstName = (String) result.getProperty("firstName"); • String lastName = (String) result.getProperty("lastName"); • Long height = (Long) result.getProperty("height"); • }
![Page 22: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/22.jpg)
Java ORM API
JDO(Java Data
Object)
JPA(Java Per-sistence
API)
![Page 23: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/23.jpg)
DataNucleus
• DataNucleus attempts to make the whole process of persisting data a transparent process.
• The idea revolves around the developer having a series of Java classes that need persisting.
http://www.datanucleus.org/products/accessplatform/development_process.html
![Page 24: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/24.jpg)
JDO Model Define(Annotation)• import javax.jdo.annotations.PersistenceCapable;
@PersistenceCapablepublic class Employee { @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.I-DENTITY) private Key key;
• @Persistent private String firstName;
@Persistent private String lastName;
@Persistent private Date hireDate; }
![Page 25: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/25.jpg)
JDO Model Define(Annotation)• import javax.jdo.annotations.PersistenceCapable;
// …
@PersistenceCapablepublic class Employee { @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.IDEN-TITY) private Key key;
• @Persistent private String firstName;
@Persistent private String lastName;
@Persistent private Date hireDate; }
![Page 26: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/26.jpg)
JDO Model(Create)• PersistenceManager pm =
PMF.get().getPersistenceManager();
Employee e = new Employee("Alfred", "Smith", new Date());
try { pm.makePersistent(e); } finally { pm.close(); }
![Page 27: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/27.jpg)
JDO Model(Update)• public void updateEmployeeTitle(User user, String
newTitle) { PersistenceManager pm = PMF.get().getPersistenceManager(); try { Employee e = pm.getObjectById(Employee.class, “Sun-Jin”); if (titleChangeIsAuthorized(e, newTitle) { e.setTitle(newTitle); } else { throw new UnauthorizedTitleChangeExcep-tion(e, newTitle); } } finally { pm.close(); }}
![Page 28: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/28.jpg)
JDO Model(Delete)• public void updateEmployeeTitle(User user, String
newTitle) { PersistenceManager pm = PMF.get().getPersistenceManager(); try { Employee e = pm.getObjectById(Employee.class, “Sun-Jin”);
• pm.deletePersistent(e); } finally { pm.close(); }}
![Page 29: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/29.jpg)
JDO Model(Query)• import java.util.List;
import javax.jdo.Query;
// ...
Query query = pm.newQuery(Employee.class); query.setFilter("lastName == lastNameParam"); query.setOrdering("hireDate desc"); query.declareParameters("String lastNameParam");
try { List<Employee> results = (List<Employee>) query.execute("Smith"); if (!results.isEmpty()) { for (Employee e : results) { // ... } } else { // ... no results ... } } finally { query.closeAll(); }
![Page 30: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/30.jpg)
Demo
![Page 31: S03 hybrid app_and_gae_datastore_v1.0](https://reader035.vdocuments.pub/reader035/viewer/2022062615/54857878b4af9f7d0d8b4e25/html5/thumbnails/31.jpg)
감사합니다 .
http://code.google.com/p/devmento-gae-seminar