Implement OGC Catalog Service for the Web

I. Project summary

This page summarizes that work done during the Google Summer of Code 2018.

1. Scope and requirement

The goal of this project is to improve the Apache SIS services modules in order to meet some needs of space agencies. Particulary, this includes is to focus on implemening a OGC catalog service for the web that the basis is to use the Apache SIS storage modules comply with CSW standard and creating a simple web application to ensure that the module is on the right track, meets the requirements of the user.
This work requires basic understanding of at least two international standards published by the Open Geospatial Consortium (OGC): the OGC Catalog Service on the Web (CSW) 3.0 and the OGC Web Services Common Standard (OWS) 2.0. Before to start coding, the first part of this GSOC 2018 was read those standards and combined with exploring the appropriate web technologies to implement the CSW service.
Example: Apache CXF , Restful API ...

2. Context

The most generic parts of this work will be intergrated in the Apache SIS. The web application will be push in a new github repository that will be linked to the Apache SIS.
The programming enviroment is:

Programming language: Java.
Java platform: JDK8.
Library: Apache SIS on JDK8 branch, GeoAPI 4.0, Apache CXF 3.2.4 , jetty9.4.11.v20180605 .
IDE: Netbeans.
Build tools: Maven.
Source code repository: clone on GitHub.
Other languages : HTML, CSS, Javascript... .

3. Work done

The work has been committed on a clone of the Apache SIS project.
This is the link commits by me:
The part web service :
https://github.com/haonguyen123/sis/commits/CSW?author=haonguyen123
The part web client :
https://github.com/haonguyen123/CSWWebApplication/commits?author=haonguyen123

4. Work details

This section give more details on some parts of the work.

4.1. Analysis the web service

In this part, we will analysis the web service and then select the appropriate technology and design the classes for the system.

Figure 1: The CSW System map

This HTTP protocol binding maps each of the abstract model operations, specified in the general model (see OGC 12-168), to a corresponding Catalogue Services for the Web (CSW) operation. This clause describes the request and response messages that are common to all web-based catalogue services. The basic message exchange pattern is illustrated in Figure 1.

The interaction between a client and a server is accomplished using a standard HTTP request-response model. Specifically, a client sends a request to a server using HTTP, and expects to receive a response to the request or an exception message.

Request and response messages are encoded as keyword-value pairs either within a request URI or using an XML entity-body.

So, to implement a web service using HTTP protocol, the Apache CXF framework is one of the top choices for this project. It helps to build and develop services using frontend programming APIs, like JAX-WS and JAX-RS. These services can speak a variety of protocols such as SOAP, XML/HTTP, RESTful HTTP, or CORBA and work over a variety of transports such as HTTP, JMS or JBI.

4.2. Maven dependencies

In this part, we need edit the pom.xml file to include the following dependencies:

<groupId>org.eclipse.jetty</groupId>

<artifactId>jetty-http</artifactId>

</dependency>

<groupId>org.eclipse.jetty</groupId>

<artifactId>jetty-io</artifactId>

</dependency>

<groupId>org.eclipse.jetty</groupId>

<artifactId>jetty-server</artifactId>

</dependency>

<groupId>org.eclipse.jetty</groupId>

<artifactId>jetty-util</artifactId>

</dependency>

<groupId>javax.servlet</groupId>

<artifactId>javax.servlet-api</artifactId>

<scope>provided</scope>

</dependency>

<groupId>org.apache.cxf</groupId>

<artifactId>cxf-rt-frontend-jaxrs</artifactId>

</dependency>

<groupId>org.apache.cxf</groupId>

<artifactId>cxf-rt-transports-http-jetty</artifactId>

</dependency>

<groupId>org.apache.httpcomponents</groupId>

<artifactId>httpclient</artifactId>

</dependency>

These dependencies will be used primarily to add a JAX-RS based webservice (section 4.6) in CatalogService class and create the main class which will start server (section 4.7) in RunServer class.

<groupId>org.apache.cxf</groupId>

<artifactId>cxf-rt-rs-extension-search</artifactId>

</dependency>

<groupId>org.apache.olingo</groupId>

<artifactId>olingo-odata2-core-incubating</artifactId>

</dependency>

<groupId>org.locationtech.jts</groupId>

</dependency>

</dependencies>

These dependencies will be used primarily to DiscoveryImpl class that support for GetRecords operation to query the records.

4.3. Creating Java data objects for Request and Response (Create POJOs classes)

XML Schema Document files (XSD), Unified Modeling Language (UML) in the document OGC® Catalogue Services 3.0 Specification - HTTP Protocol Binding and OGC Web Service Common Implementation Specification , we will develop the Java data object.

Example: with Record object. We will name this implementation class as Record. The following is the code listing of the Record.java object.

package org.apache.sis.services.csw.discovery;

import javax.xml.bind.annotation.*;

@XmlType(name = "RecordType", namespace = Namespaces.CSW, propOrder = {

"title",

"creator",

"subject",

"description",

"publisher",

"contributor",

"date",

"type",

"format",

"identifier",

"source",

"language",

"relation",

"rights",

"coverage",})

@XmlRootElement(name = "Record", namespace = Namespaces.CSW)

public class Record extends AbstractRecord {

private String title;

private String creator;

private String subject;

private String description;

private String publisher;

private String contributor;

private Date date;

private String type;

private String format;

private String identifier;

private String source;

private String language;

private String relation;

private BoundingBox coverage;

private String rights;

@XmlElement(name = "title", namespace = Namespaces.DC)

public String getTitle() {

return title;

}

public void setTitle(String title) {

this.title = title;

}

@XmlElement(name = "creator", namespace = Namespaces.DC)

public String getCreator() {

return creator;

}

public void setCreator(String creator) {

this.creator = creator;

}

@XmlElement(name = "subject", namespace = Namespaces.DC)

public String getSubject() {

return subject;

}

public void setSubject(String subject) {

this.subject = subject;

}

@XmlElement(name = "abstract", namespace = Namespaces.DCT)

public String getDescription() {

return description;

}

public void setDescription(String description) {

this.description = description;

}

@XmlElement(name = "publisher", namespace = Namespaces.DC)

public String getPublisher() {

return publisher;

}

public void setPublisher(String publisher) {

this.publisher = publisher;

}

@XmlElement(name = "contributor", namespace = Namespaces.DC)

public String getContributor() {

return contributor;

}

public void setContributor(String contributor) {

this.contributor = contributor;

}

@XmlElement(name = "modified", namespace = Namespaces.DCT)

public Date getDate() {

return date;

}

public void setDate(Date date) {

this.date = date;

}

@XmlElement(name = "type", namespace = Namespaces.DC)

public String getType() {

return type;

}

public void setType(String type) {

this.type = type;

}

@XmlElement(name = "format", namespace = Namespaces.DC)

public String getFormat() {

return format;

}

public void setFormat(String format) {

this.format = format;

}

@XmlElement(name = "identifier", namespace = Namespaces.DC)

public String getIdentifier() {

return identifier;

}

public void setIdentifier(String identifier) {

this.identifier = identifier;

}

@XmlElement(name = "source", namespace = Namespaces.DC)

public String getSource() {

return source;

}

public void setSource(String source) {

this.source = source;

}

@XmlElement(name = "language", namespace = Namespaces.DC)

public String getLanguage() {

return language;

}

public void setLanguage(String language) {

this.language = language;

}

@XmlElement(name = "relation", namespace = Namespaces.DC)

public String getRelation() {

return relation;

}

public void setRelation(String relation) {

this.relation = relation;

}

@XmlElement(name = "BoundingBox", namespace = Namespaces.OWS)

public BoundingBox getCoverage() {

return coverage;

}

public void setCoverage(BoundingBox coverage) {

this.coverage = coverage;

}

@XmlElement(name = "rights", namespace = Namespaces.DC)

public String getRights() {

return rights;

}

public void setRights(String rights) {

this.rights = rights;

}

The Request and Response data objects need to be serialized in the required format, such as XML, for communicating between a RESTful service and a client. To serialize the data objects, you need to use one of the data binding components or create your own custom data binding components, which create a mapping between Java objects and XML (or the required format). CXF uses JAXB as the default data binding component. JAXB uses annotations to define the mapping between Java objects and XML.
The @XmlRootElement annotations associated with Record class map the Record class to an XML root element. The attributes contained within the Record object by default are mapped to @XmlElement. The @XmlElement annotations are used to define elements within the XML. The @XmlRootElement and @XmlElement annotations allow you to customize the namespace and name of the XML element. If no customizations are provided, the JAXB runtime by default would use the same name for the attribute as the XML element. CXF handles this mapping of Java objects to XML transparently behind the scenes.
The following is an XML that maps to the Record data object.

<csw30:Record
  xmlns:csw30="http://www.opengis.net/cat/csw/3.0"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:dct="http://purl.org/dc/terms/"
  xmlns:ows="http://www.opengis.net/ows/2.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.opengis.net/cat/csw/3.0
                   ../../../csw/3.0/record.xsd">
  <dc:creator>U.S. Geological Survey</dc:creator>
  <dc:contributor>State of Texas</dc:contributor>
  <dc:publisher>U.S. Geological Survey</dc:publisher>
  <dc:subject>Elevation, Hypsography, and Contours</dc:subject>
  <dc:subject>elevation</dc:subject>
  <dct:abstract>Elevation data collected for the National Elevation Dataset
(NED) based on 30m horizontal and 15m vertical accuracy.</dct:abstract>
  <dc:identifier>ac522ef2-89a6-11db-91b1-7eea55d89593</dc:identifier>
  <dc:relation>OfferedBy</dc:relation>
  <dc:source>dd1b2ce7-0722-4642-8cd4-6f885f132777</dc:source>
  <dc:rights>Copyright (c) 2004, State of Texas</dc:rights>
  <dc:type>Service</dc:type>
  <dc:title>National Elevation Mapping Service for Texas</dc:title>
  <dct:modified>2004-03-01</dct:modified>
  <dc:language>en</dc:language>
  <ows:BoundingBox>
<ows:LowerCorner>-108.44 28.229</ows:LowerCorner>
<ows:UpperCorner>-96.223 34.353</ows:UpperCorner>
  </ows:BoundingBox>
</csw30:Record>

Record class is one in just one of many object classes deployed in the Web Service with OGC CSW Standards. With other classes we do the same.

Note: Currently, these classes are created manually based on the XSD files provided by OGC and use JAXB annotations.

4.4. Operations definitions

Figure: 2: Protocol sequence diagram

Figure 2 shows the request/response message pairs for all the operations defined for the web catalogue service (CSW). There are three classes of operations: service operations which are operations a client may use to interrogate the service to determine its capabilities; discovery operations which a client may use to determine the information model of the catalogue and query catalogue records; and management operations which are used to create or change records in the catalogue.

In Apache SIS, these three classes will in turn correspond to : OGC_Service.java , Discovery.java , Manager.java and OGC_ServiceImpl.java, DiscoveryImpl.java,ManagerImpl.java are their implementation classes.

Curently, the CSW web service implementing three main operations :

- GetCapabilities operation : The mandatory GetCapabilities operation allows CSW clients to retrieve service metadata from a server. The response to a GetCapabilities request is an XML document containing service metadata about the server.

- GetRecordById Operation : The GetRecordById operation is an implementation of the GetResourceById operation from the general model. This operation presumes that a previous query has been performed in order to obtain the identifiers that may be used with this operation. For example, records returned by a GetRecords operation may contain references to other records in the catalogue that may be retrieved using the GetRecordById operation. This operation is also a subset of the GetRecords operation, and is included as a convenient short form for retrieving and linking to records in a catalogue.

- GetRecords Operation: The primary means of resource discovery in the general model is the Discovery.query operation. In the HTTP protocol binding this operation is implemented as the mandatory GetRecords operation, which does a search operation.

And other operation as:

- GetFile Operation : This operation isn't a standard of OGC but it needs to be added so that users can download the data they need .

4.5. Use the Apache SIS storage modules to get data

In this part, the data collected will be stored in a structured path similar to:

- home
-haonguyen
-data
- Landsat5
- LT051250532010021101T1-SC20171005023341
- Landsat7
- Landsat8
- NetCDF
- cami_0000-09-01_64x128_L26_c030918.nc

Next, we create a Data Access object and use the Apache SIS storage module to get the Metadata from the collected data. Creating a Data Access object is a common way to access the application's data store.

We will now develop the Data Access object. Example, in ApacheSIS CSW project , name this implementation class as DiscoveryDAO.

4.6. Add a JAX-RS based webservice

In this part, we will develop the implementation class to realize CSW Service. In ApacheSIS CSW project, this class name is CatalogService.

The CatalogService also had various annotations defined on the class and methods definition. These annotations are JAX-RS annotations, which are used at runtime to create the RESTful service resource. In JAX-RS terminology, a class which has the JAX-RS annotations defined is termed as a Resource class

Example:
- The @Path annotation defines the URI path that a resource class or class method will serve a request for.
- The @Produces annotation defines the content type that the method or resource class can produce and send back to the client.
- @GET and @POST are called Request Method Designators,
- The @QueryParam annotation is used to inject URI query parameter into Java method.
In this class, each method implementation will talk to data store use the CategoryDAO and carry out the corresponding operation.

4.7. Create the main class which will start server

In this part, we create a main class use the CXF org.apache.cxf.jaxrs.JAXRSServerFactoryBean class, which exposes the CategoryService over HTTP without the overhead of deploying the CatalogService in a web container.

II. Test Operations

This is link for the test service document:
https://docs.google.com/document/d/1A5Jy9tuiyQQyex_h7HdekGGT0TpX90GrMewzrewXfuA/edit?usp=sharing

III. Client Interface

Home page

List Data return after make Text search with q = geotiff

Detail Dataset when click on a record

List Data return after make query with constraint =title==*LC*

List Data return after make search records with recordids =sst_mnmean

List Data return after make spatial search with bbox

List Data return after make spatial search with polygon

List Data return after make temporal search with time

List Data return after make sortby date

IV. Future improvement

Based on the work done this summer, in the future we will:
- Additional operations have not yet been implemented (GetDomain, Harvert, Transaction operation).
- Complete database system.
- Complete client interface with login/logout , role , publish data ...
- Platform for developing other web service standards : WMS, WCS …

V. References

- OGC® Catalogue Services 3.0 Specification - HTTP Protocol Binding
- OGC® Catalogue Services 3.0 - General Model
- OGC Web Service Common Implementation Specification
- REST and SOAP Services with Apache CXF

Tìm kiếm Blog này

GSOC Project 2018