DATA PERSISTENCE

As my previous articles, this also separated to six parts.

1. Introduction to data persistence

2. Data, Files, Databases, and DBMSs

3. Application to Files/DB

4. ORM

5. NOSQL and Hadoop

6. Information retrieval

When we talking about the role of data in information systems process with data persistence data and convert them into information. The data should persist for later use, to maintain the status, for logging purposes, to further process and derive knowledge (data science). Data can be stored, read, updated/modified, and deleted. At run time of software systems, data is stored in main memory, which is volatile. Data should be stored in non-volatile storage for persistence. There are two main ways of storing data.

• Files
• Databases

There are many formats for storing data

            -Plain-text, XML, JSON, tables, text files, images, etc…

What is data?

• Data are row facts, Can be processed (by application components) and converted to meaningful information.
• And there are so many data types, it separated to two parts,

Quantitative –   Numerical

                              Textual

                              Boolean

                              Data time

                       Qualitative –    Visibility

                                                      Modifiability

                                                      Usability

Working data is contained in computer memory, and memory is volatile. Data should be saved into non-volatile storages for persistence.

What is database?

                Databases are created and managed in database servers. SQL is used to process databases.

DDL –CRUD databases

DML –CRUD data in databases

        Database types

• Hierarchical databases
• Network databases
• Relational databases
• Non-relational databases (NoSQL)
• Object-oriented databases
• Graph databases
• Document databases

What is database server?

            It is similar to data house where the website store or maintain their data and information. A Database Server is a computer in a LAN that is dedicated to database storage and retrieval. Actually, database server holds the database management system and the databases.

What is database management system?

            DBMSs are used to connect to the DB servers and manage the DBs and data in them •PHPMyAdmin

•MySQL Workbench

Some other DBMS examples include:

        MySQL

        SQL Server

        Oracle

        dBASE

        FoxPro

Data arrangement

•Data warehouses

•Big Data

      -Volume

      -Variety

      -Velocity

Data Warehouse

• This is mainly an architecture, not a technology. With varieties of SQL based data sources, extracting data. And help for generating analytic reports.
• In preferences organization wants to know some informed decision, they prefer  to choose data warehousing, as for reports they need reliable or believable data from the sources
•  And handle mainly structured data.
• If organization want to know some informed decision, they prefer to choose data warehousing.
• As for this kind of report they need reliable or believable data from the sources.

 Big Data

• Big data mainly a technology, which stands on volume, velocity and variety of the data.
• Volume defines the amount of data coming from different sources
• Velocity refers to the speed of data processing.
• And varieties refer to the number of types of data.
• If organization need to compare with lot of big data, which contain valuable information and help them to take better decision (like how to lead more revenue. More profitability, more customers etc.), they obviously preferred big data approach.

Now let’s see how the application components communicate with files and databases. Files    and DBs are external components. They are existing outside the software system. Software can connect to the files/DBs to perform CRUD operations on data.

 File –File path

URL DB –connection string

To process data in DB

 •SQL statements

•Prepared statements

•Callable statements

SQL statements	Prepared statements	Callable statements
Execute standard SQL statements from the application Statement stmt= con.createStatement();  stmt.executeUpdate(“update STUDENT set NAME =”+ name + “ where ID =”+ id + “)”;	The query only needs to be parsed (or prepared) once, but can be executed multiple times with the same or different parameters. PreparedStatementpstmt= con.prepareStatement(“update STUDENT set NAME = ? where ID = ?”); pstmt.setString(1, “MyName”); pstmt.setInt(2, 111); pstmt.executeUpdate();	Execute stored procedures CallableStatementcstmt= con.prepareCall(“{call anyProcedure(?, ?, ?)}”); cstmt.execute();

Then will see the need for ORM, explaining the development with and without ORM,

• There are different structures for holding data at runtime

•Application holds data in objects

•Database uses tables (entities)

• But mapping data in objects to the tables are more easy using Objet Relational Mapping(ORM)
• Mismatches between relational and object models

• Granularity : Object model has more granularity than relational model.

• Subtypes : Subtypes (means inheritance) are not supported by all types of relational databases.

• Identity : Like object model, relational model does not expose identity while writing equality.

•Associations : Relational models cannot determine multiple relationships while looking into an object domain model.

• Data navigation : Data navigation between objects

Next we will discuss the POJO, Java Beans, and JPA, indicating their similarities and differences,

• POJO

POJO stands for Plain Old Java Object. It is an ordinary Java object, not bound by any special restriction other than those forced by the Java Language Specification and not requiring any class path. POJOs are used for increasing the readability and re-usability of a program. POJOs have gained most acceptance because they are easy to write and understand. They were introduced in EJB 3.0 by Sun microsystems.

• A POJO should not:

•Extend pre-specified classes.

•Implement pre-specified interfaces.

•Contain pre-specified annotations.

Beans

• Beans are special type of Pojos. There are some restrictions on POJO to be a bean

• All JavaBeans are POJOs but not all POJOs are JavaBeans.

• Serializable i.e. they should implement Serializable interface. Still some POJOs who don’t implement Serializable interface are called POJOs because Serializable is a marker interface and therefore not of much burden.

• Fields should be private. This is to provide the complete control on fields.

• Fields should have getters or setters or both.

• A no-arg constructor should be there in a bean.

• Fields are accessed only by constructor or getter setters.

Java Persistence API (JPA)

•An API/specification for ORM

•Uses

  •POJO classes

  •XML based mapping file (represent the DB)

  •A provider (implementation of JPA)

Then, will see what re the ORM tools available for different platforms like java, php, .net etc.,

• Java ORM Tools are,

Hibernate

IBatis/MyBatis

Toplink

• PHP ORM Tools are,

Doctrine

Now will see that, needs for NoSQL indicating the benefits and also different types of NoSQL databases,

Not Only SQL (NOSQL)

•Relational DBs are good for structured data

•For semi-structured and un-structured data, some other types of DBs can be used

•Key-value stores

•Document databases

•Wide-column stores

•Graph stores

Benefits of NoSQL

•When compared to relational databases, NoSQL databases are more scalable and provide superior performance, and their data model addresses several issues that the relational model is not designed to address:

•Large volumes of rapidly changing structured, semi-structured, and unstructured data

NoSQL DB servers

•MongoDB

•Cassandra

•Redis

•Amazon DynamoDB

•Hbase

Next will see what Hadoop is, explaining the core concepts of it,

Hadoop

• The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

•It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

• Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures.

Hadoop core concepts

• Hadoop Distributed File System (HDFSTM): A distributed file system that provides high throughput access to application data

• Hadoop YARN: A framework for job scheduling and cluster resource management.

• Hadoop Map Reduce: A YARN-based system for parallel processing of large data sets.

Now, we will discuss the concept of IR, identifying tools for IR,

•Data in the storages should be fetched, converted into information, and produced for proper use

•Information is retrieved via search queries

     • Keyword search

      • Full-text search

•The output can be

     • Text

     • Multimedia

•The information retrieval process should be,

      • Fast/performance

      •Scalable

      •Efficient

      •Reliable/Correct

So finally, I hope you got a good idea of data persistence, will meet in the next article.

REFERENCES

http://ecomputernotes.com/fundamental/what-is-a-database/what-is-a-database-server

https://www.techopedia.com/definition/24361/database-management-systems-dbms

https://smw-rda.esc.rzg.mpg.de/index.php/Data_Arrangement

Big Data vs Data Warehouse – Find Out The Best Differences