UNIT-4 FUNDAMENTALS OF ANN AND EXPERT SYSTEM
Structure and Functioning of Biological Neural Network (BNN)
Figure 1: Three Stages of Biological Neural System
The neural system of the human body consists of three stages:
1. Receptors
2. A Neural Network 3. Effectors.
The receptors receive the stimuli either internally or from the external world, and then pass the information into the neurons in a form of electrical impulses.
The neural network then processes the inputs then makes proper decision of outputs.
Finally, the effectors translate electrical impulses from the neural network into responses to the outside environment.
Figure 1 shows the bidirectional communication between stages for feedback.
Neuron
The fundamental element of the neural network is called a neuron.
Figure 2: Structure of Neuron in human brain
A neuron mainly consists of three parts:
1. Dentrites 2. Soma 3. Axon
Dentrites are the tree-like structure that receives the signal from surrounding neurons, where each line is connected to one neuron.
Soma is the body of the neuron.
Axon is a thin cylinder that transmits the signal from one neuron to others.
At the end of axon, the contact to the dendrites is made through a synapse.
The human brain contains over ten billion (109) neurons, each of which is connected, on average to several thousand other neurons. These connections are known as synapse, and the human brain contains about 60 trillion (1012) such connections.
Artificial Neural Network (ANN)
Artificial Neural Network (ANN) is an efficient computing system whose central theme is borrowed from the analogy of biological neural networks.
ANN acquires a large collection of units that are interconnected in some pattern to allow communication between the units. These units, also referred to as nodes or neurons, are simple processors which operate in parallel.
1. Every neuron is connected with other neuron through a connection link.
2. Each connection link is associated with a weight that has information about the input signal. The weight is the most useful information for neurons to solve a particular problem because the weight usually excites or inhibits the signal that is being communicated.
3. Each neuron has an internal state, which is called an activation signal.
Output signals, which are produced after combining the input signals and activation rule, may be sent to other units.
ANN versus BNN
Similarities based on the terminology used.
Biological Neural Network (BNN) Artificial Neural Network (ANN)
Soma Node
Dendrites Input
Synapse Weights or Interconnections
Axon Output
Comparison between ANN and BNN based on some criteria.
Criteria BNN ANN
Processing Massively parallel, slow but superior than ANN
Massively parallel, fast but inferior than BNN
Size 1011 neurons and 1015 interconnections
102 to 104 nodes
Learning They can tolerate ambiguity Very precise, structured and formatted data is required to tolerate ambiguity
Fault tolerance
Performance degrades with even partial damage
It is capable of robust performance, hence has the potential to be fault tolerant Storage
capacity
Stores the information in the synapse
Stores the information in continuous memory locations
Model of Artificial Neural Network
The following diagram represents the general model of ANN followed by its processing.
For the above general model of artificial neural network, the net input can be calculated as follows −
yin= x1.w1+x2.w2+x3.w3…xm.wm
∑
The output can be calculated by applying the activation function over the net input.
Y=F(yin)
Output = function (net input calculated)
Processing of ANN depends upon the following three building blocks:
Network Topology
Adjustments of Weights or Learning
Activation Functions
Network Topology
A network topology is the arrangement of a network along with its nodes and connecting lines. According to the topology, ANN can be classified as the following kinds –
1. Feedforward Network 2. Feedback Network
1. Feedforward Network
It is a non-recurrent network having processing units/nodes in layers and all the nodes in a layer are connected with the nodes of the previous layers.
The connection has different weights upon them.
There is no feedback loop means the signal can only flow in one direction, from input to output. It may be divided into the following two types −
Single layer feedforward network − Here the input layer is fully connected to the output layer. It has only one weighted layer.
Multilayer feedforward network – It has more than one weighted layer.
As this network has one or more layers between the input and the output layer, it is called hidden layers.
2. Feedback Network
A feedback network has feedback paths, which means the signal can flow in both directions using loops. This makes it a non-linear dynamic system, which changes continuously until it reaches a state of equilibrium. It may be divided into the following types −
Fully recurrent network − It is the simplest neural network architecture because all nodes are connected to all other nodes and each node works as both input and output.
Jordan network − It is a closed loop network in which the output will go to the input again as feedback.
Adjustments of Weights or Learning
Learning, in artificial neural network, is the method of modifying the weights of connections between the neurons of a specified network. Learning in ANN can be classified mainly into supervised learning, unsupervised learning.
1. Supervised Learning
This learning is done under the supervision of a teacher. This learning process is dependent.
During the training of ANN under supervised learning, the input vector is presented to the network, which will give an output vector.
This output vector is compared with the desired output vector.
An error signal is generated, if there is a difference between the actual output and the desired output vector.
On the basis of this error signal, the weights are adjusted until the actual output is matched with the desired output.
2. Unsupervised Learning
This type of learning is done without the supervision of a teacher. This learning process is independent.
During the training of ANN under unsupervised learning, the input vectors of similar type are combined to form clusters. When a new input pattern is applied, then the neural network gives an output response indicating the class to which the input pattern belongs.
There is no feedback from the environment as to what should be the desired output and if it is correct or incorrect. Hence, in this type of learning, the network itself must discover the patterns and features from the input data, and the relation for the input data over the output.
Activation Functions
It may be defined as the extra force or effort applied over the input to obtain an exact output. In ANN, we can also apply activation functions over the input to get the exact output. Followings are some activation functions of interest −
1. Linear Activation Function
It is also called the identity function as it performs no input editing. It can be defined as −
F(x) = x
2. Sigmoid Activation Function It is of two types as follows −
Binary sigmoidal function − This activation function performs input editing between 0 and 1. It is positive in nature. It is always bounded, which means its output cannot be less than 0 and more than 1. It is also strictly increasing in nature, which means more the input higher would be the output. It can be defined as
F(x) = sigm(x)
Bipolar sigmoidal function − This activation function performs input editing between -1 and 1. It can be positive or negative in nature. It is always bounded, which means its output cannot be less than -1 and more than 1. It is also strictly increasing in nature like sigmoid function. It can be defined as
F(x) = sigm(x)
Perceptron
The perceptron is a simple neuron that employs supervised learning rule and is able to classify the data into two classes.
Operational characteristics of the perceptron:
It consists of a single neuron with an arbitrary number of inputs along with adjustable weights, but the output of the neuron is 1 or 0 depending upon the threshold (greater than or less than equal to t respectively). It also consists of a bias whose weight is always 1. Following figure gives a schematic representation of the perceptron.
Perceptron thus has the following three basic elements −
Links − It would have a set of connection links, which carries a weight including a bias always having weight 1.
Adder − It adds the input after they are multiplied with their respective weights.
Activation function − It limits the output of neuron. A basic activation function is a Heaviside step function that has two possible outputs. This function returns 1, if the input is positive, and 0 for any negative input.
Learning Algorithm for perceptron
Perceptron network can be trained for single output unit as well as multiple output units.
Learning Algorithm for Single Output Unit
Step 1 − Initialize the following to start the training −
Weights
Bias
Learning rate α
Step 2 − Continue step 3-8 when the stopping condition is not true.
Step 3 − Continue step 4-6 for every training vector x.
Step 4 − Activate each input unit as follows − xi = si ( i = 1 to n)
Step 5 − Now obtain the net input with the following relation −
Here ‘b’ is bias and ‘n’ is the total number of input neurons.
Step 6 − Apply the following activation function to obtain the final output.
Step 7 − Adjust the weight and bias as follows − Case 1 − if y ≠ t then,
Case 2 − if y = t then,
Here ‘y’ is the actual output and ‘t’ is the desired/target output.
Step 8 − Test for the stopping condition, which would happen when there is no change in weight.
XOR Problem
The XOR, or “exclusive or”, problem is a classic problem in ANN research. It is the problem of using a neural network to predict the outputs of XOR logic gates given two binary inputs. An XOR function should return a true value if the two inputs are not equal and a false value if they are equal. All possible inputs and predicted outputs are shown in figure 1.
XOR is a classification problem and one for which the expected outputs are known in advance. It is therefore appropriate to use a supervised learning approach.
On the surface, XOR appears to be a very simple problem, however, Minksy and Papert (1969) showed that this was a big problem for neural network architectures of the 1960s, known as perceptrons.
Training Algorithm for Multiple Output Units
The following diagram is the architecture of perceptron for multiple output classes.
Step 1 − Initialize the following to start the training −
Weights
Bias
Learning rate α
For easy calculation and simplicity, weights and bias must be set equal to 0 and the learning rate must be set equal to 1.
Step 2 − Continue step 3-8 when the stopping condition is not true.
Step 3 − Continue step 4-6 for every training vector x.
Step 4 − Activate each input unit as follows −
Step 5 − Obtain the net input with the following relation −
Here ‘b’ is bias and ‘n’ is the total number of input neurons.
Step 6 − Apply the following activation function to obtain the final output for each output unit j = 1 to m −
Step 7 − Adjust the weight and bias for x = 1 to n and j = 1 to m as follows − Case 1 − if yj ≠ tj then,
Case 2 − if yj = tj then
Here ‘y’ is the actual output and ‘t’ is the desired/target output.
Step 8 − Test for the stopping condition, which will happen when there is no change in weight.
Multilayer Perceptron
A multilayer perceptron (MLP) is a deep, artificial neural network. It is composed of more than one perceptron. They are composed of an input layer to receive the signal, an output layer that makes a decision or prediction about the input, and in between those two, an arbitrary number of hidden layers that are the true computational engine of the MLP. MLPs with one hidden layer are capable of approximating any continuous function.
Multilayer perceptrons are often applied to supervised learning problems: they train on a set of input-output pairs and learn to model the correlation (or dependencies) between those inputs and outputs. Training involves adjusting the parameters, or the weights and biases, of the model in order to minimize error.
Backpropagation is used to make those weigh and bias adjustments relative to the error, and the error itself can be measured in a variety of ways, including by root mean squared error (RMSE).
Why We Need Backpropagation?
While designing a Neural Network, in the beginning, we initialize weights with some random values or any variable for that fact.
So, it‟s not necessary that whatever weight values we have selected will be correct, or it fits our model the best.
Okay, fine, we have selected some weight values in the beginning, but our model output is way different than our actual output i.e. the error value is huge.
Now, how will you reduce the error?
Basically, what we need to do, we need to somehow explain the model to change the parameters (weights), such that error becomes minimum.
Let‟s put it in an another way, we need to train our model.
One way to train our model is called as Backpropagation. Consider the diagram below:
Backpropagation
Back Propagation Neural (BPN) is a multilayer neural network consisting of the input layer, at least one hidden layer and output layer. As its name suggests, back propagating will take place in this network. The error which is calculated at the output layer, by comparing the target output and the actual output, will be propagated back towards the input layer.
Architecture
The architecture of BPN has three interconnected layers having weights on them. The hidden layer as well as the output layer also has bias, whose weight is always 1, on them. BPN works in two phases. One phase sends the signal from the input layer to the output layer, and the other phase back propagates the error from the output layer to the input layer.
ANN Application Areas
Followings are some of the areas, where ANN is being used.
1. Speech Recognition
Speech occupies a prominent role in human-human interaction. Therefore, it is natural for people to expect speech interfaces with computers.
2. Character Recognition
It is an interesting problem which falls under the general area of Pattern Recognition. Many neural networks have been developed for automatic recognition of handwritten characters, either letters or digits.
3. Signature Verification Application
Signatures are one of the most useful ways to authorize and authenticate a person in legal transactions. Signature verification technique is a non-vision based technique.
4. Human Face Recognition
It is one of the biometric methods to identify the given face.
Expert System
An expert system is a computer program that uses artificial intelligence technologies to simulate the judgment and behaviour of a human or an organization that has expert knowledge and experience in a particular field.
Characteristics of Expert Systems
High performance
Understandable
Reliable
Highly responsive
Phases in building Expert Systems
The following points highlight the five main stages to develop an expert system. The stages are:
1. Identification 2. Conceptualisation
3. Formalisation (Designing) 4. Implementation
5. Testing (Validation, Verification and Maintenance).
Stage # 1. Identification:
Before we can begin to develop an expert system, it is important to describe, with as much precision as possible, the problem which the system is intended to solve. We must determine the exact nature of the problem and state the precise goals which indicate exactly how the expert system is expected to contribute to the solution.
Stage # 2. Conceptualisation:
Once it has been identified for the problem an expert system is to solve, the next stage involves analysing the problem further to ensure that its specifics, as well as generalities, are understood.
In the conceptualisation stage, the knowledge engineer frequently creates a diagram of the problem to depict graphically the relationships between the objects and processes in the problem domain. It is often helpful at this stage to divide the problem into a series of sub-problems.
Stage # 3. Formalisation (Designing):
During the formalization stage, the problem is connected to its proposed solution. An expert system is supplied by analyzing the relationships depicted in the conceptualization stage. The knowledge engineer begins to select the techniques which are appropriate for developing this particular expert system.
Stage # 4. Implementation:
During the implementation stage the formalised concepts are programmed into the computer which has been chosen for system development, using the predetermined techniques and tools to implement a „first-pass‟ (prototype) of the expert system.
Stage # 5. Testing (Validation, Verification and Maintenance):
Knowledge engineer does not expect the testing process to verify that the system has been constructed entirely correctly. Rather, testing provides an opportunity to identify the weaknesses in the structure and implementation of the system and to make the appropriate corrections.
Expert System Architecture
The user interacts with the system through a user interface which may use menus, natural language or any other style of interaction. User Interface controls the dialog between the user and the system.
The Knowledge Base contains the knowledge necessary for understanding, formulating, and solving problems.
Then an inference engine is used to reason with both the expert knowledge and data specific to the particular problem being solved. The expert knowledge will typically be in the form of a set of IF-THEN rules. The case specific data includes both data provided by the user and partial conclusions based on this data.
Almost all expert systems also have an explanation subsystem, which allows the program to explain its reasoning to the user.
Some systems also have a knowledge base editor which helps the expert or knowledge engineer to easily update and check the knowledge base.
Expert System Shells: The general purpose bit is referred to as an expert system shell. The shell will provide the inference engine, a user interface, an explanation system and sometimes a knowledge base editor. Given a new kind of problem to solve (say, car design), we can usually find a shell that provides the right sort of support for that problem, so all we need to do is provide the expert knowledge. There are numerous commercial expert system shells, each one appropriate for a slightly different range of problems. Using shells to write expert systems generally greatly reduces the cost and time of development (compared with writing the expert system from scratch).
DIFFERENCE BETWEEN EXPERT SYSTEM AND TRADITIONAL SYSTEM
The principle distinction between expert systems and traditional problem solving programs is the way in which the problem related expertise is coded.
EXPERT SYSTEM TRADITIONAL SYSTEM
Problem expertise is encoded in both program and data structures.
The entire problem related expertise is encoded in data structures only, none is in programs.
Generally in expert systems, the use of knowledge is vital.
But in traditional system data is used more efficiently than knowledge.
Expert systems are capable of explaining how a particular conclusion is reached and why requested information is needed during a process.
Traditional systems are not capable of explaining a particular conclusion for a problem. These systems try to solve in a straight forward manner.
Expert system generally uses the symbolic representations for knowledge i.e. the rules, different forms of networks, frames, scripts etc. and performs their inference through symbolic computations.
But traditional systems are unable to express these terms. They just simplify the problems in a straight forward manner and are incapable to express the “how, why” questions.
The problem solving tools those are present in expert system.
The problem solving tools are absent in traditional systems.
The solution of the problem is more accurate than a traditional system.
Less accurate than expert system.
Pros and cons of an expert system:
Pros of an expert system:
1. Consistency
Consistency is the main benefit of an expert system. Since it is a computer- based system, all the knowledge or logic are programmed into it. If it meets the same situation, it will make the same decision again and again because the decisions are made based on some rules and logic.
2. Memory
It is one of the most important advantages of an expert system. It has a huge memory to store a large amount of knowledge and there is no chance of memory loss by default.
3. Logic
The logic in an expert system is very clear. In an expert system, all the rules, conditions and their understanding to make a decision is always clear because it is programmed into it.
4. Persistence/Accessibility
The expert systems are always available. They can be accessed anytime i.e 24*7.
5. Availability
The expert systems are available for all at the same time. Multiple users can access or use an expert system simultaneously and get responses from it immediately. (There is no kind of situations like wait or hold on, the system is busy with someone else and you can‟t use or ask any questions to it right now.)
6. Longevity
All of the human experts in the world have a fixed age limit because a human never lives forever. But in case of an expert system, if you can condense all of the knowledge and experience of a human expert in a computer it will be accessible forever.
Cons of an expert system:
1. Data integrity
The expert systems need to be updated manually. So, the expert system all itself doesn‟t learn. That is why data integrity is one of the major disadvantages of an expert system. Since the world we are living is constantly changing, so the system has to be updated manually by some set of programmer or some human experts in that domain.
2. Time & Cost
The time and cost required to buy or set up an expert system are very high. It requires a huge amount of time to acquire the knowledge that is needed to develop an expert system. On the other hand, the knowledge engineers are in small quantity in the industry and very expensive.
3. Specific
The expert systems are generally developed for a specific domain.
4. Emotionless
The human experts have a sort of awareness about the situation that means how they feel, how much effective they are in a situation. But expert systems have no awareness about any situation that they face.
5. Common-sense
Common-sense is the main issue with the expert systems. They can take some dumb decision sometimes because they have to go through by following the rules and regulations by which they are programmed. They can‟t provide a solution for the totally new kind of problem.
