Performance Testing Framework in a Heterogeneous and Hybrid Smart Grid Communication Network

Heterogeneous and hybrid smart grid communication network is comprised of different communication mediums and technologies. Performance evaluation is one of the main concerns for smart grid communication system. In any smart grid communication implementation, to determine the performance factor of the network, a testing of an end-to-end process flow is required. An effective and coordinated testing procedure plays a crucial role in evaluating the performance of smart grid communications. Therefore, this study proposes a testing framework which specifies the types of communication mediums and technologies, the evaluation criteria and software tools to carry out the testing. The proposed testing scheme is used as a guideline to analyze and assess the performance of smart grid communication system.


INTRODUCTION
Over the decades, the main energy sources that the electric power systems use are fossil fuels, including oil, coal and natural gas.Unfortunately, these fossil fuels are not renewable.Hence, the world is now in an energy crisis because these sources are being consumed rapidly.This emerging issue has raised the need for finding alternative energy resources that can sustain for a long period of time.These renewable energies include solar, wind, marine, hydropower, bio-energy and so on.They are also known as green energy due to the fact that they do not release carbon dioxide (CO2) into the atmosphere in the process of generating electrical power.
Over the past few years, applications of intelligent technologies on energy system have drawn a lot of attention among researchers, especially the term "Smart Grid" has become more and more popular.Smart gird has been considered as one of the green technologies because of its sustainability.It has recently emerged as the next generation of electric power delivery system which can solve multiple problems on ever increasing load demand.Smart grid is a modernized electricity distribution system that consists of diverse applications supported by different information and communication technologies.Since smart grid is a complex system which comprises of various intelligent devices coexisting on the same network (Ken, 2012), the communication built on top of smart grid infrastructure is normally a heterogeneous and hybrid system.This communication paradigm is needed to ensure a two-way flow of electricity, communication and information between the utility and household (Jayant et al., 2011;Agustin et al., 2011).Recently, a large number of researches have been carried out to evaluate a variety of communication technologies and architectures for Smart Grid (Thilo, 2011).In order to examine the feasibility and scalability of any system technology and architecture, a test bed for smart grid communication network to carry out evaluation is required.
Among the network research community, there always exists a need to evaluate new approaches, methods and systems by carrying test cases similar to those appear in real environment.Since the smart grid communication network is made up of a variety of communication devices, the network behavior is one of the main concerns when evaluating the network performance.Having an effective and coordinated testing procedure is therefore very important.Currently, there are a large number of testing tools and utilities available with different functions to serve different purposes.Some tools are utilized for laboratory testing where they can be used to analyze and evaluate the performance of new systems with the implementation of real networking hardware (Gang et al., 2010).Also, there are some other tools that can be used for emulation and simulation environments.Without the appropriate tools to carry out network testing, new systems might operate with unpredictable behavior and sometimes unacceptable performance when they are actually implemented in real environments.Especially in smart grid, multiple communication technologies creates major challenges for conventional testing approaches, as legacy testing schemes are designated only for a single communication technology or standard in the network.There is a lack of a holistic testing scheme for heterogeneous network that comprises of diverse information and communication technologies.Currently, there is no standard way to evaluate and benchmark the performance of smart grid communication system.Therefore, a testing framework is developed in this study to help network administrators analyze and evaluate the network performance of a heterogeneous and hybrid smart grid communication network.The scope of this project is to carry out the testing schemes for TNBR-Uniten smart grid testbed.

Smart grid test-bed:
Network infrastructure: Smart grid comprises of several systems, including SCADA (Supervisory Control and Data Acquisition Systems), EMS (Energy Management Systems), DCS (Distributed Control Systems), AMI (Advanced Metering Infrastructure), AMR (Automatic Meter Reading), etc. Devices that are used to support these systems consist of RTU (Remote Terminal Units), PLC (Power Line Communication) modems, smart meters, data concentrators and so on (Agustin et al., 2011).Particularly in TNBR-UNITEN smart grid testbed, there are totally 15 substations as shown in Fig. 1.
Communication architecture: Communication is a crucial component of smart grid as the next generation intelligent and reliable power delivery system.In smart grid communication network, no single technology can cater for the entire power delivery grid system.Therefore, both communication mediums, wired and wireless are proposed in this smart grid testbed.The communication technologies that support the network include Power Line Communication (PLC), unlicensed Radio Frequency (RF) and WiMax/4G.
PLC is one of the common technologies used to transmit data over wired network.It is catered for neighborhood area network and substation-to-substation.Even though the main function of PLC technology is to transmit electrical power, it can also be used to send data over the network.It operates by transmitting the modulated carrier signals over the existing power cables.Since the installation of dedicated network wires is not required, PLC provides an alternative broadband Fig. 1: Smart grid network diagram networking infrastructure (Wenye et al., 2011).PLC includes both narrowband and broadband technologies.Narrowband PLC operates at lower frequencies, lower data rates and lower range.Broadband PLC operates at higher frequencies, higher data rates and shorter range.Narrowband PLC is the communication technology normally used in smart grid.
Besides wired technologies, wireless can be used to transmit data signal.For example, RF mesh is suitable to use in smart metering applications.This is because it enables fast and reliable two-way communication without the need of wired backhaul.In addition, RF mesh technology allows multi-hop topologies where communication link is established from one node to another until it reaches the final destination.The ability to dynamically form ad-hoc communication network among intermediate nodes between the source and destination increases the communication range.The capabilities of self-healing lets RF mesh systems overcome various propagation problems which are fairly common in Neighbor Area Network (NAN).When certain links are blocked or down in the network, the system will find alternative paths to carry the communication through the mesh (Bill et al., 2010).
In addition to RF mesh, WiMax/4G is also another type of wireless technology that can be used for intersubstations and backbone.Like RF mesh, WiMax/4G uses the existing broadband infrastructure set up by wireless carriers without building a new one.The technology allows transmission of data with higher power and further distance.Therefore, it is suitable for long distance communication like in Wire Area Network (WAN) (Palak et al., 2010).Higher bandwidth and longer range of WiMax also provides backhaul connectivity for other wireless networks such as mesh RF and the network control centre.Moreover, it can support smart meter applications as well, as smart meters transmit data to the concentrators, which in turn are connected to WiMax base stations.

Network testing and base lining:
There are a number of researches that have been carried out in order to develop a framework for testing large scale test beds.Torens (2010) used a software environment to replace the lengthy operation of manual tests.Setting up this environment is necessary because deploying, controlling and monitoring a large number of nodes on a network are normally difficult as the network scale is growing.Thorough testing of such system is therefore more complicated and time-consuming.However, by developing a framework to automate the process of test cases, the deployment time for real test scenarios can be reduced significantly.
In addition, Christian et al. (2010) also used software tools to design simulation environment to test networks prior to their deployment.Evaluating the performance of large communication networks before deploying them in a real environment is very important.
During the implementation of such networks, certain performance parameters need to be considered in order to avoid bottlenecks or other network problems of the deployed systems.
Besides software simulation, modeling is another method to test the feasibility of a communication network prior to its implementation (Nita et al., 2012).By building a base lining model, the capacity of the network can be estimated.Moreover, the model also identifies applications bottlenecks to help network administrators locate possible congestion points and have enough information to analyze and evaluate the network performance.
In any communication system, it is important for network administrators to identify unexpected behaviors, predict possible problems that might occur in the network and quickly response to secure the problems.This is especially true in a large-scale communication network where a vast number of applications and technologies are implemented.In additions, performance evaluation of such network also involves real-time analysis of massive amount of data, which continuously generated from those devices that make up the network infrastructure.Therefore, network base lining is a critical step in network testing to detect unusual behaviors, anticipate future problems and prevent them from happening.Base lining is a process of establishing performance patterns for a communication network that is considered as normal or expected at all time (Robert, 2012).Base lining is used to determine the normal ranges of network behavior, identify abnormal or unexpected traffic patterns, predict and avoid possible problems and help network administrators optimize the network capacity in the future.Unusual traffic patterns are normally identified by behaviors that are significantly different from the expected range of the base lining function.These abnormal behaviors can be good indicators of hardware malfunction, connection failure or link breakdown.
In any network evaluation and management, a network manager must baseline a network in order to understand how the network behaves under test conditions.Without it, network administrators will not have the information needed to analyze and evaluate the network.A baseline is a collection of statistical measurements taken over a certain period of time to illustrate network performance (Vlatko, 2012).It records the measurement results and uses them to describe normal operating conditions of the network.In a typical communication system, some devices or services may operate in normal and desirable states, while others might show some problems that are not supposed to happen.However, with the help of baselining, potential network problems will be uncovered.By observing successive baselines taken frequently over a long period of time, network administrators can discover the trends and base on them to plan for the future in terms of network capacity growth.Therefore, baselining is used to identify the normal network conditions and detect arising problems, recognize changes and trends, anticipate and resolve problems to improve network performance.
Network baselining consists of three main steps, namely collecting data, generating statistics and analyzing the recorded outcome.First of all, before collecting data, a sample period must be chosen.A sample period is the total period of time over which baseline measurements are made and a sample interval is the period of time over which each individual statistics is sampled.Small sample intervals and short measurement periods are suitable when instantaneous characterization of network condition is required.In contrast, longer sample intervals and measurement periods are appropriate for baselining or attaining an over view of the network condition.The data is collected using a protocol analyzer during a fixed period of time, at similar time periods and at regular intervals.For instance, 1 sec samples are taken for periods up to 2 h, 1 min samples for periods up to 24 h, or 10 min or 1 hour samples, for periods more than two days (Vlatko, 2012).These samples are needed to provide enough information for network administrators to baseline and benchmark the network.
Once the data has been collected, it needs to be presented and analyzed.By analyzing the obtained data, network manager can identify abnormal traffic patterns and detect any significant changes in the network such as high levels of network utilization, low average data packet size, or high level of error frames.Thereby, they can understand how the network operates and anticipate future changes in the network behavior before the real problems actually happen.
The baseline is established by developing a testing framework for the testbed environment.The purpose of developing a performance testing framework is to simplify and automate network performance evaluation (Catalin et al., 2004).The framework describes a number of tests to carry out, a list of devices to use and a collection of tools to conduct the testing.This framework is also used to test a certain communication link or service in a network, measure several parameters and generate performance statistics.The collected data provides information on the impact of measurement metrics on the network performance.These performance metrics include response time, network throughput and the size of transmitted data.
The testing framework proposed in this study combines various testing techniques available for different communication mediums and technologies and compares which technique is suitable for heterogeneous and hybrid smart grid communication network.Several tests are carried out to measure a variety of network parameters and the most practical parameters are selected to evaluate the network performance.

Communication technology:
There are a variety of technologies used for communications, but a single technology cannot suit all the applications in smart grid.Therefore, a set of technologies must be used to support these applications.Each power system application requires a certain communication technology and a specific communication technology needs a particular testing technique to evaluate its performance.
The communication network in this Smart Grid testbed is supported by three main technologies, including WiMax, Mesh RF and PLC.WiMax is used for inter-substations and backbone and is catered for Wide Area Network (WAN).Mesh RF is another wireless technology that supports communication among substations.Last but not least, PLC is used for neighborhood area network (NAN) and smart meter applications.
PLC is one the communication technologies for wired network in smart grid.It is normally utilized for transmitting electrical power over the grid, but it can also be used to transmit data.As suggested by Drosopoulos et al. (2007), the performance of PLC modems can be evaluated by placing them in various locations within the power distribution network and carrying out a variety of tests under normal and severe conditions.The average throughput was measured and the performance limitations were determined.
Besides wired network, wireless is another part of smart grid communication system.As Muhammad and Jiwa (2009) mentioned, the performance of an ad hoc wireless network can be analyzed and evaluated by determining the impact of packet size, end-to-end delay, through put and packet loss on the communication network.A number of ping tests were performed and their statistics were recorded for analysis and evaluation.The statistics showed the average, minimum and maximum RTT as well as packet loss information.
WiMax is another wireless technologies used in smart grid.Faquir et al. (2007) set up an experimental WiMax testbed, analyzed and compared the performance of a WiMax link under various load and traffic scenarios.A series of stress tests and experiments were performed in the uplink and downlink directions.These tests were carried out for different service and traffic types and at different distances from the base station.
In most of these researches, different testing procedures have been carried out for different communication mediums (wired, wireless) or different technologies (PLC, WiMax).There is no standard performance baseline for testing the heterogeneous and hybrid communication network in smart grid.Therefore, the testing scheme proposed in this study is aimed to unify different testing methods and software tools, as well as to provide a common platform for network performance evaluation based on measurement parameters.

Network evaluation criteria:
The performance of a communication network is determined by a number of parameters.According to RFC 2544 standard, these parameters can be evaluated by conducting various tests.RFC 2544 is a methodology to benchmark network devices.It describes a series of tests, including frame loss, latency, throughput, burstability, system reset and system recovery (Bradner and Mc Quaid, 1999).Among these tests, frame loss, latency and throughput are the most suitable parameters to evaluate the performance of smart grid communication system because of their indicative capabilities.Frame loss defines the percentage of frames that were not forwarded over the total number of transmitted frames.Frames are transferred over the network by various devices under steady state (constant) loads.Frame loss measurement is useful for providing information about the performance of network device in an overload scenario, as it indicates how a device would perform under various network conditions.Latency test determines the round-trip time for transmitting the test frames across the network and receiving them back to the original port.Throughput test measures the maximum rate at which frames are transferred without dropping by the device or system under test (DUT/SUT).This measurement can be used to calculate the available bandwidth of the Ethernet virtual connection.With these tests, RFC 2544 can be used as a standard for carrying test cases to measure the performance of communication system (Mirza et al., 2011).For example, Mirza et al. (2011) developed an application to analyze the functionality of TCP stack.A series of test cases were performed including throughput, latency and packet loss.The results obtained from these tests were then used to test the Ethernet functionality.Muhammad and Jiwa (2009) performed several tests to examine the effect of varying packet size, endto-end delay, communication throughput and packet loss on the performance of an ad hoc wireless network.Endto-end delay is the time taken for a packet to be transmitted across a network from source to destination.It includes transmission delay, propagation delay, processing delay and queuing delay experienced by every node in the mesh network.Beside end-to-end delay, throughput is another factor that can affect the performance of an ad hoc wireless network.The average throughput is calculated by taking the ratio of packet size and RTT.Last but not least, packet loss is also an important factor that needs to be considered in a wireless network.By monitoring the obtained results, the wireless connections among the nodes in the network were verified, where a good connection is the one with no or few packet loss.
In other researches, throughput and latency are also used to perform the quantitative evaluation in smart grid.An emulated smart grid testbed is implemented by Kush et al. (2010) in order to provide a framework for testing network requirements.In this smart grid test bed, latency and throughput are used as evaluation criteria to assess the performance of communication system and experiments are carried out to evaluate the test bed according to these defined criteria.
Similarly, using network emulation, a small-scale smart grid communication test bed is set up by Wenye et al. (2011) to evaluate the delay performance.Communication delay or latency is considered as one of the most critical network performance metrics.Particularly in smart grid, communication delay is defined as the time interval for an end-to-end process flow to transferring a message from the source to the destination system.In the network topologies of the traditional power system communication, delay is not considered as the most important performance parameter.Hence, they may not be able to meet all the requirements set by smart grid communication network.However, smart grid comprises of a large number of devices with different information and communication technologies.Due to the heterogeneous and hybrid nature of smart grid, different network topologies are deployed.Various types of networks are used to provide communication mediums to different parts of the grid.As the message travels through different segments of the network, they experience diverse types of delay, including data acquisition delay, packet processing delay, packet transmission delay, medium access delay and event responding delay.
In general, smart grid communication network can be evaluated by several parameters called performance metrics.The most common metrics in assessing the network performance are availability, packet loss, latency (or delay) and throughput (Drysdale et al., 2000;Paul, 2012).But not all of these parameters were considered in previous researches.For instance, Bradner and McQuaid (1999) and Muhammad and Jiwa (2009) did not discuss the availability in their network testing.Latency and throughput were the only two evaluation criteria mentioned by Kush et al. (2010).Plus, only delay was considered by Wenye et al. (2011) to evaluate the emulated smart grid testbed without considering the other three parameters.However, the testing framework proposed in this study considers all four performance metrics to provide the most indicative characteristics in analyzing and evaluating the smart grid communication system.These performance parameters are measured using a selective collection of software tools.

Network testing tools:
Nagios: The main challenge in managing smart grid communication network comes from the fact that different vendors provide different system to manage their own network.In this situation, network managers are required to deploy various management platforms in order to manage the whole system.In order to overcome the complexity of heterogeneous and hybrid smart grid communication system, it is vital for network administrators to select an appropriate management platform to centralize and integrate multiple management tools deployed on the same communication system.Nagios is free and open source software designed for Linux that can overcome this challenge.Nagios provides a graphical user interface, allowing network administrators to monitor various devices on the network, such as routers, switches, gateways and computers (Takao et al., 2007).In smart grid communication network, the availability of numerous devices can be monitored by Nagios, a centralized management system which provides a solution for network operators to handle multi-vendors equipment (Ahmed and Moussa, 2012).
Nagios is chosen due to the fact that it functions are suitable for monitoring the status of numerous devices in the smart grid communication system.Basic features of Nagios allow it to provide more detailed information about the availability of a selected host compared to other network management software tools.Nagios offers different types of layouts for network map.By changing a specific value in the configuration file, network administrators can modify the network layout automatically without manually drag and drop each host icon on the status map.From Nagios network map, the hosts' status (up, down, or unreachable) can be easily viewed.By arranging the hosts in a hierarchical order using 'Parent' configuration, network administrators are able to view how hosts are connected to each other and the relationship among them.This 'Parent' function also helps to distinguish between hosts that are down and those that are unreachable.In Nagios, a host's status can be divided into three different states: up, down and unreachable.A host is defined as down or unreachable if its preliminary host state is down.The difference between the two lies on their parent host state.If at least one parent is up, the host is classified as down.Otherwise, the host is considered as unreachable if all parents are either down or unreachable.The ability to differentiate between down and unreachable host states is a distinctive function of Nagios that other available software tools do not offer.This distinction is vital because it helps network managers to determine the root cause of network outages if they happen.
In addition, the status of each host on Nagios can be further categorized into a wide range of different states, such as OK, Warning, Critical and Unknown.Nagios allows network administrators to set parameters to specify these states in its configuration.For instance, a Warning or Critical state will be displayed if the roundtrip-time and percentage of packet loss for a Ping service of a particular host are larger than a userpredefined value.These state notifications can be recorded in Nagios availability report for future analysis and evaluation.In this study, a script is written in Linux to generate host availability report automatically at a specific time of the day and send to network managers via email.The availability report can be taken at different periods of time, such as today, yesterday, last 24 hours, last 7 days, last month, last year and so on.
In conclusion, Nagios proves to be particularly useful for monitoring the availability states of various devices in communication networks.It serves the monitoring purpose, especially in smart grid context, by providing necessary information to perform availability test.
Ping: Ping is one of the most common network testing tools.It is available and supported by most communication devices and operating systems.Moreover, it does not require extra installation or configuration.This is extremely important in heterogeneous networks where diverse communication equipments are deployed at different locations and network administrators do not prefer having additional software installed at these sites.Plus, ping is a very simple yet effective connectivity testing tool.In previous researches, Ping is most widely-used to measure packet loss and communication latency.
As suggested by Kush et al. (2010), ping provides round trip time (RTT) which can be utilized to troubleshoot network latency.An emulated network was deployed with a single node connecting to the host system.Constant ping tests are conducted to check the connectivity between the hosts and latency tests were performed by sending 30 Ping requests to the single node from the host system.The Ping utility sent an ICMP echo request and waited for an ICMP echo response.The RTTs, including the maximum, minimum and average RTT, were measured and recorded.The experiment was repeated ten times to have a steady latency result.
In addition, in order to analyze and investigate the performance of an ad hoc wireless network, Muhammad and Jiwa (2009) proposed Ping as an effective networktesting tool.It was used to determine the influence of packet size, end-to-end delay, communication throughput and packet loss on the performance of ad hoc wireless network.To evaluate these parameters, 100 ping requests were sent, with each ping transmitting an ICMP echo-request and waiting for an ICMP echoresponse.Each ICMP packet was transmitted from source to destination node using a default size (64 bytes).One ping request was sent every second and the ping output displayed the packet size, sequence number and Round-Trip-Time (RTT).The experiment was repeated with increasing packet size to investigate the impacts of packet size and route length on end-to-end delay, throughput and packet loss.
Due to the features that Ping offers, a script is written in Windows based on ping to carry out packet loss and latency tests as part the testing scheme for smart grid.The script automatically generates ping requests every specific period of time and runs for 24 hours.The obtained results later can be used to evaluate the performance and determine the characteristics of smart grid communication system.

Iperf and bing:
Iperf is a common network utility which is used to carry out throughput testing by measuring the maximum TCP and UDP bandwidth.It has been used in previous researches (Faquir et al., 2007) to generate traffic to analyze the performance of a WiMax testbed.In the experiment, the link for both TCP and UDP data was tested where traffic was transmitted at various distances from the base station.The traffic was generated for both uplink and downlink directions.The bandwidth, delay jitter and datagram loss were reported.Once captured at the receiver, the generated data was evaluated and analyzed.By using this method, the ability, reliability and the robustness of the WiMax link were tested.Iperf tests were repeated thirty times to obtain normally distributed output and the results were used to set a limitation of throughput rate for applications in the smart grid testbed.
Iperf is chosen as a software tool to measure throughput for Mesh RF technology in smart grid testbed because of several reasons.First of all, it is a built-in software tool that is installed in the communication devices provided by the vendor.It is compatible with the device settings and configuration.Hence, adding another software tool while the existing tool with similar functions is still in use is redundant.In case of a grid, network administrators do not have control over all grid sites and therefore cannot install additional software or services.By utilizing the preinstalled software already available in the equipment, network managers can ease the process of installation and configuration.In addition, adding new software tool might require modification of the equipment itself.This probably will change the internal settings of devices which can cause unpredictable behavior.In some cases, network administrators are also not allowed to change the setting initially configured by manufactures.As a result, Iperf is used to measure link throughput among mesh RF devices in smart grid communication system.
Besides Iperf, Bing is another throughput measuring tool.Since Bing can measure point-to-point bandwidth, it is used to conduct throughput test for WiMax technology.Unlike Iperf, Bing does not require client-server installation.It can be easily installed on Windows and Linux operating systems.After being set up at the central server, Bing can measure point-to-point throughput of a selected link.This is particularly useful when installing a software tool at one end of the link is not applicable.
Traffic generator: Smart grid is a complex network which implements diverse applications, information and communication technologies.Due to the large scale nature of smart grid, traffic will be generated by numerous applications and passed through various types of networks.A mixture of both real-time and non-realtime traffic will be generated and distributed across different parts of smart grid (Zhong et al., 2010).In this study, a traffic generator is implemented as part of the testing scheme, which has a function of generating data packets according to traffic that might occur in the smart grid communication system.The function of this simulator is to create the traffic congestion similar to the amount of data generated from various devices and components in heterogeneous and hybrid smart grid communication network.Traffic generated from these devices can cause certain problems to the network.Therefore, a traffic generator is required to carry out stress test in order to predict the network behavior and evaluate its performance.
Stress tests are performed to test outside the boundary of what the network can handle.A single board computer is used to inject traffic into smart grid communication network where the bottle necks are located.In the smart grid testbed, bottle necks are places where network switch is put together with a number of devices such as PLC modem, RTU, computer, data concentrator and smart meters.
The traffic generator in smart grid is implemented based on a single board computer running on Debian Linux as an operating system.It runs from Linux kernel space on a terminal command line to have a maximum level of traffic.The fundamental idea of implementing a new traffic generator arises from the lack of the available ones.For example, some traffic generators are designed only for Windows applications, whereas the implemented traffic generator runs on Linux.In addition, most of the existing tools have a Graphical User Interface (GUI) which occupies a lot of memory storage.Meanwhile, the traffic generator running on Soekris does not need a GUI, which can reserve more space in the memory and save execution time when generating traffic.The main purpose of implementing the traffic generator in the kernel space is to bypass the overhead of a user-application when generating packets.This allows the network traffic generator to achieve the best performance and utilize maximum network bandwidth.

RESULTS AND DISCUSSION
The testing framework used to evaluate the performance of smart grid communication system is presented in Table 1.The testing procedure was carried out to evaluate the availability, packet loss, latency and throughput of each communication technology in Smart Grid testbed.
Availability: Each device monitored by Nagios is assigned a fixed IP address and is identified as a host.The host's status can be viewed from Nagios status map as shown in Fig. 2. As the host is up, the section surrounding that particular host is green.Otherwise, it is in red.All hosts are connected to each other according to a hierarchy order from the inside out.The host that is closer to the centre of Nagios status map acts as parent of those who are further.Each host's availability is monitored over a month and the obtained results are recorded.
Ping is one of the services defined for each host.By sending an ICMP echo-request every minute, a host is considered as up if Nagios receives a successful ICMP echo-response.The results of ping are displayed on Current Network Status page (Fig. 3).Ping shows an OK status if the round-trip time and packet loss are less than a pre-defined number.Whereas, a Critical status is shown when the round-trip time and packet loss exceed this specified value.
Table 2 shows the availability of several devices supported by three communication technologies in smart grid testbed.Under stable condition, the availability meets the standard requirement for smart grid as indicated by the US Department of Energy (Communication requirements of Smart Grid Technologies.Retrieved from: http://energy.gov/gc/ downloads/ communications -requirements-smart-gridtechnologies).However, under worst case scenarios, the availability is out of the requirement range.This happens due to a number of factors.First of all, WiMax is used as the backhaul technology, thus the availability of all devices in the network highly depends on the quality of the wireless service.In addition, as the smart grid testbed is at the initial stage, some devices are still under implementation.When any modification has been made such as changing of IP address, their availability results will be affected.Lastly, low availability is also due to device failure or power trip at the substations.
Packet loss and latency: Ping, a common network testing tool, is used to measure packet loss and latency.In order to get the average values, ping script is written to send 30 ping requests every 15 min from a control centre to each device in the network.The script runs on Windows for 24 h and the ping statistics are recorded in a text file, including packet loss and the average roundtrip time.The specifications and procedure for ping test are summarized in Table 3 and Fig. 4, respectively.
The percentage of packet loss for different communication technologies is displayed in Table 4. PLC shows the lowest level of packet loss with the average percentage of 0.69%.WiMax appears to be the least reliable technology among the three because of its high packet loss (1.97%).However, the average percentage of packet loss for all technologies is still within the acceptable range (0-2.5%).
Table 5 shows the communication latency of three different technologies in smart grid.WiMax shows the     Throughput: Throughput is calculated depending on the available technologies.For instance, Bing is used to measure the throughput of WiMax links.Iperf is utilized as a bandwidth measuring tool for Mesh RF and customized software is used to calculate the throughput for PLC modems.
In order to measure WiMax throughput, a system is set up as shown in Fig. 5.At the control centre, a desktop acts as a server and communicates with three clients at three different locations throughput WiMax links.The clients can only communication with the server, but not with each other.Bing is installed in the desktop at the control centre and used a tool to measure bandwidth.At the terminal, a bing command is executed and the average throughput are calculated based on the obtained results.
In Mesh RF network, the link throughput is measured by logging into each communication device using PuTTY software and executing commands at the terminal.Iperf, a common bandwidth tester, is used to determine the throughput for several links between substations as shown in Fig. 6.The Iperf test for each link is repeated for 25 times to get the average value.Throughput for PLC modems is calculated by logging into each device using Telnet and executing commands at the terminal to measure the transmitting and receiving data rate.PLC modems are categorized into two different types, namely master and slave gateway.The master modem has an ability to communicate with its slave units.However, the slave modems can only communicate with their master, not with each other.The throughput between two modems is determined by logging into the master unit and measure the transmitting data rate to its slave device (Fig. 7).The throughput test for each link is repeated for 25 times and the average value is computed.
The practical throughput of three technologies is displayed in Table 6.WiMax throughput appears to be the one closest to the theoretical value, while PLC shows half of the maximum specified bandwidth and Mesh RF occupies only 39.29% of the theoretical amount.
Stress test: Last but not least, a series of stress tests are performed to test outside the boundary of what the network can handle.The purpose of putting the traffic generator at these locations is to generate one-way packet traffic from source to destination node in a network and carry out stress tests to analyze the network behavior under heavy load conditions (Fig. 8).The results obtain from these tests help network administrators to evaluate the performance of a communication network, identify the locations where possible problems might occur and provide a solution to secure these problems if they happen.
To generate data packets from the kernel space, a module is built and when it is loaded, the board will start to generate traffic to the network.In order to be installed The obtained results of stress tests are in Table 7.At 28 Mbps, traffic does not have significant impact on the network if the duration of traffic flow is less than 450 ms.A significant increase is shown in communication latency when the traffic duration is within the range from 450ms to 1s.With the traffic burst lasting more than 1s, the network experiences 100% of packet loss.
The results obtained from the stress test show that the maximum capacity of traffic in the communication network occupies 50% of the rated bandwidth.The maximum traffic burst allowable is 27 Mbps for 400 ms.This ensures critical traffic such as SCADA control messages can be issued without interruption.The results also indicate that the current network capacity is able to support around 25,000 meters and 200 RTUs.

CONCLUSION
A performance testing framework is presented in this study in order to unify different testing methods and software tools, as well as to provide a holistic testing scheme for heterogeneous and hybrid smart grid communication network.The performance of three different technologies in smart grid, including WiMax, Mesh RF and PLC, is analyzed and evaluated based on network performance metrics such as availability, packet loss, latency and throughput.Stress tests are also carried out to determine the operating limit and capacity of traffic in the communication network, the parameters for setting the traffic shaper and the scalability of smart grid communication network.The testing framework can be further enhanced by developing a performance index to benchmark the network performance of various smart grid communication systems.

Fig. 8 :
Fig. 8: Stress test in the single board computer, the program to generate traffic must be light enough, yet at the same time has the basic function of a traffic generating tool.Each single board computer is able to generate Ethernet II packets with the maximum traffic of 28 Mbps.The delay between each generating packet is set to zero and the packet size is 1518 bytes (maximum Ethernet frame size).Once the traffic is generated, packet loss and latency are measured to analyze and evaluate the network performance under stressed conditions.The obtained results of stress tests are in Table7.At 28 Mbps, traffic does not have significant impact on the network if the duration of traffic flow is less than 450 ms.A significant increase is shown in communication latency when the traffic duration is within the range from 450ms to 1s.With the traffic burst lasting more than 1s, the network experiences 100% of packet loss.The results obtained from the stress test show that the maximum capacity of traffic in the communication network occupies 50% of the rated bandwidth.The maximum traffic burst allowable is 27 Mbps for 400 ms.This ensures critical traffic such as SCADA control messages can be issued without interruption.The results also indicate that the current network capacity is able to support around 25,000 meters and 200 RTUs.

Table 1 :
Performance testing framework

Table 7 :
Stress test