VMCD.ORG

Focus on database architecture

Logon Storm

Posted by admin on November 16th, 2012

A Logon storm is a sudden spike in the number of client connection requests. Logon storms can occur due to a variety of factors. They could be malicious like a DoS attack. Or they could occur due to administrative actions – such as a middle tier coming online.
The process of starting a database session has inherent CPU costs associated with it – from the creation of a new OS process to attaching to the SGA. Every system has a limit above which the spawning of new database sessions can starve existing
sessions of CPU resources. This could result in many unexpected consequences including application timeouts, increased response times, un-responsiveness and other cascading effects.

The problem

errlog:

Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
 
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:386)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1054)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:308)

[grid@dcb-srv-0173 trace]$ fgrep “07-NOV-2012 16:” listener.log |fgrep “establish” |awk ‘{print $1 ” ” $2}’ |awk -F: ‘{print $1 “:” $2 }’ |sort |uniq -c
1396 07-NOV-2012 16:00
394 07-NOV-2012 16:01
218 07-NOV-2012 16:02
285 07-NOV-2012 16:03
364 07-NOV-2012 16:04
414 07-NOV-2012 16:05
436 07-NOV-2012 16:06
349 07-NOV-2012 16:07
290 07-NOV-2012 16:08
367 07-NOV-2012 16:09
551 07-NOV-2012 16:10
579 07-NOV-2012 16:11
502 07-NOV-2012 16:12
501 07-NOV-2012 16:13
468 07-NOV-2012 16:14
461 07-NOV-2012 16:15
685 07-NOV-2012 16:16
451 07-NOV-2012 16:17
448 07-NOV-2012 16:18
298 07-NOV-2012 16:19
358 07-NOV-2012 16:20
321 07-NOV-2012 16:21
451 07-NOV-2012 16:22
303 07-NOV-2012 16:23
475 07-NOV-2012 16:24
399 07-NOV-2012 16:25
412 07-NOV-2012 16:26
307 07-NOV-2012 16:27
358 07-NOV-2012 16:28
444 07-NOV-2012 16:29
510 07-NOV-2012 16:30
420 07-NOV-2012 16:31
461 07-NOV-2012 16:32
408 07-NOV-2012 16:33
299 07-NOV-2012 16:34
425 07-NOV-2012 16:35
350 07-NOV-2012 16:36
545 07-NOV-2012 16:37
324 07-NOV-2012 16:38
389 07-NOV-2012 16:39
339 07-NOV-2012 16:40
385 07-NOV-2012 16:41
455 07-NOV-2012 16:42
366 07-NOV-2012 16:43
304 07-NOV-2012 16:44
319 07-NOV-2012 16:45
465 07-NOV-2012 16:46
384 07-NOV-2012 16:47
480 07-NOV-2012 16:48
362 07-NOV-2012 16:49
563 07-NOV-2012 16:50
484 07-NOV-2012 16:51
402 07-NOV-2012 16:52
350 07-NOV-2012 16:53
339 07-NOV-2012 16:54
454 07-NOV-2012 16:55
451 07-NOV-2012 16:56
409 07-NOV-2012 16:57
444 07-NOV-2012 16:58
452 07-NOV-2012 16:59

16:00瞬间涌入了大量连接,由于这台机器的配置十分强大导致web服务器集体中断了1-2s.并没有在DB服务器端引起大的负载。

[root@dcb-srv-0173 sa]# sar -f /var/log/sa/sa07 -s 15:50:00 -e 16:05:00
Linux 2.6.18-308.el5 (dcb-srv-0173) 11/07/2012

可以明显看到16:00的波谷

Some relative posts:

cursor: pin S waits, sporadic CPU spikes
A close encounter with real world performance issues

Solution:

Using listener RATE_LIMIT parameter
This parameter indicates that a particular listening endpoint is rate limited. 
This is specified in the ADDRESS section of the listener endpoint 
configuration. For example, 
LISTENER=(ADDRESS= 
(PROTOCOL=tcp) 
(HOST=sales-server)(PORT=1521) 
(RATE_LIMIT=yes)) 
This parameter can be configured in two ways: 
1. When the parameter is set to “yes”, the endpoint is included in 
the enforcement of a listener-wide connection rate. This has to 
be used in conjunction with the CONNECTION_RATE_LISTENER
parameter.  
2. When a value greater than 0 is specified, the rate limit is enforced 
at the endpoint level.  
eg:
LISTENER=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=sales)(PORT=1521)(RATE_LIMIT=3))
(ADDRESS=(PROTOCOL=tcp)(HOST=lmgmt)(PORT=1522)(RATE_LIMIT=no)))
Use dedicated for:
 High-performance connections
 Active, long-running, data transfer intensive operations
 
Use shared for:
 Sessions that may be idle for some time
 Clients that frequently connect and disconnect
 
Use DRCP (11g):
 When you have thousands of clients which need access to a database server session for a short period of time
 Applications mostly use same database credentials, and have 
identical session settings
PHP (OCI8 extension), Python (cx_Oracle), Perl (DBI)

同理以上两种对于logon storm 有限制作用

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>