Cheshire Cat Computing

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Hi Steve,

I too had the same issue – excessive CPU utilization on my Win2K3 boxes.
I also found a way to fix the problem.

First, I have to say that all versions from 1.8.3, 1.7.2, 1.7.1 down to 1.7.0b that I tested are affected, so it is not a version specific problem. Also, it is not related to filter definitions, their number or complexity. As a matter of fact it is not really a nagevlog problem at all.

What I first found out on the systems that were affected in my case, is that the Security Event Log was constantly filled by hundreds of events of two types (copies are at the bottom of the post) and they are the reason for the high CPU utilization.

These two events were created for three reasons:

- In the ‘Local Security Policy’\‘Audit Policy’\‘Audit Object Access’ I configured both Success and Failure. Default setting is to audit Failures only. (this chnage is recommended security setting for all Windows servers that are Internet facing, or need to be highly secured)

- As a part of Windows installation, auditing is configured for the registry key \REGISTRY\MACHINE\SYSTEM\ControlSet001\Services\Eventlog\Security and subkeys

- nagevlog keeps accessing this key excessively

So to fix the problem you can either disable auditing for Successful Object Access in the Local Security Policy (not in my case), or disable auditing for the registry key (this should not be done ), which I temporarily did on my staging systems. This instantly reduced CPU utilization from 60-70% to 3-4%.

So the only question is - Why is this key so excessively accessed and probably modified – can that be changed in the next release?

Thanks


=======================================================================
Event Type: Success Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 13/12/2007
Time: 20:50:21
User: NT AUTHORITY\SYSTEM
Computer: XXXXXXXXXX
Description:
Object Open:
Object Server: Security
Object Type: Key
Object Name: \REGISTRY\MACHINE\SYSTEM\ControlSet001\Services\Eventlog\Security\Security
Handle ID: 180
Operation ID: {0,2288220020}
Process ID: 7068
Image File Name: C:\Program Files\Monitoring\nagevlog.exe
Primary User Name: XXXXXXXXXX$
Primary Domain: XXXXXXXX
Primary Logon ID: (0x0,0x3E7)
Client User Name: -
Client Domain: -
Client Logon ID: -
Accesses: DELETE
READ_CONTROL
WRITE_DAC
WRITE_OWNER
Query key value
Set key value
Create sub-key
Enumerate sub-keys
Notify about changes to keys
Create Link

Privileges: -
Restricted Sid Count: 0
Access Mask: 0xF003F


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

And

==================================================================
Event Type: Success Audit
Event Source: Security
Event Category: Object Access
Event ID: 562
Date: 13/12/2007
Time: 20:50:21
User: NT AUTHORITY\SYSTEM
Computer: XXXXXXXXXX
Description:
Handle Closed:
Object Server: Security
Handle ID: 180
Process ID: 7068
Image File Name: C:\Program Files\Monitoring\nagevlog.exe


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

==========================================================================

Hi Steve,

Actually changing the Audit settings on the registry key wasn't very smart. I'll have to undo that. That is set by default in every Windows installation. That is not an option for me and it shouldn't be for anyone.

Can you see why is nagevlog accessing this key so much and if that can be changed.

\HKLM\SYSTEM\ControlSet001\Services\EventLog\Security\Security

Meanwhile, I'll just have to live with high CPU use...

Thanks

The code which does this frequent accessing is part of the original Syslog_NT code which I build the app on - its buried deep within.

It seems that, every polling cycle, the system re-reads the list of event logs defined to work out what it needs to poll.

Probably, what I should do is get the program to cache its registry lookups where possible in order to avoid thisproblem. However, that will be a big change and I'll need to pull aparts the bits of the program I took from the NT_Syslog code (IE, the bits which poll and extract eventlog entries). This is not a quick task.

Since more and more people are likely to be using 'success' auditing (I can see why you need to audit successful writes, but audit all successful reads as well?) this will need to be fixed. Maybe I can find a spare minute in my busy time over the next 6 weeks before baby#2 is born...

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Seems there might be a bug in the code, and it scans the new entries for ALL event logs, even if there are no filters pertaining to that log. The filters should be pre-scanned to see which eventlogs need to be processed, but they aren't.

I've not been able to work on this for a long time due to family and work pressures. I'm hoping that I'll be able to find some time to look into this, the registry caching, and the unicode support, before too long...

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

OK. You might want to consider joining the mailing list
http://steveshipway.org/mailman/listinf ... ipway.org/
since I will post to the list and the forum when a new version is (finally) available.

At the moment, I'm working with Tobi on the RRDTool/MRTG code to add some features we need here. After that's put to bed I should have some more time...

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11