As the third defensive line for preventing exploitation of application vulnerabilities, SMS (the System-call Monitoring System) is an intercepting system that used to detect and prevent exploitations by monitoring an application’s runtime behavior. An important nonfunctional characteristic of SMS is that it does not require access to application source code. This empowers administrators to correct security vulnerabilities in existing applications even if they modify source code. Therefore, this eliminates reliance on software vendors to produce and distribute.
Detecting Exploits
SMS is experimented with behavioral specifications, a general form of misuse detection that implementing anomaly detection. Anomaly detectors report both known and unknown exploits by comparing the actual system-call signature with the normal signature when the application executes in the field. With misuse detectors, a programmer specifies the precise signatures of unknown intrusions, and when the application executes, it compares the actual system to the misuse signatures and reports matches as intrusions. They report exploits for which they are programmed. Thus, behavioral specifications of SMS achieve high accuracy in detecting both known and unknown attacks with low false-positive rates.
Preventing Damage
In contrast, SMS intercepts system calls as they are requested, so detection and remediation can occur before damage occurs. Real-time system-call interception must efficiently check signatures to prevent unacceptable overhead. SMS includes an event-specification language, called ASL (Abstract Specification Language), for specifying signatures in terms of system calls and their arguments. The defense refinement is aided as the system discovers new exploits and prevents damage by isolating the compromised application. Isolation places compromised applications in environments that appear legitimate butt hat actually contain no sensitive data. It also includes techniques such as inserting delays in system-call execution to reduce CPU use and denying some system calls with plausible, but false, return values.
Detection Accuracy
Real-time detectors have a fundamental accuracy advantage than after-the-fact detection as they can examine all aspects of instantaneous system state to improve detection accuracy whilst after-the-fact detectors can only look at the logged system state subset. As a real-time detector, SMS concepts apply in general to any operating system.
Prototype Design
To experiment with the concept of system-call interception for security enhancement, they made design choices that support rapid prototyping over other characteristics. They use C++ ass the surface between the specification environment and the runtime environment, decoupling the two major components of the prototype and supporting concurrent development by the Telcordia and SUNY teams. Programs written in ASL compile into C++ programs that are then compiled and link-edited to produce a runtime system-call interceptor. Two mechanisms are implemented for intercepting system calls at runtime. The earlier method modified the source code of libc, which is the library most applications use to access system calls. The latter method uses kernel-level interception by placing hooks into the kernel.
Effectiveness
SMS enables the easy construction of effective defenses for standard servers such ass ftpd and telnetd. It can remediate legitimate design decisions that went against what most end users wanted and also can change an application’s security policy, bringing it in line with the end user requirements.
Performance
System-call interception imposes a burden on all processes. It is because all system calls are intercepted regardless of whether or not each process has an explicit defense. The performance penalty of system-call interception depends on two major factors. They are the interception cost and the cost of executing defensive and reaction programs. What they consider about only covers the interception cost, which is constant per system call.