OWASP ModSecurity Core Rule Set (CRS) Project
Ryan BarnettOWASP CRS Project LeaderSenior Security Researcher
Ryan Barnett - Background•Trustwave
•SpiderLabs Research Team
•Web application firewall research/development
•ModSecurity Community Manager
•Interface with the community on public mail-list
•Steer the internal development of ModSecurity
•Author
•“Preventing Web Attacks with Apache”
Community Projects•Open Web Application Security Project (OWASP)
•Project Leader, ModSecurity Core Rule Set
•Project Contributor, OWASP Top 10
•Project Contributor, AppSensor
•Web Application Security Consortium (WASC)
•Project Leader, Web Hacking Incident Database
•Project Leader, Distributed Web Honeypots
•Project Contributor, Web Application Firewall Evaluation Criteria
•Project Contributor, Threat Classification
•The SANS Institute
•Courseware Developer/Instructor
•Project Contributor, CWE/SANS Top 25 Worst Programming Errors
Session Outline
• ModSecurity Quick Overview
• The Core Rule Set (CRS) Overview
• Basic Detection Categories
• Latest CRS Improvements
• CRS Demonstration Page
• Future Directions
ModSecurity Quick Overview
What is ModSecurity?
It is an open source web application firewall (WAF) module for Apache web servers www.modsecurity.org
Separate Rule and Audit Engines Allows full request/response HTTP logging capability
Deep understanding of HTTP and HTML Robust Parsing (form encoding, multipart, XML)
Event-based Rules Language Anti-Evasion Features (normalization functions)
Advanced Capabilities Transactional and Persistent Collections Content Injection Lua API
ModSecurity’s Rules Language Syntax
Tells ModSecurity where to look (such as ARGS,
ARGS_NAMES or COOKIES).
Tells ModSecurity where to look (such as ARGS,
ARGS_NAMES or COOKIES).
Tells ModSecurity how to process data (such
@rx, @pm or @gt).
Tells ModSecurity how to process data (such
@rx, @pm or @gt).
Tells ModSecurity what to do if a rule matches (such as deny, exec or
setvar).
Tells ModSecurity what to do if a rule matches (such as deny, exec or
setvar).
ModSecurity’s Apache Request Cycle Hooks
OWASP ModSecurity Core Rule Set (CRS) Overview
Project Info
http://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project
What is the Core Rule Set (CRS)?
A generic, plug-n-play set of WAF rules Choose your mode of operation
Standard vs. Anomaly Scoring
Detection Categories: Protocol Validation Malicious Client Identification Generic Attack Signatures Known Vulnerabilities Signatures Trojan/Backdoor Access Outbound Data Leakage Anti-Virus and DoS utility scripts
Initial Configuration
Once unpacked, edit the main config filemodsecurity_crs_10_config.conf
Customize the following items Mode of Detection – Standard vs. Anomaly Scoring Anomaly Scoring Severity Levels Enable/Disable Blocking Blocking Threshold Levels Paranoid Mode – aggressive inspection HTTP Policy Settings Choose where to log events (Apache error_log and/or
ModSecurity’s audit log)
Traditional Detection ModeSelf Contained Rules Concept
IDS/IPS mode with “self-contained” rules Like HTTP itself – the rules are stateless
No intelligence is shared between rules If a rule triggers, it will execute a disruptive/logging
action
Easier for the new user to understand Not optimal from a rules management
perspective (handling false positives/exceptions)
Not optimal from a security perspective Not every site has the same risk tolerance Lower severity alerts are largely ignored
Anomaly Scoring Detection ModeCollaborative Rules Concept
Advanced inspection/detection mode Delayed blocking
Rules set transactional variables (tx) to store temporary meta-data about the rule match
Rules also increase anomaly scores for both the attack category and global score
The anomaly score enforcement rules decide whether or not to deny transactions at the end of the inbound request phasemodsecurity_crs_49_inbound_blocking.conf
Anomaly Scoring - Debug Log View
Inspecting Anomaly Scores
Conditional Rules (Weak Sigs)
SQL Injection ExampleAggregate indicators to determine an
attackStrong indicators
Keywords such as: xp_cmdshell, varchar, Sequences such as: union …. select, select … top … 1Amount: script, cookie and document appear in the same input field
Weak indicators – meta-characters--, ;, ', …
CRS only applies weak signatures in the event a stronger signature has previously triggered
Conditional Rule Example
Event LoggingStandard vs. Correlated Events
Standard mode Rules log event data to both the Apache error_log
and the ModSecurity Audit log
Correlated mode Basic rules are considered reference events and do
not directly log to the Apache error_log Correlation rules in the logging phase analyze
inbound/outbound events and generate special events
modsecurity_crs_60_correlation.conf
Inbound/Outbound Correlation
Couple the inbound with the outbound for increased intelligence Was there an inbound attack? Was there an HTTP Status Code Error (4xx/5xx
level)? Was there an application information leak?
Correlation facilitates better incident response App error without inbound attack -> Contact Ops Inbound attack + outbound error -> Contact Security
Event Severity Ratings
Correlated Events 0: Emergency - is generated from correlation (inbound
attack + outbound leakage) 1: Alert - is generated from correlation (inbound attack +
outbound application level error)
Non-Correlated Events 2: Critical - highest severity level possible without
correlation. It is normally generated by the web attack rules (40 level files)
3: Error - is generated from outbound leakage rules (50 level files)
4: Warning - is generated by malicious client rules (35 level files)
5: Notice - is generated by the Protocol policy and anomaly files
6: Info - is generated by the search engine clients (55 marketing file)
Correlated Event Messages
Detection Mechanisms: Protocol Violations
Protocol vulnerabilities such as Response Splitting, Request Smuggling, Premature URL ending Content length only for non GET/HEAD methods Non ASCII characters or encoding in headers Valid use of headers (for example, content length is
numerical) Proxy Access modsecurity_crs_20_protocol_violations.conf
Attack requests are different due to automation Missing headers such as Host, Accept, User-Agent Host is an IP address (common worm propagation
method) modsecurity_crs_21_protocol_anomalies.conf
Detection Mechanisms: Protocol Policies
Policy is usually application specific Some restrictions can usually be applied generically White lists can be build for specific environments
Limitations on Sizes Request size, Upload size # of parameters, length of parameter modsecurity_crs_23_request_limits.conf
Items that can be allowed or restricted Methods - Allow or restrict WebDAV, block abused
methods such as CONNECT, TRACE or DEBUG File extensions – backup files, database files, ini files Content-Types (and to some extent other headers) Modsecurity_crs_30_http_policy.conf
Detection Mechanisms: Malicious Clients
Not aimed against targeted attacks, but against general malicious internet activity Offloads a lot of cyberspace junk & noise Effective against comment spam Reduce event count
Detection of Malicious Robots Unique request attributes: User-Agent header, URL,
Headers Black list of IP addresses Rate based detection Detection of security scanners Blocking can confuse security testing software
(WAFW00f) modsecurity_crs_35_bad_robots.conf
Detection Mechanisms: App Layer Attacks
Detect application level attacks such as those described in the OWASP top 10 SQL injection and blind SQL injection Cross site scripting (XSS) OS command injection and remote command access Remote file inclusion
modsecurity_crs_40_generic_attacks.conf modsecurity_crs_41_sql_injection_attacks.conf modsecurity_crs_41_xss_attacks.conf
Known Vulnerability Signatures
SpiderLabs received authorization from ET to convert their Snort rules and include them in the CRS http://www.emergingthreats.net/
Converted the following rule files emerging-web_server.rules emerging-web_specific_apps.rules
Identifying attacks against known vulnerabilities does have value Raised threat level If done correctly, lessens false positives
CRS combines the what of our generic attack payload detection with the where of ET known vuln data
Example Emerging Threats Rule
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID SELECT"; flow:established,to_server; uricontent:"/vehiclelistings.asp?"; nocase; uricontent:"vehicleID="; nocase; uricontent:"SELECT"; nocase; pcre:"/.+SELECT.+FROM/Ui"; classtype:web-application-attack; reference:cve,CVE-2006-6092; reference:url,www.securityfocus.com/bid/21154; reference:url,doc.emergingthreats.net/2007504; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/WEB_SPECIFIC_APPS/WEB_2020_Auto_gallery; sid:2007504; rev:5;)
Attack vector location –
URI + Parameter
PCRE –Weak signature
Converted Emerging Threats Rule
# (sid 2007508) ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID
SecRule REQUEST_URI_RAW "(?i:\/vehiclelistings\.asp)" "chain,phase:2,block,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:normalisePathWin,capture,ctl:auditLogParts=+E,nolog,auditlog,logdata:'%{TX.0}',id:sid2007508,rev:3,msg:'ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID ',tag:‘web-application-attack',tag:'url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/WEB_SQL_INJECTION/WEB_2020_Auto_gallery'"
SecRule &TX:'/SQL_INJECTION.*ARGS:vehicleID/' "@gt 0" "setvar:'tx.msg=ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID',setvar:tx.sqli_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rule.id}-SQL_INJECTION/SQL_INJECTION-%{matched_var_name}=%{matched_var}"
Verify the URI of the request
Verify the attack vector location from saved TX SQL
Injection data exists
Detection Mechanisms: Trojans/Backdoors
Major problem in hosting environments Uploading is allowed Some sites may be secure while others not
Upload detection Check uploading of files containing viruses (i.e. WORD
docs)util/modsec-clamscan.pl
Check uploading of http backdoor page
Access detection Known signatures (x_key header) Generic file management output (gid, uid, drwx, c:\) modsecurity_crs_45_trojans.conf
Detection Mechanisms: Information Leakage
Monitoring outbound application data HTTP Error Response Status Codes SQL Information Leakage Stack Dumps Source Code Leakage
Last line of defense if all else fails Provide feedback to application developers Important for customer experience Makes life for the hacker harder (if blocking is
used) modsecurity_crs_50_outbound.conf
Latest CRS Improvements:v2.0.9
Latest Improvements
Lua port of PHPIDS Converter.php code Advanced normalization functions More accurate use of PHPIDS Filters Centrifuge Generic Attack Payload Detection
Experimental Rules Generic attack payload detection
CRS Demonstration Page Request Header Tagging
Lua port of PHPIDS
http://phpids.net/ ~70 regular expression rules to detect common
attack payloads XSS SQL Injection RFI
Filters are heavily tested by the community and updated frequently
Trustwave SpiderLabs worked with PHPIDS lead to port code to Lua for use in ModSecurity’s API https://svn.php-ids.org/svn/trunk/lib/IDS/Converter.php https://svn.php-ids.org/svn/trunk/lib/IDS/default_filter.xml Thanks to Mario Heiderich
Lua port of PHPIDS
Example normalization functions --[[ Make sure the value to normalize and monitor doesn't contain Regex DoS --[[ Check for comments and erases them if available ]] --[[ Strip newlines ]] --[[ Checks for common charcode pattern and decodes them ]] --[[ Eliminate JS regex modifiers ]] --[[ Converts from hex/dec entities ]] --[[ Normalize Quotes ]] --[[ Converts SQLHEX to plain text ]] --[[ Converts basic SQL keywords and obfuscations ]] --[[ Detects nullbytes and controls chars via ord() ]] --[[ This method matches and translates base64 strings and fragments ]] --[[ Strip XML patterns ]] --[[ This method converts JS unicode code points to regular characters ]] --[[ Converts relevant UTF-7 tags to UTF-8 ]] --[[ Converts basic concatenations ]] --[[ This method collects and decodes proprietary encoding types ]]
PHPIDS Example Filter
<filter> <id>1</id> <rule><![CDATA[(?:"[^"]*[^-]?>)|(?:[^\w\s]\s*\/>)|(?:>")]]></rule> <description>finds html breaking injections including whitespace attacks</description> <tags> <tag>xss</tag> <tag>csrf</tag> </tags> <impact>4</impact> </filter>
Converted PHPIDS Example Filters
SecRule TX:'/^(QUERY_|REQUEST_|ARGS:).*_normalized/' "(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+)" "phase:2,capture,t:none,pass,skip:1,nolog,auditlog,msg:'Detects obfuscated script tags and XML wrapped HTML',id:'9000033',tag:'WEB_ATTACK/XSS',logdata:'%{TX.0}',severity:'2',setvar:'tx.msg=%{rule.id}-%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:'tx.%{tx.msg}-WEB_ATTACK/XSS-%{matched_var_name}=%{tx.0}'"
SecRule TX:PARANOID_MODE "@eq 1" "chain,phase:2,t:none,logdata:'%{TX.0}',severity:'2',pass,nolog,auditlog,msg:'Detects obfuscated script tags and XML wrapped HTML',id:'9000033',tag:'WEB_ATTACK/XSS'" SecRule ARGS|REQUEST_BODY|REQUEST_URI_RAW "(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+)" "capture,multiMatch,t:none,t:urlDecodeUni,t:cssDecode,t:jsDecode,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,setvar:'tx.msg=%{rule.id}-%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:'tx.%{tx.msg}-WEB_ATTACK/XSS-%{matched_var_name}=%{tx.0}'"
Centrifuge Code
Negative security approach to combating XSS and SQL Injection is doomed to fail… Unlimited ways to write functionally equivalent code Obfuscation methods, however often have certain
characteristics
PHPIDS has an interesting approach to identify attack payloads through heuristics called Centrifuge Analysis of the use of special characters
Ratio between the count of the word characters, spaces, punctuation and the non word characters If <3.49 = malicious
Normalization and stripping of any word character and spaces including line breaks, tabs and carriage returns Regex check in default_filters.xml catches results
PHPIDS Lua Port Demo
Experimental Generic Detection Rules
Two new experimental/beta generic detection rules Optional_rules/modsecurity_crs_40_experimental.conf
Restricted Character Anomaly Usage Analyzes the number and type of meta-characters present
in a payload Testing thus far shows that the detection is good except
for free-form text fields. Need to adjust the anomaly scoring for you own site
Repetitive use of non-Word characters Currently alerts if there are 4 or more special characters
found in a row
Live CRS Demonstration Page
Live CRS Demonstration Page
http://www.modsecurity.org/demo/
CRS Demonstration Page
Request will go through CRS page first and then we proxy the request to the PHPIDS page http://demo.php-ids.org/
We then inspect the inbound with the outbound and provide results CRS detected an attack CRS did not find anything malicious but PHPIDS did Neither CRS nor PHPIDS found anything malicious
A link is provided to report false negatives to our JIRA ticketing system https://www.modsecurity.org/tracker/browse/CORERULES
We have received >6700 attacks thus far
CRS Demonstration Page
Demonstration page Demo
Request Header Tagging
CRS Demonstration Page
When in a distributed architecture, you can share WAF data with downstream hosts
Similar to SMTP SPAM data added to mime-headers optional_rules/modsecurity_crs_49_header_tagging.conf
Maps to AppSensor Detection Point – RP2 (Suspicious External User Behavior)
GET /path/to/foo.php?test=1%27%20or%20%272%27=%272%27;-- HTTP/1.1Host: www.example.com User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8Accept-Language: en-us,en;q=0.5Accept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7X-WAF-Events: TX: / 999935-Detects common comment types-WEB_ATTACK/INJECTION-ARGS:test, TX:999923-Detects JavaScript location/document property access and window access obfuscation-WEB_ATTACK/INJECTION-REQUEST_URI_RAW, TX:950001- WEB_ATTACK/SQL_INJECTION-ARGS:testX-WAF-Score: Total=48; sqli=2; xss=Connection: Keep-Alive
Example Header Tagging
Future Directions
Future Directions
Preventing XSS with Content Injection Javascript Sandboxing with Active Content Signatures http://blog.modsecurity.org/2010/09/advanced-topic-
of-the-week-xss-defense-via-content-injection.html http://www.modsecurity.org/demo/demo-deny-
noescape.html
Implementing OWASP AppSensor Detection Points We currently have some mappings for existing
events Will be using Lua to implement other
aggregate/behavioral/trend detection Points
Call for Community Help
We have made great strides with CRS v2.0 but there is still much work to be done
Test out the CRS demo page and report any issues found either to the mail-list or to JIRA
Need Rule Documentation help Please sign up on our project mail-list if you want
to help https://lists.owasp.org/mailman/listinfo/owasp-
modsecurity-core-rule-set