ModSecurity Advanced Topic of the Week: Detecting Malware with Fuzzy Hashing
2015-01-14 10:12
387 查看
In the most recent release of ModSecurity
v2.9.0-RC1, we introduced a new operator called
@fuzzyHash which uses functionality from the
ssdeep tool. This blog post will demonstrate a powerful use-case with ModSecurity which is identifying malware in web attachment uploads.
ssdeep is a program for computing context triggered piecewise hashes (CTPH). Also called fuzzy hashes, CTPH can match inputs that have homologies. Such inputs have sequences of identical bytes in the same order, although bytes in between these sequences
may be different in both content and length.
A complete explanation of CTPH can be found in Identifying almost identical files using context triggered piecewise hashing from the journal Digital Investigation. There is a
free version of this paper available through the Digital Forensic Research Workshop conference, free version of Identifying almost identical files using context triggered piecewise
hashing.
There are some usage scenarios in the Quickstart guide and the Forensics Wiki entry on ssdeep.
The best example of practical fuzzy hashing usage is during forensic investigations when malware analysts want to compare unknown files with known malware families. Even if the files are different, fuzzy hashing can let you know what malware groups it is
similar too and to what degree (%). Here is
link to a previous blog post written by my SpiderLabs Research colleagues who analyze malware where they show ssdeep usage.
Here is a quick listing of the top categories of web-malware that we identified in a previous Global Security Report:
Attackers can use many different attack methods to get this malware uploaded to a target website including Remote File Inclusion (RFI) attacks but often times there are
file uploading flaws in legitimate application software that can be abused. This is where our use-case of using the new @fuzzyHash operator in ModSecurity comes into play.
Fuzzy hashing is so useful in this scenario because standard cryptographic hash testing quickly fails as script kiddies are constantly modifying their code. Granted, most of the code changes are cosmetic as they rip off each other's code and simply rename
it or slap their hacker handle on it and give gr33tz to other kiddies. These changes, however, render normal hash checks (like with md5 or sha1) almost useless.
source or you can use one of the 3rd party OS repositories (example using -
$ apt-get install ssdeep). Once ssdeep is installed, you can then configure ModSecurity.
After completing the compilation and install steps. We can now configure the rules.
Looking at this rule, we see the following:
We are inspecting a new variable that we added to ModSecurity v2.9.0-RC1 called
FILES_TMP_CONTENT. This variable will hold the data captured from multipart-form-data file attachment uploads.
We are using the @fuzzyHash operator which takes two parameters:
First parameter is the path to the ssdeep hash text file we created earlier. In this example, the file is located in the same directory as this rules config file.
Second parameter is the minimum threshold score of similarity between files. This means that our operator will only match if the file inspect matches at least 90% to a file in our ssdeep fuzzy hash file.
v2.9.0-RC1, we introduced a new operator called
@fuzzyHash which uses functionality from the
ssdeep tool. This blog post will demonstrate a powerful use-case with ModSecurity which is identifying malware in web attachment uploads.
What is ssdeep (Fuzzy Hashing)?
The ssdeep project gives the following information:ssdeep is a program for computing context triggered piecewise hashes (CTPH). Also called fuzzy hashes, CTPH can match inputs that have homologies. Such inputs have sequences of identical bytes in the same order, although bytes in between these sequences
may be different in both content and length.
A complete explanation of CTPH can be found in Identifying almost identical files using context triggered piecewise hashing from the journal Digital Investigation. There is a
free version of this paper available through the Digital Forensic Research Workshop conference, free version of Identifying almost identical files using context triggered piecewise
hashing.
There are some usage scenarios in the Quickstart guide and the Forensics Wiki entry on ssdeep.
The best example of practical fuzzy hashing usage is during forensic investigations when malware analysts want to compare unknown files with known malware families. Even if the files are different, fuzzy hashing can let you know what malware groups it is
similar too and to what degree (%). Here is
link to a previous blog post written by my SpiderLabs Research colleagues who analyze malware where they show ssdeep usage.
Detecting Web-based Malware
When dealing with web application defense, web-based malware such as webshells and backdoors are a huge problem. Here is an example screen shot of a common webshell:Here is a quick listing of the top categories of web-malware that we identified in a previous Global Security Report:
Attackers can use many different attack methods to get this malware uploaded to a target website including Remote File Inclusion (RFI) attacks but often times there are
file uploading flaws in legitimate application software that can be abused. This is where our use-case of using the new @fuzzyHash operator in ModSecurity comes into play.
Fuzzy hashing is so useful in this scenario because standard cryptographic hash testing quickly fails as script kiddies are constantly modifying their code. Granted, most of the code changes are cosmetic as they rip off each other's code and simply rename
it or slap their hacker handle on it and give gr33tz to other kiddies. These changes, however, render normal hash checks (like with md5 or sha1) almost useless.
Install ssdeep
Before turning to ModSecurity, the first step in this process is to install ssdeep on your web server host. You can install fromsource or you can use one of the 3rd party OS repositories (example using -
$ apt-get install ssdeep). Once ssdeep is installed, you can then configure ModSecurity.
Create Fuzzy Hash Listing of Malware
We have captured thousands of webshells/backdoors as part of our research efforts in SpiderLabs. With this archive, we are then able to use ssdeep to create a text file listing of fuzzy hashes. Here is a small example -$ ssdeep -r -b * > webshells-ssdeep.txt ; head webshells-ssdeep.txt ssdeep,1.1--blocksize:hash:hash,filename 24:OV9QJXQILgD9C7SwKjB8y4skRbsL4J7S52eYwsTdhd/4:QQN7SwAEvRbF7S02sTHp4,"529.php" 192:wUYrBEgQugt1Rv/2gptgSCF4iemhwZt4MWRHOE5wbyzvE1EbKUuHFHgXyc:wUYrBAeH5IU0owbWmEb0HFH3c,"AK-74 Security Team Web Shell Beta Version.php" 768:FpT8+vA/F+OxCDy1L8zWHdzD/xMm/f7+CiA:F3Y/EOxCDy140dzDpMif7+bA,"Antichat Shell. Modified by Go0o$E.php" 192:NQtjv0bj7tJ4q89i8V7y+Q7cmSUwNXMEKhsD2O0b6RWUE:Nej2jL4qinVW++cQwNXMLhsDv0w2,"Antichat Shell.php" 768:O0bU4SbqX+4jl7EBNRjal0OC5eVdVIhyTuIqYDvNTbervZWhVZ21wNX2u89VACrn:FbRSbqOMl7WtaK5ejVeI9zNvoWJD899R,"C99madShell v. 2.0 madnet edition.php" 192:NPgGy+ChwSFHkueyE+C/dw1U2Vk+EndG9X5Mf8x6p1PdFrrELtMIfiA7KcMYtv/d:NwhB8hmDDRMIfiA7KbYtv/d,"CasuS 1.5.php" 384:rIwHyY1gNv8yiVnggmS2TgFnGjmufsnoC0WgS+i3UgMGMqm7iF:rnHyegJHmggZ28Fc8oGBM7iF,"CrystalShell v.1.php" 384:FTOCnQnAJsznh9m2mwLTEtltCj2gQgrmUfycqfQNUvhdWQhv05Sh4Z6upKmM:HJwmUmiYXmM,"Cyber Shell (v 1.0).php" 384:GIr8nsnAJ0pncTyZhCmmg7ylEJKiIhyZzVxUdnA6abSczJMc84ZCrczRcmNbQ60x:EJ3yZhCE7y0bZP00Mj6G,"Cyber Shell.php"
Compile ModSecurity
During the configure step in building ModSecurity, it will try to identify the proper ssdeep source files installed on your system.$ ./configure checking for a BSD-compatible install... /opt/local/bin/ginstall -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /opt/local/bin/gmkdir -p checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk checking whether make sets $(MAKE)... yes --CUT-- checking for libpcre config script... /usr/local/bin/pcre-config configure: using pcre v8.32 checking for libapr config script... /usr/local/apr/bin/apr-1-config configure: using apr v1.5.1 checking for libapu config script... /usr/local/apr/bin/apu-1-config configure: using apu v1.5.4 checking for libxml2 config script... /usr/local/bin/xml2-config checking if libxml2 is at least v2.6.29... yes, 2.7.8 configure: using libxml2 v2.7.8 checking for pkg-config... /usr/local/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for liblua config script... no checking for lua install... /usr/local/lib /usr/local/include configure: using lua v5.1 checking for libcurl config script... /usr/bin/curl-config checking if libcurl is at least v... yes, 7.24.0 checking if libcurl is linked with gnutls... no configure: using curl v7.24.0 checking for libyajl config script... /usr/local/bin/pkg-config configure: using yajl v2.0.5 checking for ssdeep path... -lfuzzy -DWITH_SSDEEP -I/usr/local/ checking for library containing fuzzy_hash_buf... -lfuzzy checking that generated files are newer than configure... done configure: creating ./config.status
After completing the compilation and install steps. We can now configure the rules.
Adding Fuzzy Hashing Rules
Here is an example rule that can be added to a custom rules file:SecRule FILES_TMP_CONTENT "@fuzzyHash webshells-ssdeep.txt 90" \ "id:87,phase:request,t:none,block,msg:'Malicious File Attachment Hash Match.'"
Looking at this rule, we see the following:
We are inspecting a new variable that we added to ModSecurity v2.9.0-RC1 called
FILES_TMP_CONTENT. This variable will hold the data captured from multipart-form-data file attachment uploads.
We are using the @fuzzyHash operator which takes two parameters:
First parameter is the path to the ssdeep hash text file we created earlier. In this example, the file is located in the same directory as this rules config file.
Second parameter is the minimum threshold score of similarity between files. This means that our operator will only match if the file inspect matches at least 90% to a file in our ssdeep fuzzy hash file.
Testing Fuzzy Hash Rule
With these configurations in place, let's run a quick test and simply upload one of the existing webshells we have called "ironshell.php".$ curl -k -X POST -F "action=upload" -F "file=@./ironshell.php" http://localhost/[/code] We can then look at the ModSecurity audit log file:
As you can see from the highlighted ModSecurity message data, our fuzzy hashing rule matched with an ssdeep match score of 100. That makes sense as we have not modified the ironshell.php file at all. Let's make a few changes:
We can then upload this new webshell:$ curl -k -X POST -F "action=upload" -F "file=@./ironshell-new.php" http://localhost/[/code] Here is the new alert from the Apache error_log file:
As you can see, the file was still flagged however the match percentage went down to 97%. Still, this means that we can catch modifications to this ironshell webshell code family! This makes it much more powerful vs. other exact match hashing checks.Conclusion
This is a new detection capability recently released in the v2.9.0RC1 release and we need help from the community for testing. We will soon be adding this functionality to our
Trustwave SpiderLabs commercial ModSecurity rules package.
One note for usage and detection, ssdeep has a default minimum file size of 4096. This means that really small files will not work correctly with @fuzzyHash. If you want to test out reducing the minimum size threshold, you can edit the main.c file here:
// We print a warning for files smaller than this size
#define SSDEEP_MIN_FILE_SIZE 4096
Please let us know if you have any issues using this new feature.
相关文章推荐
- the name or security ID(SID) of the domain specified is inconsistent with the trust information for that domain
- The name or security ID (SID) of the domain specified is inconsistent with the trust information for that domain(转)
- Get Acquainted with the New Advanced Features of JUnit 4
- Detecting Malice with ModSecurity: GeoLocation Data
- The name or security ID (SID) of the domain specified is inconsistent with the trust information for that domain.
- Announcing the availability of ModSecurity extension for IIS
- Get Acquainted with the New Advanced Features of JUnit 4
- The name or security ID (SID) of the domain specified is inconsistent with the trust information for that domain
- Detecting Malice with ModSecurity: GeoLocation Data
- Guru of the Week 条款10:内存管理(下篇)
- Guru of the Week #5:虚函数的重新定义
- Guru of the Week 条款08:GotW挑战篇——异常处理的安全性
- Guru of the Week 条款15:类之间的关系(下篇)
- Guru of the Week 条款12:控制流(Control Flow)
- Guru of the Week 条款19:自动转换
- Guru of the Week 条款17:转型
- Guru of the Week 条款04: 类的构造技巧
- C++ articles:Guru of the Week #3:使用标准库
- Guru of the Week 条款13:面向对象程序设计
- Guru of the Week 条款07:编译期的依赖性