(fios#02) 2. elk 포렌식 분석

FORENSIC INSIGHT; DIGITAL FORENSICS COMMUNITY IN KOREA

ELK Forensics

demantos

[email protected]

http://malwarel4b.blogspot.kr

Cho Hoon

mailto:[email protected]

http://malwarel4b.blogspot.kr

forensicinsight.org Page

Table of Contents

2

▪ Introduction��

▪ How�� It�� Works��

▪ Logstash��

▪ Elasticsearch��

▪ Kibana��

▪ ELK�� for�� Analysis��

▪ ELK�� for�� Windows�� Event�� Log��

▪ Performance�� Test��

▪ Future�� Work��

▪ Reference��

▪ Q&A


Introduction

3


Introduction

4


How It Works

5

Logstash ElasticSearch Kibana

•web�� log�� (apache,�� iis,�� …⋯)�� •mail�� log�� •mactime�� •microsoft�� event�� log�� •syslog�� •plaso�� •supertimeline�� •and�� more

•grok�� •date�� •geoip�� •translate�� •mutate�� •…⋯

input

parse/filter

output

1 2

3

1 2

3

4

4

node

primary�� shard

replica�� shard

File�� or�� Network


How It Works

6

Shipper�� #1�� iptables�� +�� apache2�� +�� syslog

Shipper�� #2�� iptables�� +�� syslog

Shipper�� #3iptables�� +�� apache2�� +�� syslog

Redis Logstash Elasticsearch Kibana

Indexer

Reads�� data�� from�� Redis

Store�� data�� in�� Elasticsearch

Reads�� data�� from�� ES


Logstash

7

▪ 로그�� 수집,�� 중앙화,�� 파싱,�� 저장,�� 검색을�� 위한�� 통합�� 프레임웍��

▪ Raw�� Data를�� 받아서�� 지정된�� 형식으로�� 파싱해서�� Elasticsearch로�� 전달��

▪ 입력�� 받은�� 데이터는�� 지정된�� index명으로�� 인덱싱되며,�� Elasticsearch에서�� index명을�� 통해�� 데이터�� 조회�� 가능��

▪ 주요�� 설정�� 파일��

• /etc/default/logstash��

✓ logstash�� 성능�� 및�� 환경설정�� 관련�� 설정�� 파일��

• /etc/logstash/conf.d/*.conf��

✓ logstash�� 입력,�� 필터,�� 출력�� 설정�� 파일


Logstash

8

▪ /etc/default/logstash��

• LS_OPTS="-w�� 4"��

✓ CPU�� 코어의�� 개수를�� 지정하여�� 작업�� 성능�� 향상��

• LS_HEAP_SIZE="500m"��

✓ logstash�� 프로세스가�� 사용할�� 메모리�� 사이즈

logstash 18235 1 99 10:15 ? 01:38:31 /usr/bin/java -‐Djava.io.tmpdir=/var/lib/logstash -‐Xmx500m -‐XX:+UseParNewGC -‐XX:+UseConcMarkSweepGC -‐Djava.awt.headless=true -‐XX:CMSInitiatingOccupancyFraction=75 -‐XX:+UseCMSInitiatingOccupancyOnly -‐jar /opt/logstash/vendor/jar/jruby-‐complete-‐1.7.11.jar -‐I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -‐f /etc/logstash/conf.d -‐l /var/log/logstash/logstash.log -‐w 4


Logstash

9

▪ /etc/logstash/conf.d/*.conf��

• 데이터를�� 어떻게�� 받을�� 것인지��

• 받은�� 데이터를�� 어떻게�� 파싱할�� 것인지��

• 파싱된�� 데이터를�� 어디로�� 전달할�� 것인지

input

parse/filter

output

collectd,drupal_dblog,elasticsearch,eventlog,exec,file, ganglia, gelf, gemfire, generator, graphite, heroku, imap, invalid_input, irc, jmx, log4j, lumberjack, pipe, puppet_facter, rabbitmq, rackspace, redis, relp, s3, snmptrap, sqlite, sqs, stdin, stomp, syslog, tcp, twitter, udp, unix, varnishlog, websocket, wmi, xmpp, zenoss, zeromq

advisor, alter, anonymize, checksum, cidr, cipher, clone, collate, csv, date, dns, drop, elapsed, elasticsearch, environment, extractnumbers, fingerprint, gelfify, geoip, grep, grok, grokdiscovery, i18n, json, json_encode, kv, metaevent, metrics, multiline, mutate, noop, prune, punct, railsparallelrequest, range, ruby, sleep, split, sumnumbers, syslog_pri, throttle, translate, unique, urldecode, useragent, uuid, wms, wmts, xml, zeromq

boundary, circonus, cloudwatch, csv, datadog, datadog_metrics, elasticsearch, elasticsearch_http, elasticsearch_river, email, exec, file, ganglia, gelf, gemfire, google_bigquery, google_cloud_storage, graphite, graphtastic, hipchat, http, irc, jira, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, rabbitmq, rackspace, redis, redmine, riak, riemann, s3, sns, solr_http, sqs, statsd, stdout, stomp, syslog, tcp, udp, websocket, xmpp, zabbix, zeromq

[참고]�� http://logstash.net/docs/1.4.2/


Logstash

10

input { tcp { type => "apache" port => 18080 } } filter { if [type] == "apache" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] } } } output { if [type] == "apache" { elasticsearch { index => "logstash-‐iistest02" host => localhost } } }

Block

Block

Block

plugin


Logstash

11

▪ Input��

• logstash는�� 지정된�� 포트를�� 오픈하고�� 유입되는�� 데이터를�� 받아서�� 다음�� 과정으로�� 전달한다.��

• cat /IIS/W3C1234/ex* | nc -‐vv localhost 18003

input { tcp { type => "w3c_extended_iis" port => 18003 } }

input { syslog { type => "syslog" port => 5514 } }


Logstash

12

▪ Filter

filter { if [type] == "w3c_extended_iis" { # drop comment lines if ([message] =~ /^#/) { drop{} } csv { columns => ["date", "time", "s_ip", "cs_method", "cs_uri_stem", "cs_uri_query", "s_port", "cs_username", "c_ip", "cs_user_agent", "sc_status", "sc_substatus", "sc_win32_status", "time_taken"] separator => " " } mutate { merge => ["date", "time"] } mutate { join => ["date", " "] } date { match => ["date", "YYYY-‐MM-‐dd HH:mm:ss" ] timezone => ['UTC'] } geoip { source => "c_ip" }


Logstash

13

▪ Filter

# extract macb info if ("m" in [macb]) { mutate { add_tag => ["modified"] } } if ("a" in [macb]) { mutate { add_tag => ["accessed"] } } if ("c" in [macb]) { mutate { add_tag => ["changed"] } } if ("b" in [macb]) { mutate { add_tag => ["birth"] } } # extract file extension grok { match => ["path", "(?<filename>[^/]+?)?$"] } grok { match => ["filename", "((\.(?<ext>[^./]+))?)?$"] } mutate { lowercase => ["ext"] remove_field => ["message", "perms", "uid", "gid"] } } }


Logstash

14

▪ Filter�� -�� grok��

• 정규표현식을�� 이용하여�� 임의의�� 문자열이나�� 구조를�� 파싱할�� 수�� 있는�� 플러그인��

• 공식적으로�� 배포하는�� 다양한�� 정규표현식�� 존재��

✓ https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns��

• 구조��

• 다양한�� 형태의�� 로그(멀티�� 라인�� 로그�� 포함)의�� 파싱�� 가능��

• 필요한�� 경우�� 정규표현식을�� 이용하여�� 직접�� 만들어서�� 사용�� 가능

%{syntax:semantic}

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %

{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-‐)

COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}


▪ Filter�� -�� grok

Logstash

15

Dec 26 10:45:01 localhost postfix/pickup[27869]: 841D26FFA8: uid=0 from=<root>

%{SYSLOGTIMESTAMP:timestamp}

%{SYSLOGHOST:hostname}

postfix\/(?<component>[\w._\/%-‐]+)

(?:\[%{POSINT:pid}\]):

(?<queueid>[0-‐9A-‐F]{,11}):

%{GREEDYDATA:message}

Built-in�� pattern

User�� defined�� pattern


Logstash

16


• grok�� Test�� App�� :�� http://grokdebug.herokuapp.com


Logstash

17


• grok�� constructor�� :�� http://grokconstructor.appspot.com/


Logstash

18

▪ Filter�� -�� useragent

if [cs_user_agent] != "" { useragent { source => "cs_user_agent" prefix => "user_agent." }


Logstash

19

▪ Filter�� -�� geoip

geoip { source => "c_ip"


Logstash

20

▪ Output��

• elasticsearch�� 플러그인의�� index�� 옵션에�� 지정된�� 이름으로�� 인덱싱이�� 되며,�� 이후�� Elasticsearch에서�� 해당�� index명을��

이용하여�� 화면에�� 출력��

• 한번�� 인덱싱된�� 데이터의�� index는�� 변경이�� 불가능

output { if [type] == "w3c_extended_iis" { elasticsearch { index => "logstash-‐%{[type]}-‐%{+YYYY.MM.dd}" host => "localhost" } } }


Elasticsearch

21

▪ Apache�� Lucene(1)�� 기반으로�� 인덱싱�� 데이터�� 검색��

▪ RESTfully�� as�� JSON�� over�� HTTP��

▪ 다양한�� 용도의�� API(2)�� 제공��

▪ 플러그인(HQ,�� Head)을�� 이용하여�� 웹을�� 통해�� cluster,�� node,�� indices�� 등의�� 정보를�� 쉽게�� 확인�� 가능��

▪ 주요�� 설정�� 파일��

• /etc/default/elasticsearch��

✓ 성능�� 및�� 환경설정�� 관련�� 설정�� 파일��

• /etc/elasticsearch/elasticsearch.yml��

✓ ES�� 인덱싱�� 관련�� 설정�� 파일��

▪ Elasticsearch�� vs�� RDBMS

Elasticsearch RDBMS

Index Database

Type Table

Document Row

Field Column

(1)�� http://lucene.apache.org/��

(2)�� http://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

1 2

3

1 2

3

4

4


Elasticsearch

22

▪ /etc/default/elasticsearch��

• ES_HEAP_SIZE=4g��

✓ elasticsearch�� 프로세스가�� 사용할�� 메모리�� 사이즈��

✓ 물리�� 메모리�� 사이즈의�� 절반(최대�� 32GB)��

• MAX_LOCKED_MEMORY=unlimited��

✓ Maximum�� locked�� memory�� size��

✓ 메모리�� 스왑�� 방지를�� 위해�� elasticsearch.yml의�� 옵션�� 중�� bootstrap.mlockall�� 옵션을�� true로�� 사용할�� 경우�� 이�� 값을�� unlimited

로�� 설정해야�� 함��

✓ ES_HEAP_SIZE도�� 설정해야�� 함

elastic+ 8363 1848 8 15:51 ? 00:02:13 /usr/lib/jvm/java-‐7-‐openjdk-‐amd64//bin/java -‐Xms4g -‐Xmx4g -‐Xss256k -‐Djava.awt.headless=true -‐XX:+UseParNewGC -‐XX:+UseConcMarkSweepGC -‐XX:CMSInitiatingOccupancyFraction=75 -‐XX:+UseCMSInitiatingOccupancyOnly -‐XX:+HeapDumpOnOutOfMemoryError -‐XX:+DisableExplicitGC -‐Dfile.encoding=UTF-‐8 -‐Delasticsearch -‐Des.pidfile=/var/run/elasticsearch.pid -‐Des.path.home=/usr/share/elasticsearch -‐cp :/usr/share/elasticsearch/lib/elasticsearch-‐1.4.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -‐Des.default.config=/etc/elasticsearch/elasticsearch.yml -‐Des.default.path.home=/usr/share/elasticsearch -‐Des.default.path.logs=/var/log/elasticsearch -‐Des.default.path.data=/var/lib/elasticsearch -‐Des.default.path.work=/tmp/elasticsearch -‐Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch


Elasticsearch

23

▪ /etc/elasticsearch/elasticsearch.yml��

• 별도의�� 설정을�� 변경하지�� 않아도�� 무방함��

• but,�� 인덱싱�� 등의�� 성능�� 향상을�� 위해서�� 몇가지�� 설정을�� 해주어야�� 함��

• index.number_of_shards

• index.number_of_replicas

• indices.memory.index_buffer_size

• index.store.type

• index.translog.flush_threshold_ops

• index.refresh_interval

• bootstrap.mlockall


Elasticsearch

24

▪ Tokenization��

• ES는�� 사전에�� 정의된�� 토큰(공백,�� 점,�� 콤마�� 등)을�� 이용해서�� 입력된�� 데이터를�� 분할하여�� 인덱싱��

• User-Agent,�� 국가명�� 등에는�� 공백이�� 포함되어�� 있어�� 비정상적으로�� 출력됨��

✓ 새로운�� 인덱스�� 생성시�� 각각의�� 필드에�� ".raw"라는�� 새로운�� 필드를�� 추가로�� 생성하게�� 하여�� 이러한�� 문제�� 해결�� 가능��

• .raw�� 필드가�� 생성되게�� 하기�� 위해서는�� index명이�� "logstash-"로�� 시작해야�� 함


Elasticsearch

25


Elasticsearch

26


Kibana

27


ELK for Analysis

28


ELK for Analysis

29

File A File B File C


ELK for Analysis

30

▪ ELK는�� 로그�� 분석�� 및�� 모니터링�� 도구이다.��

▪ 모니터링�� 도구로만�� 사용할�� 수�� 있는가?��

• 포렌식�� 관점에서도�� 사용�� 가능��

• 텍스트�� 데이터의�� 시각화를�� 통한�� 이점�� 존재��

• plaso와�� 같은�� 도구를�� 이용하여�� 타임라인�� 데이터�� 생성�� 후�� ELK�� 활용�� 가능��

▪ 분석�� 방법론�� 필요��

• mactime,�� timeline�� 데이터를�� 이용하여�� 분석할�� 경우�� 방법론�� 필요��

• 특정�� 방법론을�� 적용하여�� 시간의�� 범위를�� 좁혀�� 가면서�� 분석


ELK for Windows Event Log

31

▪ nxlog��

• 오픈소스�� 로그�� 관리�� 도구��

• 윈도우�� 시스템에�� 서비스�� 형태로�� 설치/동작��

• 다양한�� 형태의�� 로그�� 파싱�� 모듈�� 보유 define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> # Windows Event Log <Input eventlog> Module im_msvistalog # Module im_mseventlog Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input> <Output out> Module om_tcp Host 192.168.1.126 Port 3515 </Output> <Route 1> Path internal, eventlog => out </Route>

Input

Processor

Output

im_xxx

pm_xxx

om_xxx



32

▪ nxlog��

• 필요한�� 경우�� 이벤트�� 로그를�� 필터링하여�� 전송�� 가능

<Input in> Module im_msvistalog Exec if ($TargetUserName == 'SYSTEM') OR ($EventType == 'VERBOSE') drop(); </Input>

<Input unix> Module im_uds uds /dev/log </Input>

<Processor filter> Module pm_filter Condition $raw_event =~ /failed/ or $raw_event =~ /error/ </Processor>

<Output out> Module om_file File "/var/log/error" </Output>

<Route 1> Path unix => filter => out </Route>



33

binary

json



34



35



36



37


Performance Test

38

Case #1 (default)

Case #2 Case #2’ Case #3 Case #3’ Case #4 Case #4’

index.number_of_replicas 1 1 1 0

index.number_of_shards 5 3 3 1

index.translog.flush_threshold_ops 5000 50000 50000 50000

index.refresh_interval 5s 30s 30s -1

indices.memory.index_buffer_size 10% 30% 50% 50%

index.store.type % mmapfs mmapfs mmapfs

bootstrap.mlockall - - TRUE TRUE

MAX_LOCKED_MEMORY - - unlimited unlimited

indexing duration 180m 168m 135m 174m 143m 165m 145m

docs / primary size 9,600,201/3.0GB 9,600,201/2.8GB 9,600,201/2.7GB 9,600,201/2.7GB 9,600,201/2.7GB 9,600,201/2.7GB 9,600,201/2.7GB

indexing duration 65m 67m 48m 67m 48m 59m 49m

docs / primary size 3,303,249/926.5M 3,303,249/888.1M 3,303,249/889.6M 3,303,249/886.7M 3,303,249/890.4M 3,303,249/894.0M 3,303,249/983.4M

• 테스트�� 시스템�� 사양��

Intel(R) Core(TM) i5-‐2540M 2.60GHz (4 Core)/ 8G RAM / Ubuntu 14.04.2 LTS 64bit • 공통�� 설정��

/etc/default/elasticsearch : ES_HEAP_SIZE=4g /etc/default/logstash : LS_HEAP_SIZE=500m / LS_OPTS=-‐w 4

raw data 2.2G

raw data 828M

• 일반�� 케이스는�� 로그�� 데이터�� 전송과�� 동시에�� indices가�� 자동으로�� 생성되게끔�� 하여�� 테스트��

• 각�� 케이스별�� 추가�� 케이스는�� indices의�� 설정값이�� config�� 파일에�� 설정된�� 것과�� 다르게�� 표시되어�� 인덱스를�� 수동으로�� 생성하고�� 일부�� 설정을�� API를�� 통해서�� 수정한�� 후�� 로그�� 데이터를�� 전송��

• 각각의�� 테스트는�� 로그�� 데이터를�� 인덱싱�� 중인�� indices만�� open하고,�� 다른�� indices들은�� 모두�� close된�� 상태에서�� 수행하였음


Performance Monitoring

39

Marvel


Future Work

40

▪ 현장에서�� 활용�� 가능한�� 방안은?��

• 로그�� 사이즈(레코드�� 개수)에�� 따라서�� 분석�� 시간이�� 달라짐��

• 현장에서�� 로그를�� 인덱싱해서�� 분석하기엔�� 노트북�� 성능이�� 좋지�� 않음��

▪ 다양한�� 형태의�� 원시�� 데이터를�� 파싱할�� 수�� 있는�� logstash�� config�� 파일�� 준비��

• 윈도우�� 이벤트�� 로그��

• 메모리�� 덤프에서�� 추출된�� 데이터��

• 침해사고�� 아티팩트��

• plaso/log2timeline�� 설정�� 파일�� 수정�� 필요��

▪ Elasticsearch의�� 설정값�� 및�� API에�� 대한�� 연구�� 필요��

• 인덱싱�� 속도에�� 영향을�� 많이�� 미치는�� 것으로�� 판단됨��

▪ Timeline�� 데이터의�� 시각화�� 이후�� 분석�� 방법론�� 필요��

▪ NT�� 계열�� (Windows�� 2003)�� 이벤트�� 로그의�� 비정상적인�� 파싱�� 문제�� 해결


Reference

41

▪ ElasticSearch�� and�� Logstash�� Tuning��

• http://jablonskis.org/2013/elasticsearch-and-logstash-tuning/index.html��

▪ Finding�� the�� needle�� in�� the�� haystack�� with�� ELK��

• https://digital-forensics.sans.org/summit-archives/dfirprague14/

Finding_the_Needle_in_the_Haystack_with_FLK_Christophe_Vandeplas.pdf��

▪ Elasticsearch�� -�� Advanced�� settings�� and�� Tweaks��

• http://kufli.blogspot.kr/2014/11/elasticsearch-advanced-settings-and.html��

▪ Elasticsearch,�� Logstash�� &�� Kibana�� -�� Kevin�� Kluge��

• http://www.socallinuxexpo.org/scale12x-supporting/default/files/presentations/Scale12x%20-%20Intro%20to

%20Elasticsearch%20(Kluge).pdf��

▪ Elasticsearch�� -�� Shard�� &�� Replica��

• https://guruble.wordpress.com/2014/02/23/elasticsearch-2-shard-replica/��

▪ Collect�� &�� visualize�� your�� logs�� with�� Logstash,�� Elasticsearch�� &�� Redis��

• http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/��

▪ Logstash�� /�� Elasticsearch�� /�� Kibana�� for�� Windows�� Event�� Logs��

• http://www.ragingcomputer.com/2014/02/logstash-elasticsearch-kibana-for-windows-event-logs��

▪ Many�� slides�� on�� Slideshare�� and�� Many�� article�� on�� google�� search


Question and Answer

42

(fios#02) 2. elk 포렌식 분석

Technology