Cloudera得到世界各地受管制的行业和政府组织的信任，可以存储和分析有关人、医疗保健数据、财务数据或仅对客户本身敏感的专有信息的PB级别的高度敏感或机密的信息。
任何存储客户信息、医疗保健、财务或敏感专有信息的人都需要确保他们正在采取措施保护该数据，包括检测并防止无意或恶意访问。根据Ponemon研究所的研究，全球内幕威胁的平均成本在两年内增长了31％，达到1,145万美元，而事件频发的频率在同一时期激增了47％。一份2019报告明确了公司对这些意外更担心：内幕泄露（71％）、疏忽数据泄露（65％）、以及恶意不良行为者（60％），这些比他们对损害的账号/机器（9％）的意外更担心。
GDPR、CCPA、HIPAA、PCI DSS和FIPS-200之类的法规均要求组织采取适当措施来保护敏感信息，这些措施可包括以下三个支柱：
- 静态和动态加密-确保未经身份验证的参与者无法访问数据
- 访问控制（强身份验证和授权）–确保用户就是他们所说的身份（身份验证），并且只能访问他们被允许访问的内容（授权）
- 审计和核算–了解谁访问了什么内容、何时访问以及谁更改了权限或访问控制设置，并有可能在发生数据泄露时而不是在事发后发出警报。
在Cloudera数据平台中，我们擅长通过Cloudera共享数据体验（SDX）来提供端到端安全性。在CDP中：
- 可以使用基于TLS或SASL的加密方式对所有有线协议进行加密
- 可以使用HDFS透明数据加密（私有云）或对象存储加密（公共云）对所有静态数据进行加密
- 在公共云和私有云中，所有用户访问均通过Kerberos / SPNEGO或SAML进行身份验证。
- 所有数据访问均通过基于属性的访问控制或基于角色的访问控制（使用Apache Ranger作为SDX的一部分）进行授权。
- 再次使用Apache Ranger审核所有数据访问和数据访问控件。
保护性监控
通过有效的保护性监控计划，公司可以确保他们可以了解谁正在访问或尝试访问整个IT领域中的哪些数据以及从哪些设备进行访问。这可以通过以下方式完成：
- 合规性和报告–在谁在访问特定的数据资产之后的事实报告
- 数字取证和事件响应–在发现违规行为后对监管机构或信息专员做出响应
- 先进的威胁检测–实时监控访问事件，以识别用户级别，数据资产级别或跨系统的行为变化。某些SIEM平台（例如Securonix）包括这些类型的功能。
Cloudera数据平台中的审核
CDP中的所有数据访问组件都将审核事件发送到Apache Ranger，在其中存储它们并可以在可配置的保留期限内对其进行搜索。
在本博客中，我们将演示如何通过系统日志将这些审核事件流式传输到第三方SIEM平台，或者将它们写入本地文件，现有的SIEM代理可以在其中拾取它们。在这种体系结构中，我们将在每个服务上配置插件，以将审核事件导出到远程syslog服务器并写入本地磁盘。
能够执行复杂过滤和路由逻辑的远程syslog服务器的示例是运行Cloudera Flow NiFi服务器的ListenSyslog处理器，如此处所示。
为此，我们将配置Ranger插件以将其事件写入log4j，然后在每个服务上配置log4j设置以添加文件和syslog附加程序。
HDFS
HDFS审核所有服务的所有文件交互。使用Cloudera Manager，我们将设置以下设置：
HDFS Service Advanced Configuration Snippet (Safety Valve) for ranger-hdfs-audit.xml | Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit |
|---|---|
NameNode Logging Advanced Configuration Snippet (Safety Valve) | log4j.appender.RANGER_AUDIT=org.apache.log4j.DailyRollingFileAppenderlog4j.appender.RANGER_AUDIT.File=/var/log/hadoop-hdfs/ranger-hdfs-audit.loglog4j.appender.RANGER_AUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.RANGER_AUDIT.layout.ConversionPattern=%m%nlog4j.logger.ranger.audit=INFO,RANGER_AUDIT,SYSAUDITlog4j.appender.SYSAUDIT=org.apache.log4j.net.SyslogAppenderlog4j.appender.SYSAUDIT.threshold=INFOlog4j.appender.SYSAUDIT.syslogHost=<sysloghost>log4j.appender.SYSAUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.SYSAUDIT.layout.conversionPattern=%d{MMM dd HH:mm:ss} ${hostName}HDFS: %m%nlog4j.appender.SYSAUDIT.filter.a=org.apache.log4j.varia.LevelRangeFilterlog4j.appender.SYSAUDIT.filter.a.LevelMin=INFOlog4j.appender.SYSAUDIT.filter.a.LevelMax=INFO |
HiveServer 2
此插件将审核提交给HiveServer2的所有SQL。由于HiveServer2使用Log4j2，因此HiveServer2的配置使用与其他服务不同的语法。使用Cloudera Manager，我们将在Hive on Tez服务上设置以下设置：
Hive Service Advanced Configuration Snippet (Safety Valve) for ranger-hive-audit.xml | Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit |
|---|---|
HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) | appenders=console, DRFA, redactorForRootLogger, RANGERAUDIT, SYSAUDITloggers = Rangerlogger.Ranger.name = ranger.auditlogger.Ranger.level = INFOlogger.Ranger.appenderRefs = SYSAUDIT, RANGERAUDITlogger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDITlogger.Ranger.appenderRef.SYSAUDIT.ref = SYSAUDITappender.RANGERAUDIT.type=fileappender.RANGERAUDIT.name=RANGERAUDITappender.RANGERAUDIT.fileName=/var/log/hive/ranger-audit.logappender.RANGERAUDIT.filePermissions=rwx------appender.RANGERAUDIT.layout.type=PatternLayoutappender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L)) - %m%nappender.SYSAUDIT.type=Syslogappender.SYSAUDIT.name=SYSAUDITappender.SYSAUDIT.host = <sysloghost>appender.SYSAUDIT.port = 514appender.SYSAUDIT.protocol = UDPappender.SYSAUDIT.layout.type=PatternLayoutappender.SYSAUDIT.layout.pattern=%d{MMM dd HH:mm:ss} ${hostName} Hive: %m%n |
Impala
Impala守护程序将记录所有Impala SQL语句。同样，这将通过Cloudera Manager进行配置：
Impala Service Advanced Configuration Snippet (Safety Valve) for ranger-impala-audit.xml | Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit |
|---|---|
Impala Daemon Logging Advanced Configuration Snippet (Safety Valve) | log4j.appender.RANGER_AUDIT=org.apache.log4j.DailyRollingFileAppenderlog4j.appender.RANGER_AUDIT.File=/var/log/impalad/ranger-impala-audit.loglog4j.appender.RANGER_AUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.RANGER_AUDIT.layout.ConversionPattern=%m%nlog4j.logger.ranger.audit=INFO,RANGER_AUDIT,SYSAUDITlog4j.appender.SYSAUDIT=org.apache.log4j.net.SyslogAppenderlog4j.appender.SYSAUDIT.threshold=INFOlog4j.appender.SYSAUDIT.syslogHost=<sysloghost>log4j.appender.SYSAUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.SYSAUDIT.layout.conversionPattern=%d{MMM dd HH:mm:ss} ${hostName}Impala: %m%nlog4j.appender.SYSAUDIT.filter.a=org.apache.log4j.varia.LevelRangeFilterlog4j.appender.SYSAUDIT.filter.a.LevelMin=INFOlog4j.appender.SYSAUDIT.filter.a.LevelMax=INFO |
Solr
Solr服务器将记录所有提交给Solr API的查询。同样，这将通过Cloudera Manager进行配置：
Solr Service Advanced Configuration Snippet (Safety Valve) for ranger-solr-audit.xml | Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit |
|---|---|
Impala Daemon Logging Advanced Configuration Snippet (Safety Valve) | appenders=console, DRFA, redactorForRootLogger, RANGERAUDIT, SYSAUDITloggers = Rangerlogger.Ranger.name = ranger.auditlogger.Ranger.level = INFOlogger.Ranger.appenderRefs = SYSAUDIT, RANGERAUDITlogger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDITlogger.Ranger.appenderRef.SYSAUDIT.ref = SYSAUDITappender.RANGERAUDIT.type=fileappender.RANGERAUDIT.name=RANGERAUDITappender.RANGERAUDIT.fileName=/var/log/solr/ranger-solr.logappender.RANGERAUDIT.filePermissions=rwx------appender.RANGERAUDIT.layout.type=PatternLayoutappender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L)) - %m%nappender.SYSAUDIT.type=Syslogappender.SYSAUDIT.name=SYSAUDITappender.SYSAUDIT.host = <sysloghost>appender.SYSAUDIT.port = 514appender.SYSAUDIT.protocol = UDPappender.SYSAUDIT.layout.type=PatternLayoutappender.SYSAUDIT.layout.pattern=%d{MMM dd HH:mm:ss} ${hostName} Solr: %m%n |
Hue
Hue当前未与Ranger集成，但是可以将事件审核到文件中，包括用户登录事件以及用户下载查询结果的时间。可以通过Cloudera Manager启用此功能：
Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini | [desktop]audit_event_log_dir=/var/log/hue/audit/hue-audit.log |
|---|
输出示例
配置完这些设置后，我们可以进行测试以查看事件是否已正确发送。
以下事件由运行在具有自定义配置的远程服务器上的Rsyslog服务器记录：
HDFS
2021-05-04T03:25:36-07:00 host1.example.com HDFS: {"repoType":1,"repo":"cm_hdfs","reqUser":"teststd","evtTime":"2021-05-04 03:25:35.069","access":"open","resource":"/tstest/testfile2","resType":"path","action":"read","result":1,"agent":"hdfs","policy":-1,"reason":"/tstest/testfile2","enforcer":"hadoop-acl","cliIP":"172.27.172.2","reqData":"open/CLI","agentHost":"host1.example.com","logType":"RangerAudit","id":"41a20548-c55d-4169-ac80-09c1cca8265e-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{\"remote-ip-address\":172.27.172.2, \"forwarded-ip-addresses\":[], \"accessTypes\":[read]","cluster_name":"CDP PvC Base Single-node Cluster"}
2021-05-04T03:29:27-07:00 host1.example.com HDFS: {"repoType":1,"repo":"cm_hdfs","reqUser":"teststd","evtTime":"2021-05-04 03:29:22.375","access":"open","resource":"/tstest/testfile3","resType":"path","action":"read","result":0,"agent":"hdfs","policy":-1,"reason":"/tstest/testfile3","enforcer":"hadoop-acl","cliIP":"172.27.172.2","reqData":"open/CLI","agentHost":"host1.example.com","logType":"RangerAudit","id":"e6806644-1b66-4066-ae0d-7f9d0023fbbb-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{"remote-ip-address":172.27.172.2, "forwarded-ip-addresses":[], "accessTypes":[read]","cluster_name":"CDP PvC Base Single-node Cluster"}
在上面的示例中，第二次访问被拒绝（结果：0）。
Hive
2021-05-04T03:35:25-07:00 host1.example.com Hive:
{"repoType":3,"repo":"cm_hive","reqUser":"admin","evtTime":"2021-05-04 03:35:23.220","access":"SELECT","resource":"default/sample_07/description,salary","resType":"@column","action":"select","result":1,"agent":"hiveServer2","policy":8,"enforcer":"ranger-acl","sess":"303bbfbe-3538-4ebe-ab48-c52c80f23a35","cliType":"HIVESERVER2","cliIP":"172.27.172.2","reqData":"SELECT sample_07.description, sample_07.salary\r\nFROM\r\n sample_07\r\nWHERE\r\n( sample_07.salary \u003e 100000)\r\nORDER BY sample_07.salary DESC\r\nLIMIT 1000","agentHost":"host1.example.com","logType":"RangerAudit","id":"b6903fd2-49bd-4c8e-bad6-667ae406f301-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{"remote-ip-address":172.27.172.2, "forwarded-ip-addresses":[]","cluster_name":"CDP PvC Base Single-node Cluster","policy_version":1}Impala
2021-05-04T03:32:01-07:00 host1.example.com Impala: {"repoType":3,"repo":"cm_hive","reqUser":"admin","evtTime":"2021-05-04 03:31:54.666","access":"select","resource":"default/sample_07","resType":"@table","action":"select","result":1,"agent":"impala","policy":8,"enforcer":"ranger-acl","cliIP":"::ffff:172.27.172.2","reqData":"SELECT s07.description, s07.salary, s08.salary,\r s08.salary - s07.salary\r FROM\r sample_07 s07 JOIN sample_08 s08\r ON ( s07.code \u003d s08.code)\r WHERE\r s07.salary \u003c s08.salary\r ORDER BY s08.salary-s07.salary DESC\r LIMIT 1000","agentHost":"host1.example.com","logType":"RangerAudit","id":"f995bc52-dbdf-4617-96f6-61a176f6a727-0","seq_num":0,"event_count":1,"event_dur_ms":1,"tags":[],"cluster_name":"CDP PvC Base Single-node Cluster","policy_version":1}
2021-05-04T03:32:01-07:00 host1.example.com Impala:
Solr
在Solr审核中，默认情况下仅审核查询发生的事实：
{"repoType":8,"repo":"cm_solr","reqUser":"admin","evtTime":"2021-05-04 02:33:22.916","access":"query","resource":"twitter_demo","resType":"collection","action":"query","result":1,"agent":"solr","policy":39,"enforcer":"ranger-acl","cliIP":"172.27.172.2","agentHost":"host1.example.com","logType":"RangerAudit","id":"951c7dea-8ae7-49a5-8539-8c993651f75c-0","seq_num":1,"event_count":2,"event_dur_ms":199,"tags":[],"cluster_name":"CDP PvC Base Single-node Cluster","policy_version":2}但是，如果在Solr中启用了文档级授权，那么我们还将看到查询文本：
2021-05-04T06:23:00-07:00 host1.example.com Solr: {"repoType":8,"repo":"cm_solr","reqUser":"admin","evtTime":"2021-05-04 06:22:55.366","access":"query","resource":"testcollection","resType":"collection","action":"others","result":0,"agent":"solr","policy":-1,"enforcer":"ranger-acl","cliIP":"172.27.172.2","reqData":"{! q\u003dtext:mysearchstring doAs\u003dadmin df\u003d_text_ echoParams\u003dexplicit start\u003d0 rows\u003d100 wt\u003djson}","agentHost":"host1.example.com","logType":"RangerAudit","id":"6b14c79f-e30d-4635-bd07-a5d116ee4d0f-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"cluster_name":"CDP PvC Base Single-node Cluster"}Hue
这些行直接从Hue审核日志文件中记录。
{"username": "admin", "impersonator": "hue", "eventTime": 1620124241293, "operationText": "Successful login for user: admin", "service": "hue", "url": "/hue/accounts/login", "allowed": true, "operation": "USER_LOGIN", "ipAddress": "10.96.85.63"}
{"username": "admin", "impersonator": "hue", "eventTime": 1620131105118, "operationText": "User admin downloaded results from query-impala-46 as xls", "service": "notebook", "url": "/notebook/download", "allowed": true, "operation": "DOWNLOAD", "ipAddress": "10.96.85.63"}
总结
审计和核算是针对正在存储和处理客户、医疗保健、财务或专有信息的组织的法规安全控制，以防止内部人行为（无意和恶意）的威胁不断增加。
在此博客中，我们讨论了使用基于文件的审计和基于Syslog的审计生成将CDP中的审计事件发送到外部SIEM的方法。
有关配置和使用Apache Ranger的更多信息，请查阅CDP文档。
原文作者：Tristan Stevens
原文链接：https://blog.cloudera.com/auditing-to-external-systems-in-cdp-private-cloud-base/
