> 科技资讯 > 开发一个二方包,优雅地为系统接入ELK(elasticsearch+logstash+kibana)

开发一个二方包,优雅地为系统接入ELK(elasticsearch+logstash+kibana)

科技资讯 投稿 7900℃ 0 评论

开发一个二方包,优雅地为系统接入ELK(elasticsearch+logstash+kibana)

这篇文章主要介绍如何编写二方包，并整合到各个系统中。

xiaobawang-log主要收集三种日志类型：

    系统级别日志： 收集系统运行时产生的各个级别的日志（ERROR、INFO、WARN、DEBUG和TRACER），其中ERROR级别日志是我们最关心的。
  1. 用户请求日志： 主要用于controller层的请求，捕获用户请求信息和响应信息、以及来源ip等，便于分析用户行为。
  2. 自定义操作日志： 顾名思义，就是收集手动打的日志。比如定时器执行开始，都会习惯性写一个log.info("定时器执行开始！"的描述，这种就是属于自定义操作日志的类型。

二方包开发

废话不多说，上代码。
1、首先创建一个springboot项目，引入如下包：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.0.1</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-core</artifactId>
    <version>1.2.10</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.10</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-access</artifactId>
    <version>1.2.10</version>
</dependency>
<dependency>
    <groupId>cn.hutool</groupId>
    <artifactId>hutool-all</artifactId>
    <version>5.7.18</version>
</dependency>
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <optional>true</optional>
    <version>1.18.26</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

SysLog实体类

public class SysLog {

    /**
     * 日志名称
     */
    private String logName;

    /**
     * ip地址
     */
    private String ip;

    /**
     * 请求参数
     */
    private String requestParams;

    /**
     * 请求地址
     */
    private String requestUrl;

    /**
     * 用户ua信息
     */
    private String userAgent;

    /**
     * 请求时间
     */
    private Long useTime;

    /**
     * 请求时间
     */
    private String exceptionInfo;

    /**
     * 响应信息
     */
    private String responseInfo;

    /**
     * 用户名称
     */
    private String username;

    /**
     * 请求方式
     */
    private String requestMethod;

}

LogAction

创建一个枚举类，包含三种日志类型。

public enum LogAction {

    USER_ACTION("用户日志", "user-action",
    SYS_ACTION("系统日志", "sys-action",
    CUSTON_ACTION("其他日志", "custom-action";

    private final String action;

    private final String actionName;

    LogAction(String action,String actionName {
        this.action = action;
        this.actionName = actionName;
    }

    public String getAction( {
        return action;
    }

    public String getActionName( {
        return actionName;
    }

}

配置logstash

更改logstash配置文件，将index名称更改为log-%{[appname]}-%{+YYYY.MM.dd}-%{[action]}，其中appname为系统名称，action为日志类型。
整个es索引名称是以“系统名称+日期+日志类型”的形式。比如“mySystem-2023.03.05-system-action”表示这个索引，是由mySystem在2023年3月5日产生的系统级别的日志。

# 输入端
input {
  stdin { } 
  #为logstash增加tcp输入口，后面springboot接入会用到
  tcp {
      mode => "server"
      host => "0.0.0.0"
      port => 5043
      codec => json_lines
  }
}
 
#输出端
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["http://你的虚拟机ip地址:9200"]
    # 输出至elasticsearch中的自定义index名称
    index => "log-%{[appname]}-%{+YYYY.MM.dd}-%{[action]}"
  }
  stdout { codec => rubydebug }
}

AppenderBuilder

使用编程式配置logback，AppenderBuilder用于创建appender。

    这里会创建两种appender。consoleAppender负责将日志打印到控制台，这对开发来说是十分有用的。而LogstashTcpSocketAppender则负责将日志保存到ELK中。
  • setCustomFields中的参数，对应上面logstash配置文件的参数[appname]和[action]。
@Component
public class AppenderBuilder {

    public static final String SOCKET_ADDRESS = "你的虚拟机ip地址";

    public static final Integer PORT = 5043;//logstash tcp输入端口

    /**
     * logstash通信Appender
     * @param name
     * @param action
     * @param level
     * @return
     */
    public LogstashTcpSocketAppender logAppenderBuild(String name, String action, Level level {
        LoggerContext context = (LoggerContext LoggerFactory.getILoggerFactory(;
        LogstashTcpSocketAppender appender = new LogstashTcpSocketAppender(;
        appender.setContext(context;
        //设置logstash通信地址
        InetSocketAddress inetSocketAddress = new InetSocketAddress(SOCKET_ADDRESS, PORT;
        appender.addDestinations(inetSocketAddress;
        LogstashEncoder logstashEncoder = new LogstashEncoder(;
        //对应前面logstash配置文件里的参数
        logstashEncoder.setCustomFields("{\"appname\":\"" + name + "\",\"action\":\"" + action + "\"}";
        appender.setEncoder(logstashEncoder;

        //这里设置级别过滤器
        LevelFilter levelFilter = new LevelFilter(;
        levelFilter.setLevel(level;
        levelFilter.setOnMatch(ACCEPT;
        levelFilter.setOnMismatch(DENY;
        levelFilter.start(;
        appender.addFilter(levelFilter;
        appender.start(;

        return appender;
    }
    
    
    /**
     * 控制打印Appender
     * @return
     */
    public ConsoleAppender consoleAppenderBuild( {
        ConsoleAppender consoleAppender = new ConsoleAppender(;
        LoggerContext context = (LoggerContext LoggerFactory.getILoggerFactory(;
        PatternLayoutEncoder encoder = new PatternLayoutEncoder(;
        encoder.setContext(context;
        //设置格式
        encoder.setPattern("%red(%d{yyyy-MM-dd HH:mm:ss} %green([%thread] %highlight(%-5level %boldMagenta(%logger - %cyan(%msg%n";
        encoder.start(;
        consoleAppender.setEncoder(encoder;
        consoleAppender.start(;
        return consoleAppender;

    }

LoggerBuilder

LoggerBuilder主要用于创建logger类。创建步骤如下：

    获取logger上下文。
  1. 从上下文获取logger对象。创建过的logger会保存在LOGCONTAINER中，保证下次获取logger不会重复创建。这里使用ConcurrentHashMap防止出现并发问题。
  2. 创建appender，并将appender加入logger对象中。
@Component
public class LoggerBuilder {
    @Autowired
    AppenderBuilder appenderBuilder;

    @Value("${spring.application.name:unknow-system}"
    private String appName;

    private static final Map<String, Logger> LOGCONTAINER = new ConcurrentHashMap<>(;

    public Logger getLogger(LogAction logAction {
        Logger logger = LOGCONTAINER.get(logAction.getActionName( + "-" + appName;
        if (logger != null {
            return logger;
        }
        logger = build(logAction;
        LOGCONTAINER.put(logAction.getActionName( + "-" + appName, logger;

        return logger;
    }

    public Logger getLogger( {
        return getLogger(LogAction.CUSTON_ACTION;
    }

    private Logger build(LogAction logAction {
        //创建日志appender
        List<LogstashTcpSocketAppender> list = createAppender(appName, logAction.getActionName(;
        LoggerContext context = (LoggerContext LoggerFactory.getILoggerFactory(;
        Logger logger = context.getLogger(logAction.getActionName( + "-" + appName;
        logger.setAdditive(false;
        //打印控制台appender
        ConsoleAppender consoleAppender = appenderBuilder.consoleAppenderBuild(;
        logger.addAppender(consoleAppender;
        list.forEach(appender -> {
            logger.addAppender(appender;
        };
        return logger;
    }

    /**
     * LoggerContext上下文中的日志对象加入appender
     */
    public void addContextAppender( {
        //创建四种类型日志
        String action = LogAction.SYS_ACTION.getActionName(;
        List<LogstashTcpSocketAppender> list = createAppender(appName, action;
        LoggerContext context = (LoggerContext LoggerFactory.getILoggerFactory(;
        //打印控制台
        ConsoleAppender consoleAppender = appenderBuilder.consoleAppenderBuild(;
        context.getLoggerList(.forEach(logger -> {
            logger.setAdditive(false;
            logger.addAppender(consoleAppender;
            list.forEach(appender -> {
                logger.addAppender(appender;
            };
        };
    }

    /**
     * 创建连接elk的appender，每一种级别日志创建一个appender
     *
     * @param name
     * @param action
     * @return
     */
    public List<LogstashTcpSocketAppender> createAppender(String name, String action {
        List<LogstashTcpSocketAppender> list = new ArrayList<>(;
        LogstashTcpSocketAppender errorAppender = appenderBuilder.logAppenderBuild(name, action, Level.ERROR;
        LogstashTcpSocketAppender infoAppender = appenderBuilder.logAppenderBuild(name, action, Level.INFO;
        LogstashTcpSocketAppender warnAppender = appenderBuilder.logAppenderBuild(name, action, Level.WARN;
        LogstashTcpSocketAppender debugAppender = appenderBuilder.logAppenderBuild(name, action, Level.DEBUG;
        LogstashTcpSocketAppender traceAppender = appenderBuilder.logAppenderBuild(name, action, Level.TRACE;
        list.add(errorAppender;
        list.add(infoAppender;
        list.add(warnAppender;
        list.add(debugAppender;
        list.add(traceAppender;
        return list;
    }
}

LogAspect

使用spring aop，实现拦截用户请求，记录用户日志。比如ip、请求参数、请求用户等信息，需要配合下面的XiaoBaWangLog注解使用。
这里拦截上面所说的第二种日志类型。

@Aspect
@Component
public class LogAspect {

    @Autowired
    LoggerBuilder loggerBuilder;

    private ThreadLocal<Long> startTime = new ThreadLocal<>(;

    private SysLog sysLog;

    @Pointcut("@annotation(com.xiaobawang.common.log.annotation.XiaoBaWangLog"
    public void pointcut( {
    }

    /**
     * 前置方法执行
     *
     * @param joinPoint
     */
    @Before("pointcut("
    public void before(JoinPoint joinPoint {
        startTime.set(System.currentTimeMillis(;
        //获取请求的request
        ServletRequestAttributes attributes = (ServletRequestAttributes RequestContextHolder.getRequestAttributes(;
        HttpServletRequest request = attributes.getRequest(;
        String clientIP = ServletUtil.getClientIP(request, null;
        if ("0.0.0.0".equals(clientIP || "0:0:0:0:0:0:0:1".equals(clientIP || "localhost".equals(clientIP || "127.0.0.1".equals(clientIP {
            clientIP = "127.0.0.1";
        }
        sysLog = new SysLog(;
        sysLog.setIp(clientIP;
        String requestParams = JSONUtil.toJsonStr(getRequestParams(request;
        sysLog.setRequestParams(requestParams.length( > 5000 ? ("请求参数过长，参数长度为：" + requestParams.length( : requestParams;
        MethodSignature ms = (MethodSignature joinPoint.getSignature(;
        Method method = ms.getMethod(;
        String logName = method.getAnnotation(XiaoBaWangLog.class.value(;
        sysLog.setLogName(logName;
        sysLog.setUserAgent(request.getHeader("User-Agent";
        String fullUrl = request.getRequestURL(.toString(;
        if (request.getQueryString( != null && !"".equals(request.getQueryString( {
            fullUrl = request.getRequestURL(.toString( + "?" + request.getQueryString(;
        }
        sysLog.setRequestUrl(fullUrl;
        sysLog.setRequestMethod(request.getMethod(;
        //tkSysLog.setUsername(JwtUtils.getUsername(;
    }

    /**
     * 方法返回后执行
     *
     * @param ret
     */
    @AfterReturning(returning = "ret", pointcut = "pointcut("
    public void after(Object ret {
        Logger logger = loggerBuilder.getLogger(LogAction.USER_ACTION;
        String retJsonStr = JSONUtil.toJsonStr(ret;
        if (retJsonStr != null {
            sysLog.setResponseInfo(retJsonStr.length( > 5000 ? ("响应参数过长，参数长度为：" + retJsonStr.length( : retJsonStr;
        }
        sysLog.setUseTime(System.currentTimeMillis( - startTime.get(;
        logger.info(JSONUtil.toJsonStr(sysLog;
    }

    /**
     * 环绕通知,收集方法执行期间的错误信息
     *
     * @param proceedingJoinPoint
     * @return
     * @throws Throwable
     */
    @Around("pointcut("
    public Object around(ProceedingJoinPoint proceedingJoinPoint throws Throwable {

        try {
            Object obj = proceedingJoinPoint.proceed(;
            return obj;
        } catch (Exception e {
            e.printStackTrace(;
            sysLog.setExceptionInfo(e.getMessage(;
            Logger logger = loggerBuilder.getLogger(LogAction.USER_ACTION;
            logger.error(JSONUtil.toJsonStr(sysLog;
            throw e;
        }
    }

    /**
     * 获取请求的参数
     *
     * @param request
     * @return
     */
    private Map getRequestParams(HttpServletRequest request {
        Map map = new HashMap(;
        Enumeration paramNames = request.getParameterNames(;
        while (paramNames.hasMoreElements( {
            String paramName = (String paramNames.nextElement(;
            String[] paramValues = request.getParameterValues(paramName;
            if (paramValues.length == 1 {
                String paramValue = paramValues[0];
                if (paramValue.length( != 0 {
                    map.put(paramName, paramValue;
                }
            }
        }
        return map;
    }


}

XiaoBaWangLog

LoggerLoad主要是实现用户级别日志的收集功能。
这里定义了一个注解，在controller方法上加上@XiaoBaWangLog("操作内容",即可拦截并生成请求日志。

@Target({ElementType.TYPE, ElementType.METHOD}
@Retention(RetentionPolicy.RUNTIME
@Documented
@Component
public @interface XiaoBaWangLog {

    String value( default "";

}

LoggerLoad

LoggerLoad主要是实现系统级别日志的收集功能。
继承ApplicationRunner，可以在springboot执行后，自动创建系统级别日志logger对象。

@Component
@Order(value = 1
@Slf4j
public class LoggerLoad implements ApplicationRunner {
    @Autowired
    LoggerBuilder loggerBuilder;

    @Override
    public void run(ApplicationArguments args throws Exception {
        loggerBuilder.addContextAppender(;
        log.info("加载日志模块成功";
    }
}

LogConfig

LogConfig主要实现自定义级别日志的收集功能。
生成一个logger对象交给spring容器管理。后面直接从容器取就可以了。

@Configuration
public class LogConfig {

    @Autowired
    LoggerBuilder loggerBuilder;

    @Bean
    public Logger loggerBean({
        return loggerBuilder.getLogger(;
    }
}

代码到现在已经全部完成，怎么将上述的所有Bean加入到spring呢？这个时候就需要用到spring.factories了。

spring.factories

注意，spring.factories在2.7开始已经不推荐使用，3.X版本的springBoot是不支持使用的。

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.xiaobawang.common.log.config.AppenderBuilder,\
  com.xiaobawang.common.log.config.LoggerBuilder,\
  com.xiaobawang.common.log.load.LoggerLoad,\
  com.xiaobawang.common.log.aspect.LogAspect,\
  com.xiaobawang.common.log.config.LogConfig

测试

先将xiaobawang进行打包
新建一个springboot项目，引入打包好的xiaobawang-log。

运行springboot，出现“加载日志模块成功”表示日志模块启动成功。

结束

如果这篇文章对你有帮助，记得一键三连~

编程笔记 » 开发一个二方包,优雅地为系统接入ELK(elasticsearch+logstash+kibana)

赞同 (31) or 分享 (0)
游客 发表我的评论   换个身份
取消评论

表情
(0)个小伙伴在吐槽