loadClass导致线上服务卡顿分析

发表于 阅读次数：

一个线上服务偶尔卡顿，分析发现是loadClass导致的线程阻塞，而loadClass的原因与Feign配置有关。

现象

最近线上的一个服务偶尔出现卡顿，表现为不特定时刻出现几分钟的异常，在这段时间内响应时间激增，如图：

IMAGE

分析

首先查看日志，找到慢请求，发现在服务间Feign调用后会出现一段时间的间隔。对Feign的Logger和HttpMessageConverterExtractor开启DEBUG日志，看到在Feign输出HTTP返回数据后到Jackson反序列化之间有几秒的间隔。

1
2
2019-12-20 10:25:55.493|*,*,*|DEBUG|feign.slf4j.Slf4jLogger:(72)|[ServiceName#methodName] <--- END HTTP (1571-byte body)
2019-12-20 10:25:56.830|*,*,*|DEBUG|org.springframework.web.client.HttpMessageConverterExtractor:(100)|Reading to [com.*.DataModel<com.*.BusinessModel>]

这点很奇怪，在GC日志中也没有Stop-The-World出现。用jstack打印堆栈，发现有几个和Feign相关的线程处于BLOCKED状态：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
"http-nio-8080-exec-276" #3021 daemon prio=5 os_prio=0 tid=0x00007fbd501a8800 nid=0xdea waiting for monitor entry [0x00007fbcbd76d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
	- waiting to lock <0x00000006c6ed91d0> (a java.lang.Object)
	at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:276)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:797)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:650)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:633)
	at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.<init>(MappingJackson2HttpMessageConverter.java:59)
	at org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter.<init>(AllEncompassingFormHttpMessageConverter.java:76)
	at org.springframework.web.servlet.config.annotation.WebMvcConfigurationSupport.addDefaultHttpMessageConverters(WebMvcConfigurationSupport.java:796)
	at org.springframework.web.servlet.config.annotation.WebMvcConfigurationSupport.getMessageConverters(WebMvcConfigurationSupport.java:748)
	at org.springframework.boot.autoconfigure.http.HttpMessageConverters$1.defaultMessageConverters(HttpMessageConverters.java:185)
	at org.springframework.boot.autoconfigure.http.HttpMessageConverters.getDefaultConverters(HttpMessageConverters.java:188)
	at org.springframework.boot.autoconfigure.http.HttpMessageConverters.<init>(HttpMessageConverters.java:105)
	at org.springframework.boot.autoconfigure.http.HttpMessageConverters.<init>(HttpMessageConverters.java:92)
	at org.springframework.boot.autoconfigure.http.HttpMessageConverters.<init>(HttpMessageConverters.java:80)
	at com.yirendai.app.fortune.support.config.FeignConfig.lambda$feignDecoder$0(FeignConfig.java:26)
	at com.yirendai.app.fortune.support.config.FeignConfig$$Lambda$536/1366741625.getObject(Unknown Source)
	at org.springframework.cloud.openfeign.support.SpringDecoder.decode(SpringDecoder.java:57)
	at org.springframework.cloud.openfeign.support.ResponseEntityDecoder.decode(ResponseEntityDecoder.java:62)
	at feign.optionals.OptionalDecoder.decode(OptionalDecoder.java:36)
	at feign.SynchronousMethodHandler.decode(SynchronousMethodHandler.java:178)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:142)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
	at com.sun.proxy.$Proxy178.searchUserTag(Unknown Source)

查看registerWellKnownModulesIfAvailable处的代码，可以看到其逻辑为若classpath中有JodaTime的LocalDate，则加载Jackson对应的JodaModule（这个项目中没有引用）。

1
2
3
4
5
6
7
8
9
10
if (ClassUtils.isPresent("org.joda.time.LocalDate", this.moduleClassLoader)) {
    try {
        Class<? extends Module> jodaModuleClass = (Class<? extends Module>)
            ClassUtils.forName("com.fasterxml.jackson.datatype.joda.JodaModule", this.moduleClassLoader);
        Module jodaModule = BeanUtils.instantiateClass(jodaModuleClass);
        modulesToRegister.set(jodaModule.getTypeId(), jodaModule);
    } catch (ClassNotFoundException ex) {
        // jackson-datatype-joda not available
    }
}

LaunchedURLClassLoader.loadClass将调用ClassLoader.loadClass来加载类，加载时需要获取锁，因此在并发环境下，可能导致线程BLOCKED状态。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
    synchronized (getClassLoadingLock(name)) {
        // First, check if the class has already been loaded
        Class<?> c = findLoadedClass(name);
        if (c == null) {
            // 省略类加载代码
        }
        if (resolve) {
            resolveClass(c);
        }
        return c;
    }
}

protected Object getClassLoadingLock(String className) {
    Object lock = this;
    if (parallelLockMap != null) {
        Object newLock = new Object();
        lock = parallelLockMap.putIfAbsent(className, newLock);
        if (lock == null) {
            lock = newLock;
        }
    }
    return lock;
}

依据以上信息，出现卡顿的流程大致如下：

  1. Feign请求时会初始化MappingJackson2HttpMessageConverter时尝试加载JodaModule。
  2. 而这个类并不在classpath中，因此无法在findLoadedClass中找到，每次都需要重新加载。
  3. 执行loadClass时需要加锁，在线上高并发场景下会导致线程BLOCKED状态。

解决方式一：避免ClassLoader反复加载

可以看出卡顿的直接原因是反复尝试加载不在classpath中的JodaModule，因此将这个依赖添加到工程中。加载一次后，再次调用可以通过findLoadedClass获得，减少加载类导致的资源消耗，从而减少BLOCKED的出现。

1
2
3
4
5
<dependency>
    <groupId>com.fasterxml.jackson.datatype</groupId>
    <artifactId>jackson-datatype-joda</artifactId>
    <version>x.x.x</version>
</dependency>

解决方式二：避免HttpMessageConverters重复初始化

但是还有另一个问题需要考虑：为什么每次请求都会初始化MappingJackson2HttpMessageConverter？查看SpringDecoder代码，可以看到每次反序列化response时会调用ObjectFactory<HttpMessageConverters>来获取converters。

1
2
3
4
5
6
7
8
9
10
11
12
13
@Override
public Object decode(final Response response, Type type) throws IOException, FeignException {
    if (type instanceof Class || type instanceof ParameterizedType || type instanceof WildcardType) {
        @SuppressWarnings({ "unchecked", "rawtypes" })
        HttpMessageConverterExtractor<?> extractor = new HttpMessageConverterExtractor(
            type, this.messageConverters.getObject().getConverters());

        return extractor.extractData(new FeignResponseAdapter(response));
    }
    throw new DecodeException(response.status(),
        "type is not an instance of Class or ParameterizedType: " + type,
        response.request());
}

而在FeignConfig中配置的这个ObjectFactory的实现是new一个HttpMessageConverters对象。

1
2
3
4
5
6
7
8
9
@Bean
public Decoder feignDecoder() {
    ObjectMapper mapper = new ObjectMapper()
            .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

    HttpMessageConverter jacksonConverter = new MappingJackson2HttpMessageConverter(mapper);
    ObjectFactory<HttpMessageConverters> objectFactory = () -> new HttpMessageConverters(jacksonConverter);
    return new OptionalDecoder(new ResponseEntityDecoder(new SpringDecoder(objectFactory)));
}

HttpMessageConverters的构造方法会默认执行getDefaultConverters。其逻辑可查看WebMvcConfigurationSupport代码，其中AllEncompassingFormHttpMessageConverter的构造函数会创建MappingJackson2HttpMessageConverter对象。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public HttpMessageConverters(HttpMessageConverter<?>... additionalConverters) {
    this(Arrays.asList(additionalConverters));
}

public HttpMessageConverters(Collection<HttpMessageConverter<?>> additionalConverters) {
    this(true, additionalConverters);
}

public HttpMessageConverters(boolean addDefaultConverters, Collection<HttpMessageConverter<?>> converters) {
    List<HttpMessageConverter<?>> combined = getCombinedConverters(converters,
        addDefaultConverters ? getDefaultConverters() : Collections.emptyList());
    combined = postProcessConverters(combined);
    this.converters = Collections.unmodifiableList(combined);
}

这就是每一个请求都会初始化MappingJackson2HttpMessageConverter并触发loadClass的原因，因此每一个Feign请求的开销都很大。由于我们只需要使用自定义的MappingJackson2HttpMessageConverter来执行反序列化，可以想办法避免执行getDefaultConverters：

第一种方法是指定HttpMessageConverters的构造方法参数addDefaultConverters为false：

1
ObjectFactory<HttpMessageConverters> objectFactory = () -> new HttpMessageConverters(false, Collections.singletonList(jacksonConverter));

第二种方法则是使用Feign的JacksonDecoder：

1
2
3
4
5
6
@Bean
public Decoder feignDecoder() {
    ObjectMapper mapper = new ObjectMapper()
            .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    return new JacksonDecoder(mapper);
}
1
2
3
4
5
<dependency>
    <groupId>io.github.openfeign</groupId>
    <artifactId>feign-jackson</artifactId>
    <version>x.x.x</version>
</dependency>