Java分布式跟踪系统Zipkin（四）：Brave源码分析-HttpTracing

所有博文均在个人独立博客http://blog.mozhu.org首发，欢迎访问！

上一篇博文中，我们分析了Tracing的相关源代码，这一篇我们来看看Brave是如何在Web项目中使用的

我们先来看看普通的servlet项目中，如何使用Brave，这对我们后面分析和理解Brave和SpringMVC等框架整合有帮助

首先Chapter1/servlet25项目中配置了FrontServlet和BackendServlet以及TracingFilter

web.xml

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" version="2.5">

<display-name>Servlet2.5 Application</display-name>

<filter>
<filter-name>TracingFilter</filter-name>
<filter-class>org.mozhu.zipkin.filter.BraveTracingFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>TracingFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>

<servlet>
<servlet-name>BackendServlet</servlet-name>
<servlet-class>org.mozhu.zipkin.servlet.BackendServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>BackendServlet</servlet-name>
<url-pattern>/api</url-pattern>
</servlet-mapping>

<servlet>
<servlet-name>FrontendServlet</servlet-name>
<servlet-class>org.mozhu.zipkin.servlet.FrontendServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>FrontendServlet</servlet-name>
<url-pattern>/</url-pattern>
</servlet-mapping>
</web-app>

TracingFilter

我们使用自定义的BraveTracingFilter作为入口，其init方法中，我们初始化了Tracing，然后创建HttpTracing对象，最后调用TracingFilter.create(httpTracing)创建了tracingFilter。

doFilter方法中，所有请求将被tracingFilter来处理

BraveTracingFilter

package org.mozhu.zipkin.filter;

import brave.Tracing;
import brave.context.log4j2.ThreadContextCurrentTraceContext;
import brave.http.HttpTracing;
import brave.propagation.B3Propagation;
import brave.propagation.ExtraFieldPropagation;
import brave.servlet.TracingFilter;
import zipkin2.codec.SpanBytesEncoder;
import zipkin2.reporter.AsyncReporter;
import zipkin2.reporter.Sender;
import zipkin2.reporter.okhttp3.OkHttpSender;

import javax.servlet.*;
import java.io.IOException;
import java.util.concurrent.TimeUnit;

public class BraveTracingFilter implements Filter {
Filter tracingFilter;

@Override
public void init(FilterConfig filterConfig) throws ServletException {
Sender sender = OkHttpSender.create("http://localhost:9411/api/v2/spans");
AsyncReporter asyncReporter = AsyncReporter.builder(sender)
.closeTimeout(500, TimeUnit.MILLISECONDS)
.build(SpanBytesEncoder.JSON_V2);

Tracing tracing = Tracing.newBuilder()
.localServiceName(System.getProperty("zipkin.service", "servlet25-demo"))
.spanReporter(asyncReporter)
.propagationFactory(ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "user-name"))
.currentTraceContext(ThreadContextCurrentTraceContext.create())
.build();

HttpTracing httpTracing = HttpTracing.create(tracing);
filterConfig.getServletContext().setAttribute("TRACING", httpTracing);
tracingFilter = TracingFilter.create(httpTracing);
tracingFilter.init(filterConfig);
}

@Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
tracingFilter.doFilter(servletRequest, servletResponse, filterChain);
}

@Override
public void destroy() {
tracingFilter.destroy();
}

}

TracingFilter

TracingFilter在brave-instrumentation-servlet包中

public final class TracingFilter implements Filter {
public static Filter create(Tracing tracing) {
return new TracingFilter(HttpTracing.create(tracing));
}

public static Filter create(HttpTracing httpTracing) {
return new TracingFilter(httpTracing);
}

final ServletRuntime servlet = ServletRuntime.get();
final Tracer tracer;
final HttpServerHandler<HttpServletRequest, HttpServletResponse> handler;
final TraceContext.Extractor<HttpServletRequest> extractor;

TracingFilter(HttpTracing httpTracing) {
tracer = httpTracing.tracing().tracer();
handler = HttpServerHandler.create(httpTracing, new HttpServletAdapter());
extractor = httpTracing.tracing().propagation().extractor(HttpServletRequest::getHeader);
}
}

TracingFilter中几个重要的类

- HttpTracing - 包含Http处理相关的组件，clientParser，serverParser，clientSampler，serverSampler

- ServletRuntime - Servlet运行时类，包含根据环境来判断是否支持Servlet3异步调用等方法

- HttpServerHandler - Http处理的核心组件，基本上所有和trace相关的操作均在此类中完成

- HttpServletAdapter - HttpServlet的适配器接口，此类的引入可以让httpServerHandler类变得更为通用，因为它是一个泛型接口，跟具体的request和response无关，能和更多框架进行整合

- TraceContext.Extractor - TraceContext的数据提取器

doFilter方法

@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
HttpServletResponse httpResponse = servlet.httpResponse(response);

Span span = handler.handleReceive(extractor, httpRequest);
Throwable error = null;
try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
chain.doFilter(httpRequest, httpResponse); // any downstream filters see Tracer.currentSpan
} catch (IOException | ServletException | RuntimeException | Error e) {
error = e;
throw e;
} finally {
if (servlet.isAsync(httpRequest)) { // we don't have the actual response, handle later
servlet.handleAsync(handler, httpRequest, span);
} else { // we have a synchronous response, so we can finish the span
handler.handleSend(httpResponse, error, span);
}
}
}

首先调用handler.handleReceive(extractor, httpRequest)从request中提取Span信息

然后调用tracer.withSpanInScope(span)将Span包装成Tracer.SpanInScope，而Tracer.SpanInScope和前面博文中分析的CurrentTraceContext.Scope比较像，都实现了Closeable接口，这里的目的也一样，都是为了利用JDK7的try-with-resources的特性，JVM会自动调用close方法，做一些线程对象的清理工作。其区别是后者是SPI（Service Provider Interface），不适合暴露给真正的使用者。

这样使得chain.doFilter(httpRequest, httpResponse)里的代码能用Tracer.currentSpan拿到从请求中提取（extract）的Span信息。

最后调用handler.handleSend(httpResponse, error, span)

下面来仔细分析下handler中handleReceive和handleSend两个方法

handleReceive方法

public Span handleReceive(TraceContext.Extractor<Req> extractor, Req request) {
return handleReceive(extractor, request, request);
}

public <C> Span handleReceive(TraceContext.Extractor<C> extractor, C carrier, Req request) {
Span span = nextSpan(extractor.extract(carrier), request);
if (span.isNoop()) return span;

// all of the parsing here occur before a timestamp is recorded on the span
span.kind(Span.Kind.SERVER);

// Ensure user-code can read the current trace context
Tracer.SpanInScope ws = tracer.withSpanInScope(span);
try {
parser.request(adapter, request, span);
} finally {
ws.close();
}

boolean parsedEndpoint = false;
if (Platform.get().zipkinV1Present()) {
zipkin.Endpoint.Builder deprecatedEndpoint = zipkin.Endpoint.builder().serviceName("");
if ((parsedEndpoint = adapter.parseClientAddress(request, deprecatedEndpoint))) {
span.remoteEndpoint(deprecatedEndpoint.build());
}
}
if (!parsedEndpoint) {
Endpoint.Builder remoteEndpoint = Endpoint.newBuilder();
if (adapter.parseClientAddress(request, remoteEndpoint)) {
span.remoteEndpoint(remoteEndpoint.build());
}
}
return span.start();
}

首先调用nextSpan(extractor.extract(carrier), request)从request中提取TraceContextOrSamplingFlags，并创建Span，并将Span的kind类型设置为SERVER

然后调用parser.request(adapter, request, span)，将request的内容，将span的name改为request的method即GET或者POST，而且会将当前请求的路径以Tag(http.path)写入Span中，这样我们就能在Zipkin的UI界面中能清晰的看出某个Span是发起了什么请求。

最后为Span设置Endpoint信息，并调用start设置开始时间

handleSend方法

public void handleSend(@Nullable Resp response, @Nullable Throwable error, Span span) {
if (span.isNoop()) return;

// Ensure user-code can read the current trace context
Tracer.SpanInScope ws = tracer.withSpanInScope(span);
try {
parser.response(adapter, response, error, span);
} finally {
ws.close();
span.finish();
}
}

handleSend比较简单，调用parser.response(adapter, response, error, span)，会将HTTP状态码写入Span的Tag(http.status_code)中，如果有出错，则会将错误信息写入Tag(error)中

最后会调用Span的finish方法，而finish方法中，会调用Reporter的report方法将Span信息上报到Zipkin。

接着看下nextSpan方法

Span nextSpan(TraceContextOrSamplingFlags extracted, Req request) {
if (extracted.sampled() == null) { // Otherwise, try to make a new decision
extracted = extracted.sampled(sampler.trySample(adapter, request));
}
return extracted.context() != null
? tracer.joinSpan(extracted.context())
: tracer.nextSpan(extracted);
}

从请求里提取的对象extracted（TraceContextOrSamplingFlags），如果没有sampled信息，则由HttpSampler的trySample方法来决定是否采样

如果extracted中含有TraceContext信息，则由tracer调用joinSpan，加入已存在的trace，这种情况一般是客户端代码使用将trace信息放入header，而服务端收到请求后，则自动加入客户端发起的trace中，所以当backend的请求运行到这段代码，会joinSpan

如果extracted中不含TraceContext信息，则由tracer调用nextSpan，这种情况一般是我们用户发起的请求，比如浏览器发起，则请求header中肯定是没有trace信息的，所以当frontend的请求运行到这段代码，会新建一个span

joinSpan方法

public final Span joinSpan(TraceContext context) {
if (context == null) throw new NullPointerException("context == null");
if (!supportsJoin) return newChild(context);
// If we are joining a trace, we are sharing IDs with the caller
// If the sampled flag was left unset, we need to make the decision here
TraceContext.Builder builder = context.toBuilder();
if (context.sampled() == null) {
builder.sampled(sampler.isSampled(context.traceId()));
} else {
builder.shared(true);
}
return toSpan(builder.build());
}

public Span newChild(TraceContext parent) {
if (parent == null) throw new NullPointerException("parent == null");
return nextSpan(TraceContextOrSamplingFlags.create(parent));
}

在joinSpan方法中，会共享调用方的traceId，如果调用者没有传入sampled信息，则由服务端自己决定是否采样，即sampler.isSampled(context.traceId())

nextSpan方法

public Span nextSpan(TraceContextOrSamplingFlags extracted) {
TraceContext parent = extracted.context();
if (extracted.samplingFlags() != null) {
TraceContext implicitParent = currentTraceContext.get();
if (implicitParent == null) {
return toSpan(newRootContext(extracted.samplingFlags(), extracted.extra()));
}
// fall through, with an implicit parent, not an extracted one
parent = appendExtra(implicitParent, extracted.extra());
}
long nextId = Platform.get().randomLong();
if (parent != null) {
return toSpan(parent.toBuilder() // copies "extra" from the parent
.spanId(nextId)
.parentId(parent.spanId())
.shared(false)
.build());
}
TraceIdContext traceIdContext = extracted.traceIdContext();
if (extracted.traceIdContext() != null) {
Boolean sampled = traceIdContext.sampled();
if (sampled == null) sampled = sampler.isSampled(traceIdContext.traceId());
return toSpan(TraceContext.newBuilder()
.sampled(sampled)
.debug(traceIdContext.debug())
.traceIdHigh(traceIdContext.traceIdHigh()).traceId(traceIdContext.traceId())
.spanId(nextId)
.extra(extracted.extra()).build());
}
// TraceContextOrSamplingFlags is a union of 3 types, we've checked all three
throw new AssertionError("should not reach here");
}

在nextSpan方法中，首先找出合适的parent，当parent存在时，则新建一个child Span，否则返回new Span

到这里服务端接受到请求后，是如何记录Span信息的代码已经分析完毕，接下来我们看看作为客户端，我们是如何上报Span信息

FrontServlet

首先我们看到FrontServet中init方法里，我们初始化了OkHttpClient，并将TracingInterceptor拦截器添加到OkHttpClient的NetworkInterceptor拦截器栈中，然后还用CurrentTraceContext中的ExecutorService的包装方法，将Dispatcher中的ExecutorService包装后设置到OkHttpClient中。

package org.mozhu.zipkin.servlet;

import brave.http.HttpTracing;
import brave.okhttp3.TracingInterceptor;
import okhttp3.Dispatcher;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;

public class FrontendServlet extends HttpServlet {

private final static Logger LOGGER = LoggerFactory.getLogger(FrontendServlet.class);

private OkHttpClient client;

@Override
public void init(ServletConfig config) throws ServletException {
super.init(config);
HttpTracing httpTracing = (HttpTracing) config.getServletContext().getAttribute("TRACING");
client = new OkHttpClient.Builder()
.dispatcher(new Dispatcher(
httpTracing.tracing().currentTraceContext()
.executorService(new Dispatcher().executorService())
))
.addNetworkInterceptor(TracingInterceptor.create(httpTracing))
.build();
}

@Override
protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
LOGGER.info("frontend receive request");
Request request = new Request.Builder()
.url("http://localhost:9000/api")
.build();

Response response = client.newCall(request).execute();
if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);

PrintWriter writer = resp.getWriter();
writer.write(response.body().string());
writer.flush();
writer.close();
}

}

public final class TracingInterceptor implements Interceptor {
// ...

final Tracer tracer;
final String remoteServiceName;
final HttpClientHandler<Request, Response> handler;
final TraceContext.Injector<Request.Builder> injector;

TracingInterceptor(HttpTracing httpTracing) {
if (httpTracing == null) throw new NullPointerException("HttpTracing == null");
tracer = httpTracing.tracing().tracer();
remoteServiceName = httpTracing.serverName();
handler = HttpClientHandler.create(httpTracing, new HttpAdapter());
injector = httpTracing.tracing().propagation().injector(SETTER);
}
}

TracingInterceptor中依赖Tracer，TraceContext.Injector，HttpClientHandler，HttpAdapter。

- TraceContext.Injector - 将Trace信息注入到HTTP Request中，即放到Http headers中

- HttpClientHandler - 和HttpServerHandler对应，也是Http处理的核心组件，基本上所有和trace相关的操作均在此类中完成

- HttpAdapter - 能从Http request中获得各种数据，比如method，请求Path，header值等

@Override public Response intercept(Chain chain) throws IOException {
Request request = chain.request();
Request.Builder requestBuilder = request.newBuilder();

Span span = handler.handleSend(injector, requestBuilder, request);
parseServerAddress(chain.connection(), span);
Response response = null;
Throwable error = null;
try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
return response = chain.proceed(requestBuilder.build());
} catch (IOException | RuntimeException | Error e) {
error = e;
throw e;
} finally {
handler.handleReceive(response, error, span);
}
}

这里代码和TracingFilter中doFilter比较相似，是一个相反的过程

- 首先将trace信息注入到request中，并创建Span对象

- 然后调用chain.proceed(requestBuilder.build())来执行发送http请求

- 最后handler.handleReceive(response, error, span)

接下来看看HttpClientHandler的handleSend方法和handleReceive方法

handleSend方法

public Span handleSend(TraceContext.Injector<Req> injector, Req request, Span span) {
return handleSend(injector, request, request, span);
}

public <C> Span handleSend(TraceContext.Injector<C> injector, C carrier, Req request, Span span) {
injector.inject(span.context(), carrier);
if (span.isNoop()) return span;

// all of the parsing here occur before a timestamp is recorded on the span
span.kind(Span.Kind.CLIENT);

// Ensure user-code can read the current trace context
Tracer.SpanInScope ws = tracer.withSpanInScope(span);
try {
parser.request(adapter, request, span);
} finally {
ws.close();
}

boolean parsedEndpoint = false;
if (Platform.get().zipkinV1Present()) {
zipkin.Endpoint.Builder deprecatedEndpoint = zipkin.Endpoint.builder()
.serviceName(serverNameSet ? serverName : "");
if ((parsedEndpoint = adapter.parseServerAddress(request, deprecatedEndpoint))) {
span.remoteEndpoint(deprecatedEndpoint.serviceName(serverName).build());
}
}
if (!parsedEndpoint) {
Endpoint.Builder remoteEndpoint = Endpoint.newBuilder().serviceName(serverName);
if (adapter.parseServerAddress(request, remoteEndpoint) || serverNameSet) {
span.remoteEndpoint(remoteEndpoint.build());
}
}
return span.start();
}

首先调用injector.inject(span.context(), carrier)将Trace信息注入request中，并将Span的kind类型设置为CLIENT

然后调用parser.request(adapter, request, span)，将request的内容，将span的name改为request的method即GET或者POST，而且会将当前请求的路径以Tag(http.path)写入Span中，这样我们就能在Zipkin的UI界面中能清晰的看出某个Span是发起了什么请求。

最后为Span设置Endpoint信息，并调用start设置开始时间

handleReceive方法

public void handleReceive(@Nullable Resp response, @Nullable Throwable error, Span span) {
if (span.isNoop()) return;
Tracer.SpanInScope ws = tracer.withSpanInScope(span);
try {
parser.response(adapter, response, error, span);
} finally {
ws.close();
span.finish();
}
}

handleReceive比较简单，当客户端收到服务端的响应后handleReceive方法会被调用，即调用parser.response(adapter, response, error, span)，会将HTTP状态码写入Span的Tag(http.status_code)中，如果有出错，则会将错误信息写入Tag(error)中

最后会调用Span的finish方法，而finish方法中，会调用Reporter的report方法将Span信息上报到Zipkin。

BackendServlet

最后看看BackendServlet，在收到请求后，将请求的header中参数user-name取出，添加到时间戳字符串尾部，并返回。

在上一篇博文中，我们看到如果我们向Frontend发送的请求中带有header user-name参数，Frontend会将这个值传递给Backend，然后backend会将它放到响应字符串中返回，以表明接收到该header。

package org.mozhu.zipkin.servlet;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Date;

public class BackendServlet extends HttpServlet {

private final static Logger LOGGER = LoggerFactory.getLogger(BackendServlet.class);

@Override
protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
LOGGER.info("backend receive request");
String username = req.getHeader("user-name");
String result;
if (username != null) {
result = new Date().toString() + " " + username;
} else {
result = new Date().toString();
}
PrintWriter writer = resp.getWriter();
writer.write(result);
writer.flush();
writer.close();
}

}

至此，我们已经分析完Brave是如何在普通的web项目中使用的，分析了TracingFilter拦截请求处理请求的逻辑，也分析了OkHttpClient是如何将Trace信息放入request中的。

后面博文中，我们还会继续分析Brave和Spring Web项目的整合方法。