您的位置：首页 > 其它

Knowing how all your components work together: distributed tracing with Zipkin

2017-04-19 18:12 405 查看
转自:http://aredko.blogspot.com/2014/02/knowing-how-all-your-components-work.html
Intoday'spostwewilltrytocoververyinterestingandimportanttopic:distributedsystemtracing.Whatitpracticallymeansisthatwewilltrytotracetherequestfromthepointitwasissuedbytheclienttothepointtheresponsetothisrequestwasreceived.Atfirst,itlooksquitestraightforwardbutinrealityitmayinvolvemanycallstoseveralothersystems,databases,NoSQLstores,caches,younameit...

In2010GooglepublishedapaperaboutDapper,alarge-scaledistributedsystemstracinginfrastructure(veryinterestingreadingbytheway).Lateron,TwitterbuiltitsownimplementationbasedonDapperpaper,calledZipkinandthat'stheonewearegoingtolookat.

WewillbuildasimpleJAX-RS2.0serverusinggreatApacheCXFlibrary.Fortheclientside,wewilluseJAX-RS2.0clientAPIandbyutilizingZipkinwewilltracealltheinteractionsbetweentheclientandtheserver(aswellaseverythinghappeningonserverside).Tomakeanexampleabitmoreillustrative,wewillpretendthatserverusessomekindofdatabasetoretrievethedata.OurcodewillbeamixofpureJavaandabitofScala(thechoiceofScalawillbeclearedupsoon).

OneadditionaldependencyinorderforZipkintoworkisApacheZookeeper.Itisrequiredforcoordinationandshouldbestartedinadvance.Luckily,itisveryeasytodo:

downloadthereleasefromhttp://zookeeper.apache.org/releases.html(thecurrentstableversionatthemomentofwritingis3.4.5)

unpackitintozookeeper-3.4.5

copyzookeeper-3.4.5/conf/zoo_sample.cfgtozookeeper-3.4.5/conf/zoo.cfg

andjuststartApacheZookeeperserver
Windows:zookeeper-3.4.5/bin/zkServer.cmd
Linux:zookeeper-3.4.5/bin/zkServer.shstart


NowbacktoZipkin.ZipkiniswritteninScala.ItisstillinactivedevelopmentandthebestwaytostartoffwithitisjustbycloningitsGitHubrepositoryandbuilditfromsources:

gitclonehttps://github.com/twitter/zipkin.git

Fromarchitecturalprospective,Zipkinconsistsofthreemaincomponents:

collector:collectstracesacrossthesystem

query:queriescollectedtraces

web:providesweb-basedUItoshowthetraces

Torunthem,ZipkinguysprovideusefulscriptsinthebinfolderwiththeonlyrequirementthatJDK1.7shouldbeinstalled:

bin/collector

bin/query

bin/web

Let'sexecutethesescriptsandensurethateverycomponenthasbeenstartedsuccessfully,withnostacktracesontheconsole(forcuriousreaders,IwasnotabletomakeZipkinworkonWindowssoIassumewearerunningitonLinuxbox).Bydefault,ZipkinwebUIisavailableonport8080.ThestoragefortracesisembeddedSQLiteengine.Thoughitworks,thebetterstorages(likeawesomeRedis)areavailable.

Thepreparationisover,let'sdosomecode.WewillstartwithJAX-RS2.0clientpartasit'sverystraightforward(ClientStarter.java):

01
package
com.example.client;
02
03
import
javax.ws.rs.client.Client;
04
import
javax.ws.rs.client.ClientBuilder;
05
import
javax.ws.rs.core.MediaType;
06
import
javax.ws.rs.core.Response;
07
08
import
com.example.zipkin.Zipkin;
09
import
com.example.zipkin.client.ZipkinRequestFilter;
10
import
com.example.zipkin.client.ZipkinResponseFilter;
11
12
public
class
ClientStarter{
13
public
static
void
main(
final
String[]args)
throws
Exception{
14
final
Clientclient=ClientBuilder
15
.newClient()
16
.register(
new
ZipkinRequestFilter(
"People"
,Zipkin.tracer()),
1
)
17
.register(
new
ZipkinResponseFilter(
"People"
,Zipkin.tracer()),
1
);
18
19
final
Responseresponse=client
20
.target(
"http://localhost:8080/rest/api/people"
)
21
.request(MediaType.APPLICATION_JSON)
22
.get();
23
24
if
(response.getStatus()==
200
){
25
System.out.println(response.readEntity(String.
class
));
26
}
27
28
response.close();
29
client.close();
30
31
//Smalldelaytoallowtracertosendthetraceoverthewire
32
Thread.sleep(
1000
);
33
}
34
}
ExceptacoupleofimportsandclasseswithZipkininit,everythingshouldlooksimple.SowhatthoseZipkinRequestFilterandZipkinResponseFilterarefor?Zipkinisawesomebutit'snotamagicaltool.Inordertotraceanyrequestindistributedsystem,thereshouldbesomecontextpassedalongwithit.InREST/HTTPworld,it'susuallyrequest/responseheaders.Let'stakealookonZipkinRequestFilterfirst(ZipkinRequestFilter.scala):

01
package
com.example.zipkin.client
02
03
import
javax.ws.rs.client.ClientRequestFilter
04
import
javax.ws.rs.ext.Provider
05
import
javax.ws.rs.client.ClientRequestContext
06
import
com.twitter.finagle.http.HttpTracing
07
import
com.twitter.finagle.tracing.Trace
08
import
com.twitter.finagle.tracing.Annotation
09
import
com.twitter.finagle.tracing.TraceId
10
import
com.twitter.finagle.tracing.Tracer
11
12
@Provider
13
class
ZipkinRequestFilter(valname:String,valtracer:Tracer)
extends
ClientRequestFilter{
14
deffilter(requestContext:ClientRequestContext):Unit={
15
Trace.pushTracerAndSetNextId(tracer,
true
)
16
17
requestContext.getHeaders().add(HttpTracing.Header.TraceId,Trace.id.traceId.toString)
18
requestContext.getHeaders().add(HttpTracing.Header.SpanId,Trace.id.spanId.toString)
19
20
Trace.id._parentIdforeach{id=>
21
requestContext.getHeaders().add(HttpTracing.Header.ParentSpanId,id.toString)
22
}
23
24
Trace.id.sampledforeach{sampled=>
25
requestContext.getHeaders().add(HttpTracing.Header.Sampled,sampled.toString)
26
}
27
28
requestContext.getHeaders().add(HttpTracing.Header.Flags,Trace.id.flags.toLong.toString)
29
30
if
(Trace.isActivelyTracing){
31
Trace.recordRpcname(name,requestContext.getMethod())
32
Trace.recordBinary(
"http.uri"
,requestContext.getUri().toString())
33
Trace.record(Annotation.ClientSend())
34
}
35
}
36
}
AbitofZipkininternalswillmakethiscodesuperclear.ThecentralpartofZipkinAPIisTraceclass.Everytimewewouldliketoinitiatetracing,weshouldhaveaTraceIdandthetracertoactuallytraceit.ThissinglelinegeneratesnewTraceIdandregisterthetracer(internallythisdataisheldinthreadlocalstate).

1
Trace.pushTracerAndSetNextId(tracer,
true
)
Tracesarehierarchicalbynature,sodoTraceIds:everyTraceIdcouldbearootorpartofanothertrace.Inourexample,weknowforsurethatwearethefirstandassuchtherootofthetrace.LaterontheTraceIdiswrappedintoHTTPheadersandwillbepassedalongtherequest(wewillseeonserversidehowitisbeingused).Thelastthreelinesassociatetheusefulinformationwiththetrace:nameofourAPI(People),HTTPmethod,URIandmostimportantly,thatit'stheclientsendingtherequesttotheserver.

1
Trace.recordRpcname(name,requestContext.getMethod())
2
Trace.recordBinary(
"http.uri"
,requestContext.getUri().toString())
3
Trace.record(Annotation.ClientSend())
TheZipkinResponseFilterdoesthereversetoZipkinRequestFilterandextracttheTraceIdfromtherequestheaders(ZipkinResponseFilter.scala):

01
package
com.example.zipkin.client
02
03
import
javax.ws.rs.client.ClientResponseFilter
04
import
javax.ws.rs.client.ClientRequestContext
05
import
javax.ws.rs.client.ClientResponseContext
06
import
javax.ws.rs.ext.Provider
07
import
com.twitter.finagle.tracing.Trace
08
import
com.twitter.finagle.tracing.Annotation
09
import
com.twitter.finagle.tracing.SpanId
10
import
com.twitter.finagle.http.HttpTracing
11
import
com.twitter.finagle.tracing.TraceId
12
import
com.twitter.finagle.tracing.Flags
13
import
com.twitter.finagle.tracing.Tracer
14
15
@Provider
16
class
ZipkinResponseFilter(valname:String,valtracer:Tracer)
extends
ClientResponseFilter{
17
deffilter(requestContext:ClientRequestContext,responseContext:ClientResponseContext):Unit={
18
valspanId=SpanId.fromString(requestContext.getHeaders().getFirst(HttpTracing.Header.SpanId).toString())
19
20
spanIdforeach{sid=>
21
valtraceId=SpanId.fromString(requestContext.getHeaders().getFirst(HttpTracing.Header.TraceId).toString())
22
23
valparentSpanId=requestContext.getHeaders().getFirst(HttpTracing.Header.ParentSpanId)match{
24
case
s:String=>SpanId.fromString(s.toString())
25
case
_=>None
26
}
27
28
valsampled=requestContext.getHeaders().getFirst(HttpTracing.Header.Sampled)match{
29
case
s:String=>s.toString.toBoolean
30
case
_=>
true
31
}
32
33
valflags=Flags(requestContext.getHeaders().getFirst(HttpTracing.Header.Flags).toString.toLong)
34
Trace.setId(TraceId(traceId,parentSpanId,sid,Option(sampled),flags))
35
}
36
37
if
(Trace.isActivelyTracing){
38
Trace.record(Annotation.ClientRecv())
39
}
40
}
41
}
Strictlyspeaking,inourexampleit'snotnecessarytoextracttheTraceIdfromtherequestbecausebothfiltersshouldbeexecutedbythesinglethread.Butthelastlineisveryimportant:itmarkstheendofourtracebysayingthatclienthasreceivedtheresponse.

1
Trace.record(Annotation.ClientRecv())
What'sleftisactuallythetraceritself(Zipkin.scala):

01
package
com.example.zipkin
02
03
import
com.twitter.finagle.stats.DefaultStatsReceiver
04
import
com.twitter.finagle.zipkin.thrift.ZipkinTracer
05
import
com.twitter.finagle.tracing.Trace
06
import
javax.ws.rs.ext.Provider
07
08
objectZipkin{
09
lazyvaltracer=ZipkinTracer.mk(host=
"localhost"
,port=
9410
,DefaultStatsReceiver,
1
)
10
}
Ifatthispointyouareconfusedwhatallthosetracesandspansmeanpleaselookthroughthisdocumentationpage,youwillgetthebasicunderstandingofthoseconcepts.

Atthispoint,thereisnothingleftontheclientsideandwearegoodtomovetotheserverside.OurJAX-RS2.0serverwillexposethesingleendpoint(PeopleRestService.java):

01
package
com.example.server.rs;
02
03
import
java.util.Arrays;
04
import
java.util.Collection;
05
import
java.util.concurrent.Callable;
06
07
import
javax.ws.rs.GET;
08
import
javax.ws.rs.Path;
09
import
javax.ws.rs.Produces;
10
11
import
com.example.model.Person;
12
import
com.example.zipkin.Zipkin;
13
14
@Path
(
"/people"
)
15
public
class
PeopleRestService{
16
@Produces
({
"application/json"
})
17
@GET
18
public
Collection<Person>getPeople(){
19
return
Zipkin.invoke(
"DB"
,
"FINDALL"
,
new
Callable<Collection<Person>>(){
20
@Override
21
public
Collection<person>call()
throws
Exception{
22
return
Arrays.asList(
new
Person(
"Tom"
,
"Bombdil"
));
23
}
24
});
25
}
26
}
27
</person>
Aswementionedbefore,wewillsimulatetheaccesstodatabaseandgenerateachildtracebyusingZipkin.invokewrapper(whichlooksverysimple,Zipkin.scala):

01
package
com.example.zipkin
02
03
import
java.util.concurrent.Callable
04
import
com.twitter.finagle.stats.DefaultStatsReceiver
05
import
com.twitter.finagle.tracing.Trace
06
import
com.twitter.finagle.zipkin.thrift.ZipkinTracer
07
import
com.twitter.finagle.tracing.Annotation
08
09
objectZipkin{
10
lazyvaltracer=ZipkinTracer.mk(host=
"localhost"
,port=
9410
,DefaultStatsReceiver,
1
)
11
12
definvoke[R](service:String,method:String,callable:Callable[R]):R=Trace.unwind{
13
Trace.pushTracerAndSetNextId(tracer,
false
)
14
15
Trace.recordRpcname(service,method);
16
Trace.record(
new
Annotation.ClientSend());
17
18
try
{
19
callable.call()
20
}
finally
{
21
Trace.record(
new
Annotation.ClientRecv());
22
}
23
}
24
}
Aswecansee,inthiscasetheserveritselfbecomesaclientforsomeotherservice(database).

ThelastandmostimportantpartoftheserveristointerceptallHTTPrequests,extracttheTraceIdfromthemsoitwillbepossibletoassociatemoredatawiththetrace(annotatethetrace).InApacheCXFit'sveryeasytodobyprovidingowninvoker(ZipkinTracingInvoker.scala):

01
package
com.example.zipkin.server
02
03
import
org.apache.cxf.jaxrs.JAXRSInvoker
04
import
com.twitter.finagle.tracing.TraceId
05
import
org.apache.cxf.message.Exchange
06
import
com.twitter.finagle.tracing.Trace
07
import
com.twitter.finagle.tracing.Annotation
08
import
org.apache.cxf.jaxrs.model.OperationResourceInfo
09
import
org.apache.cxf.jaxrs.ext.MessageContextImpl
10
import
com.twitter.finagle.tracing.SpanId
11
import
com.twitter.finagle.http.HttpTracing
12
import
com.twitter.finagle.tracing.Flags
13
import
scala.collection.JavaConversions._
14
import
com.twitter.finagle.tracing.Tracer
15
import
javax.inject.Inject
16
17
class
ZipkinTracingInvoker
extends
JAXRSInvoker{
18
@Inject
valtracer:Tracer=
null
19
20
deftrace[R](exchange:Exchange)(block:=>R):R={
21
valcontext=
new
MessageContextImpl(exchange.getInMessage())
22
Trace.pushTracer(tracer)
23
24
valid=Option(exchange.get(classOf[OperationResourceInfo]))map{ori=>
25
context.getHttpHeaders().getRequestHeader(HttpTracing.Header.SpanId).toListmatch{
26
case
x::xs=>SpanId.fromString(x)map{sid=>
27
valtraceId=context.getHttpHeaders().getRequestHeader(HttpTracing.Header.TraceId).toListmatch{
28
case
x::xs=>SpanId.fromString(x)
29
case
_=>None
30
}
31
32
valparentSpanId=context.getHttpHeaders().getRequestHeader(HttpTracing.Header.ParentSpanId).toListmatch{
33
case
x::xs=>SpanId.fromString(x)
34
case
_=>None
35
}
36
37
valsampled=context.getHttpHeaders().getRequestHeader(HttpTracing.Header.Sampled).toListmatch{
38
case
x::xs=>x.toBoolean
39
case
_=>
true
40
}
41
42
valflags=context.getHttpHeaders().getRequestHeader(HttpTracing.Header.Flags).toListmatch{
43
case
x::xs=>Flags(x.toLong)
44
case
_=>Flags()
45
}
46
47
valid=TraceId(traceId,parentSpanId,sid,Option(sampled),flags)
48
Trace.setId(id)
49
50
if
(Trace.isActivelyTracing){
51
Trace.recordRpcname(context.getHttpServletRequest().getProtocol(),ori.getHttpMethod())
52
Trace.record(Annotation.ServerRecv())
53
}
54
55
id
56
}
57
58
case
_=>None
59
}
60
}
61
62
valresult=block
63
64
if
(Trace.isActivelyTracing){
65
idmap{id=>Trace.record(
new
Annotation.ServerSend())}
66
}
67
68
result
69
}
70
71
@Override
72
overridedefinvoke(exchange:Exchange,parametersList:AnyRef):AnyRef={
73
trace(exchange)(
super
.invoke(exchange,parametersList))
74
}
75
}
Basically,theonlythingthiscodedoesisextractingTraceIdfromrequestandassociatingitwiththecurrentthread.Alsopleasenoticethatweassociateadditionaldatawiththetracemarkingtheserverparticipation.

1
Trace.recordRpcname(context.getHttpServletRequest().getProtocol(),ori.getHttpMethod())
2
Trace.record(Annotation.ServerRecv())
Toseethetracinginlive,let'sstartourserver(pleasenoticethatsbtshouldbeinstalled),assumingallZipkincomponentsandApacheZookeeperarealreadyupandrunning:

sbt'projectserver''run-maincom.example.server.ServerStarter'

thentheclient:

sbt'projectclient''run-maincom.example.client.ClientStarter'

andfinallyopenZipkinwebUIathttp://localhost:8080.Weshouldseesomethinglikethat(dependinghowmanytimesyouhaveruntheclient):




Alternatively,wecanbuildandrunfatJARsusingsbt-assemblyplugin:

sbtassembly
java-jarserver/target/zipkin-jaxrs-2.0-server-assembly-0.0.1-SNAPSHOT.jar
java-jarclient/target/zipkin-jaxrs-2.0-client-assembly-0.0.1-SNAPSHOT.jar

Ifweclickonanyparticulartrace,themoredetailedinformationwillbeshown,muchresemblingclient<->server<->databasechain.




Evenmoredetailsareshownwhenweclickonparticularelementinthetree.




Lastly,thebonuspartiscomponents/servicesdependencygraph.




Aswecansee,allthedataassociatedwiththetraceishereandfollowshierarchicalstructure.Therootandchildtracesaredetectedandshown,aswellastimelinesforclientsend/receiveandserverreceive/sendchains.Ourexampleisquitenaiveandsimple,butevenlikethatitdemonstrateshowpowerfulandusefuldistributedsystemtracingis.ThankstoZipkinguys.

ThecompletesourcecodeisavailableonGitHub.
                                            

                                        

                        
                        内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 点击这里给我发消息 
                    

                    

                    

                        

                         标签： 
                                                

                        

                    


                

            


            

                
相关文章推荐

                

                
            

        

        

            

            
            

                

                    
新的分享

                    

                        
                    

                

            


            

                

                    
章节导航