Wednesday, February 22, 2012

Figure out why is JAVA eating CPU?

In a production environment profiling is not an option, we have seen several times that our CPU has reached almost 100%, we are running tomcat on Redhat 5.5, so what is happening?

Fortunately, java comes with some great debugging tools, cooperating those tools with Linux built-in tools will let you know what is happening.

Here is what i am going to explain:
  • An introduction about java threads and its relation to what-so-called Linux LWP
  • A step-by-step
1- An introduction

as you may know, any java program starts when the JVM calls the main method, this creates a thread called the main thread and any thread you create using java code will be derived from the main thread (out-of-focus: it is always a good practice to give your threads a name, it is very useful in debugging, however, it is not required here) the same exact behavior occurs on the Linux level, the main thread for java means a process for the OS, and every thread you create using java the OS will create a Light-weight-process or LWP.
to cut it short, Java main thread = Linux process and Java thread = Linux LWP.

the idea here:
  • Ask Linux which LWP is eating the CPU.
  • Ask Java for a Thread Dump.
  • Map this LWP to a Java thread.
  • Get the part of code causing the issue.

2- A Step-By-Step:
  • Get the PID: the very first step is to know what is the Java process ID, we will use Linux tools ps and grep
    ps -A|grep java
    if you are running multiple java processes we can execute
    ps -ef|grep java

  • the next step is to get CPU usage per each LWP related to the main process, again we will use  ps  and  grep
    ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args|grep {pid} > lwpthread.txt
    the file lwpthread.txt will contain some thing similar:
    PID NLWP SZ RUSER %CPU STIME ELAPSED COMMAND
    8234 8234 1110 admin 0.3 08:11 30:15 /usr/java/jdk1.6.0_24/bin/java
    8234 8245 1110  admin 99.0 08:45 10:15 /usr/java/jdk1.6.0_24/bin/java 

    as you can see, we have an LWP(mapped to a java thread) eating the CPU, get the NLWP, that will be our lead in the next step.

  • the next step is to generate a java thread dump, there are two main ways to create a thread dump, the first one is to use the JDK tool jstack and pass the PID to it, the second way is to send a kill signal to the JVM, however in the first way you will have control over where you want to save the thread dump, while on the second way you will have the thread dump written on the java process standard output stream.
     as said, we are using tomcat, so the thread dump will be in catalina.out
    kill -3 {pid}
    the thread dump will be printed to the file with full stack trace. the file will contain the thread you are after, but first convert your LWP id from DEC to HEX so 8245 would be 2035, now open the thread dump with text editor and search for 2035, you will find something similar:
    "TP-Processor234786" daemon prio=10 tid=0x00002aaad8024800 nid=0x2035 runnable [0x00002aaadef29000]
    java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.get(HashMap.java:303)
    at ......

and you are done!

105 comments:

javin paul said...

This is fantastic tip man, Indeed useful. Production environment is always hard for debugging with logs level upto WARN and ERROR and limited access. Please share more of your experience. bookmarked it. I have also written about Java remote debugging in Eclipse you may like.

Javin

Anonymous said...

Another way to do it is using jconsole if you had started the JVM with the following
-Dcom.sun.management.jmxremote.port=9003

you can install jconsole plugin called topthreads

and then start jconsole with the following from any machine and connect to the JVM machine on the above port

jconsole -pluginpath topthreads.jar

Moutaz Salem said...

Thanks!!!
i will be posting more findings as i go :)

Moutaz Salem said...

Hi Sumit,

as i mentioned, profiling is not an option here :)

the same applies on Jconsole and VisualVM

Mamdouh said...

That was definitely a life saver :)

Moutaz Salem said...

thanks Mamdouh :)

Boudy said...

Man we are having the same problem at our live environment .. And we're trying to figure out a way to do this for almost 2 weeks now .. Thanks a lot, keep em comin ; D

Moutaz Salem said...

Most welcomed Boudy ;)

Rajkumar Selvaraj said...

awesome article.. thanks

Moutaz Salem said...

My pleasure, Rajkumar

Hazem said...

Excellent work moutaz :)

Looking forward to read more articles by you

You are an artist!!

Moutaz Salem said...

thanks, Hazem

i learned from a Master :)

Hazem said...

How humble you are!!!

My dear friend you are a real distinguished master, we learned a lot from you, really we did, and we still as you see :)

keep doing well, and always share :)

Stas Ostapenko said...

Hi. Thnaks for the post :) Btw, instead of "ps -Aef | grep java" you can use jps tool.

Moutaz Salem said...

of course, Stas
thanks for the tip :)

Anonymous said...

Very useful indeed, thanks very much.

Moutaz Salem said...

Most Welcomed, yorichunt

Mayur Choubey said...

This is just incredible... thanks a ton...

Moutaz Salem said...

you are more than welcome, Mayur

Unknown said...

Great Article

Online Java Training
Online Java Training India
Java Training Institutes in Chennai
Java Training in Chennai

Java Training




Best Recommended books for Spring framework
Java Interview Questions

fatih tekin said...

command should have been as below without grep otherwise it only gives your own grep command process id

ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args

Moutaz Salem said...

@fatih tekin: that is not entirely true, we are grepping on a certain PID, so the PID of the grep will be ignored

Unknown said...

I find this post on identifying high CPU usage issue was really useful. And the way you explained it was really nice and clear.

Thanks a lot.!!

IT said...

Java Spring Hibernate Training Institutes in Chennai J2EE Training Institutes in Chennai J2EE Training Institutes in Chennai Java Spring Hibernate Training Institutes in Chennai | Hibernate Training Institutes in Chennai Core Java Training Institutes in Chennai Core Java Training Institutes in Chennai

Hibernate Online Training Hibernate Online Training Hibernate Training in Chennai Hibernate Training in Chennai Java Online Training Java Online Training Hibernate Training Institutes in ChennaiHibernate Training Institutes in Chennai

Unknown said...
This comment has been removed by the author.
Dinesh said...

When using visual VM CPU is at 100%..however the system is only using 40% of CPU overall as seen from top..using Websphere..is there a place any where which can restrict CPU in a JVM?

rmouniak said...

I have read your blog its very attractive and impressive,Thanks for sharing

Java Online Training

Unknown said...

Thankx for sharing may be useful but my java eating cpu 200+ issue not resolved so please sharing any other option for resolving

Government Jobs said...

organic cold pressed oils
natural cold pressed oils
organic oil
organic oil in jaipur
organic cold pressed oil in jaipur
natural oil
natural oil shop in jaipur
pure herbal oil
ayurvedic oil store in jaipur
ayurvedic oil

Government Jobs said...

law college
law college in Jaipur
Best law college in Jaipur
Law Course In Jaipur
Top College Of law In Jaipur
Vidyasthali Law College
Best Law College
Jaipur Law College

Government Jobs said...


organic cold pressed oils
natural cold pressed oils
organic oil
natural oil
pure herbal oil
ayurvedic oil store in jaipur
ayurvedic oil

Government Jobs said...

organic cold pressed oils
natural cold pressed oils
organic oil
natural oil
pure herbal oil
ayurvedic oil store in jaipur
ayurvedic oil

Government Jobs said...

law college
law college in Jaipur
Best law college in Jaipur
Law Course In Jaipur
Top College Of law In Jaipur
Vidyasthali Law College
Best Law College
Jaipur Law College

Government Jobs said...

law college
law college in Jaipur
Best law college in Jaipur
Law Course In Jaipur
Top College Of law In Jaipur
Vidyasthali Law College
Best Law College
Jaipur Law College

Government Jobs said...

law college
law college in Jaipur
Best law college in Jaipur
Law Course In Jaipur
Top College Of law In Jaipur
Vidyasthali Law College
Best Law College
Jaipur Law College

Government Jobs said...

Thanks for sharing this unique information with us. Your post is really awesome. Your blog is really helpful for me..
Top college in Jaipur
law college
law college in Jaipur
Best law college in Jaipur

Government Jobs said...

Thanks for sharing this unique information with us. Your post is really awesome. Your blog is really helpful for me..
organic oil
organic oil in jaipur
organic cold pressed oils
ayurvedic oil store in jaipur

aditya said...

Thanks for sharing this unique information with us. Your post is really awesome. Your blog is really helpful for me..
hospital equipment
sarthak maditech
hospital equipment suppliers
hospital equipment manufacturers
medical equipment manufacturers in jaipur
operation theater lights
hospital suction machine
alcohol breath tester
hospital furniture manufacturer

siva said...

Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me. I am a regular follower of your blog.
Really very informative post you shared here. Kindly keep blogging.
Java training in Btm layout
Java training in Rajaji nagar
Java training in Kalyan nagar
Java training in Kalyan nagar
Java training in Jaya nagar

service care said...

This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me..
samsung mobile service center near me
samsung service centres in chennai
samsung mobile service center in velachery
samsung mobile service center in porur
samsung mobile service center in vadapalani

digitalsourabh said...

PHP Training in Bhopal
Graphic designing training in bhopal
Python Training in Bhopal
Android Training in Bhopal
Machine Learning Training in Bhopal
Digital Marketing Training in Bhopal
https://99designs.com/blog/trends/top-10-web-design-trends-for-2014/

spot said...

super your blog
andaman tour packages
andaman holiday packages
web development company in chennai
Math word problem solver
laptop service center in chennai
Austin Homes for Sale
andaman tourism package
family tour package in andaman

anusha said...


MEAN Stack Training in Chennai MEAN Stack Training in Chennai with real time projects. We are Best MEAN Stack Training Institute in Chennai. Our Mean Stack courses are taught by Industrial Experts which would help you to learn MEAN Stack development from the scratch.

Techxinx said...

Thanks for this post, I really appriciate. I have read posts,
all are in working condition. and I really like your writing style.
autocad in bhopal
3ds max classes in bhopal
CPCT Coaching in Bhopal
java coaching in bhopal
Autocad classes in bhopal
Catia coaching in bhopal

shivam said...

Flying Shift - Packers & Movers in Bhopal

verdure skills india pvt ltd said...

nice blog
get best placement at VSIPL

digital marketing services
Web development Services
seo network point

vijay said...

Awesome..I read this post so nice and very imformative information...thanks for sharing
aws Training in Bangalore
python Training in Bangalore
hadoop Training in Bangalore
angular js Training in Bangalore
bigdata analytics Training in Bangalore
python Training in Bangalore
aws Training in Bangalore

hami said...


I am obliged to you for sharing this piece of statistics
here and updating us together with your inventive steerage.
sd movies point

Durai Moorthy said...

This is very good quality article and interesting..& This post has a nice one. share more updates.
aws Training in Bangalore
python Training in Bangalore
hadoop Training in Bangalore
angular js Training in Bangalore
bigdata analytics Training in Bangalore
python Training in Bangalore
aws Training in Bangalore

Shopclues Winner List said...

Shopclues winner list 2020 here came up with a list of offers where you can win special shopclues prize by just playing a game & win prizes.
Shopclues lucky draw 2020
Shopclues winner name 2020
Shopclues prize list 2020

Rajesh Anbu said...

Your article is very informative. Thanks for sharing the valuable information.
aws Training in Bangalore
python Training in Bangalore
hadoop Training in Bangalore
angular js Training in Bangalore
bigdata analytics Training in Bangalore
python Training in Bangalore
aws Training in Bangalore

Situs Judi Online Terpercaya said...

NAGAQQ | AGEN BANDARQ | BANDARQ ONLINE | ADUQ ONLINE | DOMINOQQ TERBAIK

Yang Merupakan Agen Bandarq, Domino 99, Dan Bandar Poker Online Terpercaya di asia hadir untuk anda semua dengan permainan permainan menarik dan bonus menarik untuk anda semua

Bonus yang diberikan NagaQQ :
* Bonus rollingan 0.5%,setiap senin di bagikannya
* Bonus Refferal 10% + 10%,seumur hidup
* Bonus Jackpot, yang dapat anda dapatkan dengan mudah
* Minimal Depo 15.000
* Minimal WD 20.000

Memegang Gelar atau title sebagai QQ Online Terbaik di masanya

Games Yang di Hadirkan NagaQQ :
* Poker Online
* BandarQ
* Domino99
* Bandar Poker
* Bandar66
* Sakong
* Capsa Susun
* AduQ
* Perang Bacarrat (New Game)

Tersedia Deposit Via pulsa :
Telkomsel & XL

Info Lebih lanjut Kunjungi :
Website : NAGAQQ
Facebook : NagaQQ Official
Kontakk : Info NagaQQ
linktree : Agen Judi Online
WHATSAPP : +855977509035
Line : Cs_nagaQQ
TELEGRAM : +855967014811


BACA JUGA BLOGSPORT KAMI YANG LAIN:
agen bandarq terbaik
Winner NagaQQ
Daftar NagaQQ
Agen Poker Online

kimi jhaveri said...

Hi, your article was of great help. I loved the way you shared the information, thanks.
Amazing article, I highly appreciate your efforts, it was highly helpful. Thank you CEH Training ,CEH Certification, CEH Online Course, Ethicalhacking

sachin nagar said...

Hi, This is a great article. Loved your efforts on it buddy. Thanks for sharing this with us.
Get cissp
it training courses.
CISSP training ,cissp exam cost, CISSP certification. .Get VMware, vmware training, vmware course, vmware online training., vmware interview questions and answers, vmware Certification, AWS, aws training, aws course, aws certification training, aws online training
Get PMP pmp certification, pmp training, pmp certification in gurgaon,pmp certification cost, pmp training certification

sachin nagar said...

Hi this is the nice blog, thanks for sharing us
Get Azure, azure training,azure certification,microsoft azure training,azure course,azure online training

sachin nagar said...

Hi, This is your awesome article , I appreciate your effort, thanks for sharing us.
cism training
cism certification

cisa training,
cisa certification
cisa exam

Ranvijay Singh said...

SEO Training in Bhopal
Digital Marketing Training in Bhopal
SEO Company in Bhopal
Digital Marketing Company in Bhopal
High PR Profile Creation Sites
Web 2.0 Sites list
Directory Submission Sites list
Article Submission Sites list
Social Bookmarking Sites list
Digital Marketing Course in Bhopal

Anonymous said...

Informative post, i love reading such posts. Read my posts here
Unknownsource
http://unsurpassedesports.esportsify.com/profile/globalemployees
http://unsurpassedesports.esportsify.com/forums/scrims-ps4/277/hire-laravel-developers

Gorish dua said...

Such an incredible blog post. Thanks a lot.
Wedding planner in Goa
Wedding planner in Delhi
Wedding planner in Mumbai
Wedding planner in Gurgaon

Unknown said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot.

python certification Online Training in bangalore

python certification courses in bangalore

python certification classes in bangalore

python certification Online Training institute in bangalore

python certification course syllabus

best python certification Online Training

python certification Online Training centers

Unknown said...

Very interesting blog Thank you for sharing such a nice and interesting blog and really very helpful article.

ORACLE APPS FINANCE FUNCTIONAL training in bangalore

ORACLE APPS FINANCE FUNCTIONAL training institutes in bangalore

ORACLE APPS FINANCE FUNCTIONAL courses in bangalore

ORACLE APPS FINANCE FUNCTIONAL course in bangalore

Oracle apps finance functional training institutes in bangalore

ORACLE APPS FINANCE FUNCTIONAL classes in bangalore

ORACLE APPS FINANCE FUNCTIONAL course fees in bangalore

best ORACLE APPS FINANCE FUNCTIONAL training institutes in bangalore

ORACLE APPS FINANCE FUNCTIONAL training centers in bangalore

ORACLE APPS FINANCE FUNCTIONAL certification in bangalore

Oracle apps finance functional courses in bangalore

ORACLE APPS FINANCE FUNCTIONAL certification courses in bangalore

ORACLE APPS FINANCE FUNCTIONAL course fees

Oracle apps finance functional training in bangalore

Oracle apps finance functional training in bangalore

best Oracle apps finance functional training institutes in bangalore

Oracle apps finance functional training in bangalore

ORACLE APPS FINANCE FUNCTIONAL training

ORACLE APPS FINANCE FUNCTIONAL training in btm layout

ORACLE APPS FINANCE FUNCTIONAL course in marathahalli

Unknown said...

Thank you for excellent article.You made an article that is interesting.

SAP Online Training

SAP Classes Online

SAP Training Online

Online SAP Course

SAP Course Online

Unknown said...

I have recently visited your blog profile. I am totally impressed by your blogging skills and knowledge.

SAP ABAP Online Training

SAP ABAP Classes Online

SAP ABAP Training Online

Online SAP ABAP Course

SAP ABAP Course Online

James Williams said...

Great Post. Thanks for sharing your idea.
Java Online Training
Python Online Training
PHP Online Training

vidyasthali law college said...



such an amazing college for law students to build career amazing study material with high profile facilities and teachers.
best law college

About College

Best Law College in jaipur

Information about college

Best Law College in jaipur
www.vidyasthalilawcollege.com

balraj jain said...

Vidyasthali Group of Institutions was founded by an eminent group of academics and industry leaders who are masters of the top significant achievements and accomplishments. Vidyasthali is a reputed B-school in Jaipur.
vidyasthali institute of technology science&managment
best techonology science and management institute in jaipur
courses in vidyasthali group of institute
no 1 technology science & management college in jaipur
best management
vitsm
vidyasthali institute of technology science and managment
techonlogy science&management college near me,
best technology
best management
VITSM teaching staff
vitsm courses
top science college
best science colleges in jaipur
best science college in jaipur
vitsm fees structure
vitsm fee structure
vitsm result
vitsm result

Python said...

Thanks for Sharing.
Data Science Online Training
python Online Training

meritstep said...

I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
in their life, he/she can earn his living by doing blogging.Thank you for this article.
java online training

meritstep said...

I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
in their life, he/she can earn his living by doing blogging.Thank you for this article.
best java online training

meritstep said...

I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
in their life, he/she can earn his living by doing blogging.Thank you for this article.
java online training

meritstep Technology said...

Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
mulesoft online training
best mulesoft online training
top mulesoft online training

saketh321 said...


I have a mission that I’m just now working on, and I have been at the look out for such information ExcelR Data Science Course In Pune

Python said...

Thank you for sharing.
Data Science Online Training
Python Online Training
Salesforce Online Training

Rajesh Adhikesavan said...

Male fertility doctor in chennai
Std clinic in chennai
Erectile dysfunction treatment in chennai
Premature ejaculation treatment in chennai

Franklin said...

online student management system
school software
school management erp

hont said...

school inventory management system
student parent login
online exam system
role of school mangement software

Best Training Institute said...

Thankyou for sharing this kind of valuable information with us. Your post give great knowledge
Servicenow training in bangalore

topseocompanyinbhopal said...

Digibrom is the Best Digital Marketing
Training & Services In Bhopal
Digibrom is the Best Digital Marketing Training & Services in Bhopal, providing complete digital growth for your business. We are one of the leading Digital Marketing company in Bhopal, and we are experts in digital marketing & web design. The market penetration along with the use of digital technology is our power. We serve businesses according to the need and requirements of the customers and deliver quality work in time because Digibrom is the best digital marketing training institute and service provider. We create likable content that increases the time spent by the customer on the internet.
Digital Marketing training company in bhopal
Digital Marketing training in Bhopal

jency said...

email and task management in usa
project management automation in usa
b2b web development in usa
retail process automation in usa
industries IT services in usa

call center software said...

I really want to appreciate the way to write this
omni-channel
ivrs
ip-pbx

Addons said...

ecommerce service company in usa
shopify website development in usa
prestashop development in usa
magento development in usa
oscommerce development in usa

Persephone Summers said...

buy juicy fruit online
buy gelato strain online
Buy dark star strain online
buy hawaiian skunk strain online
buy bc big bud strain leafly
buy auto flowering seeds online
buy brass knuckles vape recall 2018
buy alaskan thunder fuck online
buy cannabis seeds bank online

derick said...

british shorthair kittens
british shorthair cat for sale

Jobi said...

Thank you very much for this great post.
Baron Zemo Coat

Unknown said...

Pretty remarkable post. I simply came across your blog and desired to say that I have really enjoyed searching your blog posts.
buy stamina booster tablets online

johnson said...

buy marijuana-flowers online
buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy auto-flowering-seeds online
buy psychedelics online
“buy psychedelics online usa
buy cannabis-concentrates-online

johnson said...

buy marijuana-flowers online
buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy auto-flowering-seeds online
buy psychedelics online
“buy psychedelics online usa
buy cannabis-concentrates-online

johnson said...

buy marijuana-flowers online
buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy auto-flowering-seeds online
buy psychedelics online
“buy psychedelics online usa
buy cannabis-concentrates-online

johnson said...

buy marijuana-flowers online
buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy auto-flowering-seeds online
buy psychedelics online
“buy psychedelics online usa
buy cannabis-concentrates-online

Antino Labs said...

Thanks for sharing the information.
Mobile application development company in gurgaon
Digital transformation companies in india

salome said...

useful article.thanks for sharing Angular training in Chennai

johnson said...

buy marijuana-flowers online
buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy granddaddy-purple online
buy psychedelics online
buy cannabis-concentrates-online
buy og-kush online
buy dmt-nn-dimethyltryptamine online
buy blue-cheese-weed online
buy purple-haze online
buy strawberry-cough online
buy black-diamond-kush online
buy blue-dream online
buy moon-rock online

Unknown said...

what is contrave

silicon wives

sky pharmacy

atx 101 uk

macrolane buttock injections london

hydrogel buttock injections

buying vyvanse online legit

buy dermal fillers online usa

mesotherapy injections near me

xeomin reviews

xingxing said...

abrigo aviador mujer abdomen
gucci forocoches
sonic 3 and knuckles juego online
short de bain islamique
stihl klamotten
brabus rocket 900 amazon
new balance icarus
sonnenbrille herren mit seitenschutz
hoodie pepe jeans cyan
shorts fitness feminino
melhores marcas de mochilas masculinas
quadri da disegnare amazon
aharry potter x vans
culle e passeggini per neonati amazon
camisa da ferroviária
galocha rosa feminina
deichmann bolsos
bolso de mano de hombre guess entusiasta
tenda da sole spiaggia amazon
nike pegasus 94
Nike Air Force 270
borraccia decathlon amazon
sostituzione cinghia distribuzione seat mii
veste per battesimo amazon

xingxing said...

calzoncillos puma hombre
adidas originals panna scarpe uomo basse
grossista condizionatori
costumi interi amazon
vestito da elsa disney
adidas yeezy semi frozen yellow on feet
second hand golf 5 gti for sale
piumini daunenstep amazon
pittura x pareti amazon trascorrere
fiore garofano amazon
air force 1 nike outfit men basket
nike air zip up hoodie
king kong jeans
zapatillas running galaxy 4 azul marino mujer adidas
blazer basicos mujer
nike air max de bebe principal opción
zapatos de charol mujer con taco
nike metcon flyknit 3 para running
bolsas para comprimir equipaje
chanclas new balance niña
nike pegasus grey and green
amazon sandalias reef
polo simulator

johnson said...

buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy granddaddy-purple online
buy psychedelics online
buy cannabis-concentrates-online
buy og-kush online
buy dmt-nn-dimethyltryptamine online
buy blue-cheese-weed online
buy purple-haze online
buy strawberry-cough online
buy black-diamond-kush online
buy blue-dream online
buy moon-rock online
buy blue-dream-feminized online
profile.php?id=100069707270977 online
buy marijuana-flowers/sativa/ online
buy marijuana-flowers/indica-strains/ online
buy marijuana-flowers/hybrid/ online


johnson said...

buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy granddaddy-purple online
buy psychedelics online
buy cannabis-concentrates-online
buy og-kush online
buy dmt-nn-dimethyltryptamine online
buy blue-cheese-weed online
buy purple-haze online
buy strawberry-cough online
buy black-diamond-kush online
buy blue-dream online
buy moon-rock online
buy blue-dream-feminized online
profile.php?id=100069707270977 online
buy marijuana-flowers/sativa/ online
buy marijuana-flowers/indica-strains/ online
buy marijuana-flowers/hybrid/ online


johnson said...

buy marijuana-edibles online
buy cbd-oils online
buy vapes-and-carts online
buy accessories online
buy auto-flowering-seeds online
buy granddaddy-purple online
buy psychedelics online
buy cannabis-concentrates-online
buy og-kush online
buy dmt-nn-dimethyltryptamine online
buy blue-cheese-weed online
buy purple-haze online
buy strawberry-cough online
buy black-diamond-kush online
buy blue-dream online
buy moon-rock online
buy blue-dream-feminized online
profile.php?id=100069707270977 online
buy marijuana-flowers/sativa/ online
buy marijuana-flowers/indica-strains/ online
buy marijuana-flowers/hybrid/ online



Unknown said...

what is contrave

silicon wives

sky pharmacy

atx 101 uk

macrolane buttock injections london

buy vyvanse online usa uk

buying vyvanse online legit

where can i buy dermal fillers without license

mesotherapy injections near me

xeomin reviews

diamondsymons2 said...

casquette ford Immigration
chaussure carla moreau
triciclo usato milano
chaussures de sécurité facom
foot locker nike react element
scatola protezione stagna
nike metcon 5 rojas
zapatillas de agua niña
veste velour carhartt
damen lack schnürer
cappotto rosso scarpe nike
fiore garofano amazon
nike presto extreme pas cher
nike pegasus 94
shorts fitness feminino
nike t shirt tumblr
quadri da disegnare amazon
chinelo slide da melissa
tenis all star feminino branco couro
calzoncillos puma hombre
veste per battesimo amazon
nike air force modelo agotado
cartera levis hombre
filtre pour aspirateur samsung sc4780
autoradio gps vdo
nike tribute veste
soldes chaussures homme caterpillar hiver cafétéria
amazon sandalias reef
chanclas new balance niña
kit de creation de bracelets utilisation
wc suspendu mural amazon
foulard en soie homme
all star converse star player
paragliding přilby
nike anzug nba rot
camisa da ferroviária
tapie adidas credit lyonnais
gucci handbags uk
air force 1 nike outfit men basket

bedortheadora said...

pittura x pareti amazon trascorrere
pull training nike
nike sb air max bruin vapor
jogging fille 14 ans adidas
outlet nike floresta
nike dual fusion tr iii
maglietta ciglia stradivarius
nike jogginghose dunkelgrau damen
nike academy16 knt tracksuit 2
borraccia decathlon amazon
camisa ponte preta aranha
sandalias de esparto mujer
kiabi bottine femme
polo lacoste ton sur ton
pepe jeans tami
crampon vissé adidas
brassiere garcon
vestido de niña rosa palo
maglia termica nuoto bambina
adidas zx flux adv virtue sock w
nike cortez prm
beutel zipper amazon
nike tanjun damen weiß 44
scarpe pirelli uomo impedire
polo ralph lauren jeanshemd damen
risparmiare batteria iphone ios 12
style année 80 femme
Nike Air Force 270
kimono en jean
pantaloni puma femei pollice Linea di metallo
shein camicie donna cameriera
bañador neopreno mujer oysho
sudadera nike vintage
giacca da prestigiatore
sandalias courofeito a mao
grossista condizionatori
piumini daunenstep amazon
zapatillas cruyff mujer

qwe said...

nike sb prod x
gucci leather wristband
botas australianas mustang
magnum pi jeans
nike pegasus 94
scatola protezione stagna
shein camicie donna cameriera
nike les halles
nike blazer low 3d
vestido de niña rosa palo
giacca da prestigiatore
sapatenis masculino elastico
sujetador media copa abierto
polo shirt juventus
ladies summer shorts
gucci slides cheap
sandalias agua cerradas
survetement adidas homme militaire
nike t shirt tumblr
nike tanjun damen weiß 44

qwe said...

tumbona terraza ikea
nike zoom strike sprinter
gants si assault factory pilot noir oakley
camisa cuadros franela mujer
galocha rosa feminina
adidas run falcon on feet
ghete galbene piele
hama stiftplatte einhorn amazon
moleton nike preço
veste velour carhartt
bañador neopreno mujer oysho
jogging fille 14 ans adidas
melhores marcas de mochilas masculinas
beistelltisch mit stauraum
nike air max 97 silber
adidas niño verde agua
béret a pompon bretelles uniforme
rompecabezas de obras de arte para arma
venta de bicicletas para ejercicio
sudadera nike vintage
lidl online dirndl
adidas noir doré
foot locker nike react element
filtre pour aspirateur samsung sc4780
adidas stabil x junior

bedortheadora said...

nike air max de bebe
sfera vestido flecos mostaza
filtre pour aspirateur samsung sc4780
tende doccia milano amazon
adidas superstar purple
jogging fille 14 ans adidas
af1 louis vuitton
sandália rasteira bottero
zapatillas bimba y lola outlet
chinelo slide da melissa
leggings mit spitzenabschluss
cartera levis hombre
ropa de crossfit nike
camisa da ferroviária
jersey rombos hm
gucci forocoches
sudadera nike vintage
shirt mit schnürung am ausschnitt
gants si assault factory pilot noir oakley
hama stiftplatte einhorn amazon
outlet nike floresta
sacoche lv district
nike hoodie xxl ebay
timberland herren 3 eye
kit citofono terraneo amazon
nike presto extreme pas cher
gucci handbags uk
nike huarache kobe bryant
tight leggings adidas
bolsas de deporte nike negra hombre grande
style année 80 femme
adidas personalised shoes
sandalia rafaela melo
survetement adidas homme militaire
vestido de niña rosa palo
adidas noir doré
adidas rock climbing
adidas boost 350 black

3RI Technologies said...

Good information posted for all of us, keep writing such blogs.

Data Science Training in Pune