NoSql Database Cluster Configuration for high Performance

posted in: Projects | 0

This post focuses on current commodity hardware. Its optimal setting for various types of use cases and how your choice of NoSql database can be configured to use it to achieve high performance.

Hardware Performance

Disk Drive Performance

For disk drives, most time consuming operation is seek times. From wikipedia, seek time of 7200 RPM drives is about 4 ms. So in worst case, disk drives can read about 2500 blocks per second. It needs at least 4-5 disk seeks (for database table with over 100K records, it will need at least 4-5 levels of btree) for Primary key Btree index lookup. So realistically, its performance is about 500 records per second with 5 seeks.

For sequential reads/writes (for append only databases or kafka like use case), 7200 RPM disk drive can do about 100 Mbytes per second. Or for 1KB record size it can read/write about 100,000 records per second.

SSD Performance

High performance SSDs today can do about 100,000 IOPs per second of about 4K size. Performance of SSD is almost same as sequential disk accesses.

It may be a good idea to use disk drives for append only files like redo logs or append type workloads like in kafka.

Object Serialization/deserialization Performance

Based on performance analysis done on this blog, it is about 250,000 serialization/de-serialization of 1,000 byte payload using ProtoBufs on Intel Intel Core i3-3240 (Ivy Bridge) 3.40GHz processor. It should linearly scale with more CPUs and cores as this operation is mostly CPU bound.

Typical hardware configuration

For cost and performance, we will use typical hadoop recommended configuration as our baseline.

#CPUs/Cores

Based on the price, power consumption, use case and rack size, try to get as many CPUs as possible per machines.

Its always better to scale up to certain limit before thinking about scaling out. Typical hardware with 8, 16 or even 32 CPUs is possible in todays commodity hardware configurations.

Just to put some perspective, typical memory hash map implementations like redis can do ~100,000 get/set per second per CPU. So most of NoSql workloads are going to be memory and disk bound (and not CPU bound) as throughput of disk based NoSqls is never going to be 100,000 queries per second per CPU.

Physical Memory

Try to get as much physical memory as possible within constraints of price, size and power consumption. It is possible to have memories ranging from 72 GB, 128GB, 256GB and 512GB. More the better.

Disk Drives and SSDs

SSDs beat disk drives in all respect except the size. Commodity hardware can have 14 disk drives of 2-4TB sizes, that amount to about 24 TB to 48 TB storage with disk.

Cost and physical slot size requirements stop SSDs to reach to that kind of storage per machine. Typical SSD storage could range from 1TB up to 10TBs.

Choosing hard drives or SSDs

Pure size perspective hard drives make sense. You will need to fill up 48 TBs before considering about sharding. But performance is a big bottleneck with hard drives. With 14 drives, you can have 14*500=7000 record QPS.

SSDs can do about 100,000 IOPS or 20,000 QPS for record (1/5th due to btree lookup). With 2-10 SSDs per hardware, one can expect about 20,000*10=200,000 QPS for records.

You can choose hybrid approach. Based on the use case some collections could be in SSDs and some in hard drives. For Example – If you are building E-commerce site then, use case usually behaves in a way that 80-90% of queries use hot records from memory. For example, there will be lot of queries on iPhone or standard items which could be in the memory. Or queries to get user preferences etc. In this case, it may be OK to use hard drives. For append only use cases like redo logs, kafka use disk drives.

RAID

For  random accesses use RAID 0 for performance. Most of NoSql databases provide redundancy, so redundant RAID configurations may not be required. But if you are using disk drives and data size requirements are going to be less than available size then you may consider using redundancy RAID configurations. At least single disk failures may not trigger node rebalancing.

For append only use cases, redo logs, kafka etc don’t use RAID. As these use cases rely heavily on the sequential access speed of disk.

Payloads

OLTP

OLPT use cases mostly tend to operate on single record. It is easier from scalability, extensibility and maintenance point of view to have schemas properly normalized. De-normalized schemas create maintenance issues as some future use cases may not be easy to implement if schema is not properly normalized. Classic example is, teacher changing her class timings. For normalized schema just one record in one table needs to change. But for de-normalized schemas, records of all students taking that class might have to be updated.

For OLTP workloads it is better to choose NoSql which supports relational type schemas and joins.

New wave of NewSql databases is picking up where they are focusing on “functionality of SQL/relational databases with scale of NoSql” mantra.

Data warehousing / Analytics

Use case in this case are mostly to insert records in fact tables and be able to join with dimension tables. Key here is dimension tables change very rarely. Sometimes these kinds of dimension tables are called slowly changing dimensions. For Example – Walmart adding support for new product or adding new store.

Due to their insert only nature, their size can go infinitely. Because of this proper sharding of data is a absolute must. Also, due to their append/insert only nature, it is ok to de-normalize schema up to certain extent. Schema de-normalization, doesn’t mean joins are not required. In these payloads as well, you still will need a join support for joining with dimension tables. For Example – To count number of products sold in Walmart store A from Cereal section in bottom rack. In this case there are 3 dimensions used; Store dimension, Section dimension and Rack dimension. Rack and Section dimensions may be hierarchical and may not be easily mapped to some metadata to avoid joins. In this case it may be best to use joins (and we will be better off using database supporting joins) rather than name-value or document databases which don’t support joins.

Bottom line is, data warehousing type of payloads need de-normalized data schema with shardable, horizontally scalable databases, but it may not be a bad idea to select latest type of NewSql databases which are highly scalable and support joins and other relational database constructs. Google released datastore which is highly scalable and also support some basic SQL like queries and joins.

Real time analytics

In this case, data from data warehousing is fed back to OLTP databases (or other databases which can be queried in real time) so that they can use this data to serve user better. For Example – Sites providing with recommendations based on user current activity.

This could be huge amounts of mostly read only data (data might change once a day, this change could be of insert, update or delete type) fed from data warehousing which will be queried by realtime use case. For Example – Say user is watching some movie; based on that user is provided with recommendations. Recommendations engine in this case uses both real time OLTP data (to get current user activities) and data warehousing data to use in recommendations algorithms to come up with best possible recommendations for that user.

NoSQL Databases

Due to scale and size of data, any selected NoSql database should support sharding and replication.

Most of current wave of top NoSql databases are mostly document or name/value and not completely designed to take full advantage of current commodity hardware specs.

MongoDB

use RAID 0 for collection and index files.

MongoDB architecture almost forces use of SSDs which limits maximum storage capacity per node. Which further forces going for sharding when other resources like CPU or memory haven’t reached their peak capacity.

Due to its memory mapped files and table locking for writes, it cannot use commodity hardware stack to its full capacity. Sharding further causes, hardware, operational, maintenance costs sky rocketing along with fragile structure of cluster since even sharding cannot completely solve underlying architectural issues.

On top, their loose (document) schema model comes in a way of future enhancements as due to need for atomicity, schema de-normalization is forced which in turn increases the document size and eventually suffers the performance and other issues crop up due de-normalized schema.

Bad queried can further degrade cluster performance since not frequently used records can clog up physical memory due to memory mapped files.

Cassandra

Cassandra read and writes could be sequential (their architecture document is not clear if reads are sequential). SSTables are written to disk sequentially. They are read periodically sequentially to merge in to  one SSTable.

Even though reads and writes are sequential, overtime due to increasing file count disk fragmentation can occur. They loose speed of sequential accesses which will show up in the performance and throughput.

Based on this information, we can assume that Cassandra node may not use RAID and should not need to use SSD.

Couchbase

Couchbase is document database like MongoDB so de-normalization issues discussed in MongoDB also apply to Couchbase.

Couchbase is classic example of append only database where every insert/update/delete is appended to datafile. And Compaction keeps compacting data files for every bucket.

If the data size is bigger than the physical memory then, random access reads may be required to fetch the record.

Since performance of disk drives and SSD is same for sequential accesses writes can use either SSD or disk drives. For reads SSD will outperform hard drives due to their random access nature.

So based on use case and data size you may have to choose between SSD and disk drive.

  • If physical memory is about same as data size per node then use disk drive. I would say data size bigger by up to 50%, use disk drive.
  • If your use case is such that 80-90% of queries use hot records from memory then use disk drive.
  • Any other case, use SSD, but note that you might have to start sharding earlier than you wanted.

Kafka

Even though Kafka is not a NoSql database, it shows up in most of the NoSql cluster databases.

Its payload is mostly append only disk drive type with not much CPU needs. It usually needs as many disk drives as possible with no need for RAID.

0 Responses

  1. Your comment is awaiting moderation.

    Excellent advice. Cheers.
    canada viagra
    online canadian pharcharmy – https://www.canadianwwwonlinepharmacy.com/

    online rx pharmacy Ml b9efa0b

  2. Your comment is awaiting moderation.

    Nicely put. With thanks.
    ed meds online without doctor prescription
    canadian pharmacies without an rx – https://www.canadianpharmacyu.com/

    online canadian pharcharmy Et 8b13_66

  3. Your comment is awaiting moderation.

    Nicely put, Thanks a lot!
    best 10 online canadian pharmacies
    best canadian prescription prices – https://www.canadianpharmacy-guide.com/

    canadian pharmacies online prescriptions Pell 0606202

  4. Your comment is awaiting moderation.

    Wow a lot of helpful data!
    canada pharmaceuticals online
    canadian pharmacies that ship to us – https://www.trustedwebpharmacy.com/

    ed meds online without doctor prescription Sr 0406062

  5. Your comment is awaiting moderation.

    Whoa a lot of helpful tips!
    canadian pharmacy world
    no prescription pharmacies – http://canadianpharmaceuticalsonlinerx.com/

    online pharmacy without prescription kele aff7b9e

  6. Your comment is awaiting moderation.

    Fantastic facts. Thank you!
    canadian pharcharmy
    canadian pharmacy reviews – https://www.canadianpharmacy-ltd.com/

    canadian pharmacies online Pt 8904060

  7. Your comment is awaiting moderation.

    Nicely put, Appreciate it.
    canadian pharmacies online
    online canadian pharcharmy – https://www.canadianpharmacyopen.com/

    canada pharmacies prescription drugs Paky 02d94b8

  8. Your comment is awaiting moderation.

    Wonderful posts. With thanks.
    canadian pharmacies
    prescription drugs without doctor approval – https://canadianpharmaciescubarx.com/

    canadian pharmacies without prescriptions Hism ff7b9ef

  9. Your comment is awaiting moderation.

    Cheers! Awesome information!
    canadian pharcharmy
    canadian mail order pharmacies – https://www.canadianwwwonlinepharmacy.com/

    canadian pharmaceuticals online el b9efa0b

  10. Your comment is awaiting moderation.

    Seriously quite a lot of awesome data.
    online canadian pharmacy
    canada drugs direct – https://www.canadianpharmacy-ltd.com/

    canada pharmacy biT 1e89040

  11. Your comment is awaiting moderation.

    Useful posts. Appreciate it!
    canadian pharmacy viagra
    cialis from canada – https://www.canadianpharmacyopen.com/

    canadapharmacyonline kato 0_66570

  12. Your comment is awaiting moderation.

    This is nicely put. !
    canadian pharmaceuticals online
    canadian pharmaceuticals online – https://www.canadianonlinepharmacytrust.com/

    trusted overseas pharmacies Ma 1e89040

  13. Your comment is awaiting moderation.

    Great forum posts. Kudos!
    aarp approved canadian online pharmacies
    canada online pharmacies – https://www.safeonlinecanadian.com/

    health canada drug database Kn 6202d94

  14. Your comment is awaiting moderation.

    Thank you! Ample stuff!

    canadian pharcharmy
    canadian drugstore – https://www.canadianpharmacy-guide.com/

    prescription drug sn 0406062

  15. Your comment is awaiting moderation.

    You actually reported it perfectly. online canadian pharmacy canadian drugstore reviews – http://canadianpharmaceuticalsonlinerx.com/ canadian prescription gob aff7b9e

  16. Your comment is awaiting moderation.

    You have made your point!
    canadian pharmaceuticals online reviews
    highest rated canadian pharmacies – https://www.canadianpharmacy-ltd.com/

    canadian prescriptions online Pt 2d94b8b

  17. Your comment is awaiting moderation.

    Nicely put, Regards.
    canadian pharmacy world
    online pharmacies of canada – https://www.canadianpharmacyopen.com/

    canadianpharmacyking.com Sa 1e89040

  18. Your comment is awaiting moderation.

    You’ve made your point very effectively!.
    canadian pharmacies that ship to us
    canadian online pharmacies legitimate – https://www.pharmacyonline-canadian.com/

    canadian rx pharmacy online mn 02d94b8

  19. Your comment is awaiting moderation.

    Reliable postings. Thanks a lot.
    canadian pharmacy king
    discount prescription drug – https://www.canadianpharmacyopen.com/

    canadian pharmaceuticals online reviews ea b1e8904

  20. Your comment is awaiting moderation.

    You revealed this superbly.
    canadian pharmacies online
    canada pharmacies online pharmacy – https://www.canadianpharmacy-ltd.com/

    drug price kah 606202d

  21. Your comment is awaiting moderation.

    Great facts. Many thanks!
    online pharmacies
    canadian pharmacies-24h – https://www.canadianpharmacycom.com/

    online pharmacies canada kn 9040606

  22. Your comment is awaiting moderation.

    This is nicely said. .
    canadian pharmaceuticals online
    prescription cost comparison – https://www.canadianwwwonlinepharmacy.com/

    canadian drugs without prescription fog 5aff7b9

  23. Your comment is awaiting moderation.

    Very good information. Thanks.
    canadian viagra
    the canadian pharmacy – https://www.canadianpharmacy-ltd.com/

    canada pharma limited llc ml b8b14_6

  24. Your comment is awaiting moderation.

    Incredible tons of useful data.
    buy medication without an rx
    canadian pharmacies-247 – https://www.canadianpharmacyu.com/

    canadian pharmacy drugs online nex b9efa0b

  25. Your comment is awaiting moderation.

    Really tons of amazing tips.
    canadian online pharmacy
    best online pharmacy stores – https://www.trustedwebpharmacy.com/

    rx pharmacy Faws 0406062

  26. Your comment is awaiting moderation.

    Cheers. I value this!
    ed meds online without doctor prescription
    canada viagra – https://www.canadianonlinepharmacytrust.com/

    canadian pharcharmy Nib b1e8904

  27. Your comment is awaiting moderation.

    Cheers, Quite a lot of data.

    ed meds online without doctor prescription
    best online pharmacy – https://www.canadianpharmacycom.com/

    canada pharmacy no prescription Sn efa0b1e

  28. Your comment is awaiting moderation.

    Info very well utilized.!
    canadian pharmacy
    canadian king pharmacy – http://canadianpharmacyntv.com/

    overseas pharmacies Hism fa0b1e8

  29. Your comment is awaiting moderation.

    You explained it exceptionally well.
    london drugs canada
    canadianpharmacyservice.com – https://www.safeonlinecanadian.com/

    canadian pharmaceuticals online reviews zed f7b9efa

  30. Your comment is awaiting moderation.

    You made the point.
    canadianpharmacy
    canadian pharmaceuticals online reviews – https://www.canadianpharmacy-ltd.com/

    canadian drugstore reviews vect 6202d94

  31. Your comment is awaiting moderation.

    Nicely put, Appreciate it.
    canadian pharmacies
    canadapharmacyonline – https://www.canadianpharmacyopen.com/

    drugs from canada Dug 0406062

  32. Your comment is awaiting moderation.

    Wonderful stuff. With thanks.
    canadian pharcharmy
    walgreens pharmacy – https://www.canadianonlinepharmacytrust.com/

    buying prescription drugs canada Bret fa0b1e8

  33. Your comment is awaiting moderation.

    Nicely put. Regards!
    trust pharmacy canada
    online canadian pharmacies – https://www.canadianpharmacy-ltd.com/

    canadadrugsonline Cics 4060620

  34. Your comment is awaiting moderation.

    Whoa a lot of beneficial material!
    ed meds online without doctor prescription
    cialis online – https://www.canadianpharmacy-ltd.com/

    medication costs mig 18_7fa5

  35. Your comment is awaiting moderation.

    You definitely made your point!
    online canadian pharmacy
    canadian pharmacy certified canada pharmacy online – https://www.canadianwwwonlinepharmacy.com/

    drug price Mag 94b8b12

  36. Your comment is awaiting moderation.

    You expressed it fantastically.
    canadian pharmacy
    compare prescription prices – https://www.pharmacyonline-canadian.com/

    prescription drug assistance Paky ff7b9ef

  37. Your comment is awaiting moderation.

    Awesome material. Kudos!
    health canada drug database
    canadian pharcharmy online no precipitation – https://www.pharmacyonline-canadian.com/

    canadian pharcharmy onlinecanadian online pharmacy Rind 9040606

  38. Your comment is awaiting moderation.

    You actually suggested it wonderfully!
    canada rx
    canadian online pharmacies legitimate – https://www.canadianpharmacycom.com/

    pharmacy prices mn e890406

  39. Your comment is awaiting moderation.

    Very good advice. Thanks a lot!
    canadian viagra
    compare prescription prices – https://www.safeonlinecanadian.com/

    prescription cost pade ff7b9ef

  40. Your comment is awaiting moderation.

    Nicely put, Appreciate it.
    northwest pharmacy
    walgreens pharmacy – https://canadianpharmaciescubarx.com/

    online pharmacy no prescription needed fat 06202d9

  41. Your comment is awaiting moderation.

    You expressed it fantastically!
    canadian pharmacies
    best online pharmacy stores – https://www.canadianpharmacy-ltd.com/

    canadian pharmacy reviews Rall f7b9efa

  42. Your comment is awaiting moderation.

    Terrific stuff. Kudos!
    canada pharmaceuticals online
    canadian prescription drugs – https://www.canadianpharmacyu.com/

    costco pharmacy pricing pade 06202d9

  43. Your comment is awaiting moderation.

    With thanks! I like this.
    canadian pharmacy
    canada drug – https://www.canadianpharmacy-ltd.com/

    cheap drugs online ot 4060620

  44. Your comment is awaiting moderation.

    Good data. Thanks a lot.
    canadianpharmacy
    canadian discount pharmacy – https://www.safeonlinecanadian.com/

    canada pharmaceuticals online Kr 1e89040

  45. Your comment is awaiting moderation.

    You explained this well!
    canadian pharmacy king
    cvs pharmacy online – https://www.pharmacyonline-canadian.com/

    drugs from canada save 8b16_8f

  46. Your comment is awaiting moderation.

    Awesome stuff. Cheers!
    canadian pharmaceuticals online
    list of approved canadian pharmacies – https://www.canadianpharmacyu.com/

    legal canadian prescription drugs online Sa efa0b1e

  47. Your comment is awaiting moderation.

    Terrific forum posts. Thanks a lot.
    canadian pharmaceuticals online
    canadian online pharmacies legitimate by aarp – https://www.canadianpharmacyu.com/

    pain meds online without doctor prescription ka b16_665

  48. Your comment is awaiting moderation.

    Many thanks! I value it.
    online pharmacies in usa
    canadian pharcharmy online fda approved – https://www.pharmacyonline-canadian.com/

    no prescription pharmacy hew b1e8904

  49. Your comment is awaiting moderation.

    Wonderful forum posts. Appreciate it!
    legal canadian prescription drugs online
    legitimate online pharmacies – https://www.canadianpharmacyopen.com/

    online pharmacy without a prescription ka 7b9efa0

  50. Your comment is awaiting moderation.

    Incredible quite a lot of valuable information.
    canadian online pharmacies legitimate by aarp
    canadapharmacyonline com – https://www.canadianwwwonlinepharmacy.com/

    canadapharmacy com ka 0606202

  51. Your comment is awaiting moderation.

    Excellent data. Cheers.
    canada online pharmacies
    canadian pharmacy no prescription – https://www.canadianpharmacyopen.com/

    drugs from canada with prescription Pa 2d94b8b

  52. Your comment is awaiting moderation.

    You expressed it fantastically!
    canadian pharcharmy
    prescription drug price comparison – https://www.pharmacyonline-canadian.com/

    trust pharmacy canada Vat a0b1e89

  53. Your comment is awaiting moderation.

    Superb advice. Cheers!
    canadianpharmacy
    canada pharmacies/account – http://canadianpharmacyntv.com/

    londondrugs disp b9efa0b

  54. Your comment is awaiting moderation.

    Fantastic stuff. Thanks!
    pharmacy northwest canada
    canada drugs online – http://canadianpharmaceuticalsonlinerx.com/

    pharmacies near me Hism a0b1e89

  55. Your comment is awaiting moderation.

    Thanks. Terrific stuff.
    canadian pharmacy king
    canadian online pharmacy reviews – https://www.pharmacyonline-canadian.com/

    north west pharmacy canada hew 0b1e890

  56. Your comment is awaiting moderation.

    Truly loads of superb information!
    canadian pharmacy king
    best online pharmacy – https://www.canadianpharmacy-guide.com/

    canadian pharmacies that ship to us hew b9efa0b

  57. Your comment is awaiting moderation.

    You have made your point!
    canada pharmaceuticals online
    price prescription drugs – https://www.canadianpharmacycom.com/

    walgreens pharmacy hew b8b12_7

  58. Your comment is awaiting moderation.

    Great data. Appreciate it.
    ed meds online without doctor prescription
    canadian pharmacies – https://www.canadianonlinepharmacytrust.com/

    canadian pharmacies-247 Mr 6202d94

  59. Your comment is awaiting moderation.

    Well spoken genuinely! .
    canadian pharcharmy
    mail order pharmacies – https://www.canadianpharmacyu.com/

    canada pharmacy online Pi 6_8f9f0

  60. Your comment is awaiting moderation.

    Many thanks, Quite a lot of advice.

    canadian drug
    buy prescription drugs without doctor – https://canadianpharmaciescubarx.com/

    king pharmacy lime 02d94b8

  61. Your comment is awaiting moderation.

    You actually explained this wonderfully.
    canadian pharmacy
    canadian viagra – https://www.canadianpharmacy-guide.com/

    prescription cost comparison Ma fa0b1e8

  62. Your comment is awaiting moderation.

    Kudos! Useful stuff!
    canadian pharmacy world
    doctor prescription – https://www.safeonlinecanadian.com/

    king pharmacy Kah 9040606

  63. Your comment is awaiting moderation.

    You made your point very effectively!.
    canadian pharmaceuticals online
    medical information online – https://www.pharmacyonline-canadian.com/

    canadian pharcharmy hepe 4060620

  64. Your comment is awaiting moderation.

    Fantastic advice. Thank you.
    best 10 online canadian pharmacies
    online canadian discount pharmacy – https://www.pharmacyonline-canadian.com/

    cialis online fat 8904060

  65. Your comment is awaiting moderation.

    Many thanks. An abundance of forum posts.

    canadian pharmacy king
    online pharmacies of canada – https://www.canadianpharmacyopen.com/

    online pharmacies no prescription sen f7b9efa

  66. Your comment is awaiting moderation.

    Nicely put. Cheers!
    online pharmacy
    canadian prescriptions – https://www.pharmacyonline-canadian.com/

    canada drugs online pharmacy fat 4b8b12_

  67. Your comment is awaiting moderation.

    Excellent stuff. Regards!
    online pharmacies
    my canadian pharmacy – https://www.trustedwebpharmacy.com/

    pharmacy tech Nib 4b8b13_

  68. Your comment is awaiting moderation.

    Regards. An abundance of postings!

    online pharmacy
    pharmacy prices compare – https://www.canadianwwwonlinepharmacy.com/

    pharmacy without dr prescriptions box 0406062

  69. Your comment is awaiting moderation.

    Really all kinds of helpful material!
    canadian pharmacies that ship to us
    no 1 canadian pharcharmy online – https://www.canadianonlinepharmacytrust.com/

    canadian pharmacies without an rx Ma 0406062

  70. Your comment is awaiting moderation.

    Cheers! An abundance of forum posts!

    canadian pharmaceuticals online reviews
    online pharmacies canada – https://www.safeonlinecanadian.com/

    canadian online pharmacies legitimate by aarp Pa 8904060

  71. Your comment is awaiting moderation.

    Superb info. Appreciate it!
    canadian pharmaceuticals online
    canada meds – http://canadianpharmaceuticalsonlinerx.com/

    prescription drug price comparison ka 202d94b

  72. Your comment is awaiting moderation.

    Truly many of wonderful material.
    viagra 100mg
    most reliable generic viagra site – https://www.viagraforsalemsn.com/
    viagra cialis without doctors prescription
    use of viagra in hot tub PatrickFep 6_8daab

  73. Your comment is awaiting moderation.

    Excellent stuff. Appreciate it.
    viagra online
    experience of women using viagra – https://www.viagraforsalemsn.com/
    viagra peak blood levels
    viagra available safe KennethAbulK a0b1e89

  74. Your comment is awaiting moderation.

    Seriously lots of excellent facts!
    viagra for women
    practical benefits of viagra – https://www.genericonlineviagrarx.com/viagra-generic/
    france viagra
    rx viagra 100mg Lelandmut 9efa0b1

  75. Your comment is awaiting moderation.

    Many thanks! Useful stuff.
    cheap viagra
    find search viagra edinburgh free – https://genericviagramsl.com/
    better viagra or levitra
    viagra and cialis and JasonWaibe 4b8b12_

  76. Your comment is awaiting moderation.

    Wonderful facts. Appreciate it!
    viagra
    unions fight for viagra – https://www.viagraforsalemsn.com/
    viagra empty stomache
    buying viagra delivered worldwide EdwardGrawl 2d94b8b

  77. Your comment is awaiting moderation.

    Seriously a good deal of terrific information.
    viagra for women
    viagra failures – https://genericviagrantx.com/
    viagra for sale in usa counter
    tadalafil levitra generic viagra Joshuapiova 5aff7b9

  78. Your comment is awaiting moderation.

    This is nicely expressed! .
    viagra without a doctors prescription
    sub lingual viagra response – https://www.viagraonlinemsn.com/
    viagra and india
    viagra find order search pages edinburgh Patrickbut b9efa0b

  79. Your comment is awaiting moderation.

    With thanks, Loads of tips.

    buy viagra online
    viagra versus cialis versus levitra – https://genericviagramsl.com/
    long term effects viagra cialis
    buying viagra from mexico Lelandtug 0406062

  80. Your comment is awaiting moderation.

    Nicely voiced genuinely! .
    viagra for sale
    10mg vs 20mg viagra experiance – https://genericviagrarcp.com/
    491 viagra heart 708
    viagra with or without food Jasonindum e890406

  81. Your comment is awaiting moderation.

    Tips effectively utilized..
    buy viagra online
    reditabs buy viagra – https://www.viagraforsalemsn.com/
    viagra masturbation
    viagra and cavenous nerves KennethLib b14_7e4

  82. Your comment is awaiting moderation.

    Reliable write ups. Many thanks.
    viagra online
    polish viagra – https://www.viagraonlinebuymsn.com/
    viagra maker
    buy viagra pill online Edwardfap 0606202

  83. Your comment is awaiting moderation.

    Appreciate it. Loads of posts!

    buy viagra online
    herbs and viagra interaction – https://www.viagraforsalemsn.com/
    paypal accepted viagra
    viagra official site Joshuavam 5aff7b9

  84. Your comment is awaiting moderation.

    You actually stated this adequately.
    generic viagra online
    viagra or cialis – https://genericviagrantx.com/
    g postmessage viagra guest online
    kevin harvick viagra PatrickBef 1e89040

  85. Your comment is awaiting moderation.

    Terrific postings, Cheers.
    viagra pills
    can someone on amiloride take viagra – https://genericviagrarcp.com/
    viagra for wemen
    viagra longer erection KennethIllet 202d94b

  86. Your comment is awaiting moderation.

    Thank you! Ample postings!

    viagra
    keyword order viagra – https://www.viagraonlinemsn.com/
    viagra find viagra edinburgh search
    black market viagra Edwardtuh aff7b9e

  87. Your comment is awaiting moderation.

    Awesome tips. Regards!
    generic viagra 100mg
    daily viagra – https://genericviagrantx.com/
    viagra illegal use
    viagra cialis levitra compare JoshuaVOX 5aff7b9

  88. Your comment is awaiting moderation.

    Great facts. Cheers.
    generic viagra online
    viagra commercial smokestack – https://genericviagrantx.com/
    take cialis soft and viagra together
    viagra prescription low cost Lelandtix e890406

  89. Your comment is awaiting moderation.

    You actually explained it effectively!
    viagra pills
    arti viagra – https://genericviagramsl.com/
    le viagra
    buy viagra online 3$ per pill JasonLox 0606202

  90. Your comment is awaiting moderation.

    You revealed it exceptionally well!
    viagra 100mg
    if viagra doesn’t work – https://genericviagramsl.com/
    the viagra myth
    generic indian viagra KevinNUM aff7b9e

  91. Your comment is awaiting moderation.

    Awesome postings. Thanks.
    viagra
    ed treatments when viagra doesn’t work – https://genericviagrarcp.com/
    old men viagra videos
    brand name authentic viagra Kennethhet e890406

  92. Your comment is awaiting moderation.

    Regards. I value it!
    generic viagra online
    viagra canine perscription – https://genericviagramsl.com/
    viagra levitra cialis stuffy nose
    male sexual enhancing pills viagra Patrickjes 9040606

  93. Your comment is awaiting moderation.

    Info nicely taken!.
    viagra 100mg
    inexpensive viagra – https://www.viagraforsalemsn.com/
    information drug viagra
    viagra insurance medben Edwardicofs 1e89040

  94. Your comment is awaiting moderation.

    Wow many of good knowledge!
    generic viagra
    viagra interaction with nebivolol – https://www.viagraonlinemsn.com/
    generics mexico viagra
    anti viagra Joshualom aff7b9e

  95. Your comment is awaiting moderation.

    You actually mentioned that superbly!
    buy generic viagra
    chemical name of viagra – https://genericviagramsl.com/
    sites computer viagra find edinburgh free
    dominican viagra LelandChemn 5aff7b9

  96. Your comment is awaiting moderation.

    Thanks! Numerous facts!

    buy viagra
    viagra demonstration – https://genericviagramsl.com/
    sex valium and viagra
    viagra and preature ejaculation Jasonnuags fa0b1e8

  97. Your comment is awaiting moderation.

    Reliable forum posts. Regards.
    viagra for sale uk
    viagra fda approval – https://www.viagraforsalemsn.com/
    ph lipaphen viagra women
    viagra levitra cialis ads PatrickKiz 4b8b18_

  98. Your comment is awaiting moderation.

    Excellent posts. Kudos.
    generic viagra online
    buy viagra online from canada – https://www.viagraonlinebuymsn.com/
    is viagra safe for teenagers
    key acquisto viagra Kennethstarm b9efa0b

  99. Your comment is awaiting moderation.

    You actually said it well!
    generic viagra
    viagra online issues – https://genericviagramsl.com/
    uses of viagra in women
    purchase viagra online LelandExcuh 0b1e890

  100. Your comment is awaiting moderation.

    Thanks a lot, Great stuff!
    viagra
    soung viagra on the water – https://www.genericonlineviagrarx.com/viagra-generic/
    viagra triangle westlake ohio viva barcelona
    natural supplement viagra JasonHam 606202d

  101. Your comment is awaiting moderation.

    Info clearly regarded.!
    viagra generic
    uk alternative viagra – https://www.genericonlineviagrarx.com/viagra-generic/
    viagra online sales
    viagra comparison EdwardAnecy b1e8904

  102. Your comment is awaiting moderation.

    Cheers! Fantastic stuff!
    generic viagra 100mg
    robin williams when bush takes viagra – https://genericviagrantx.com/
    viagra head office toronto
    genuine viagra without prescription JoshuaBom 15_534e

  103. Your comment is awaiting moderation.

    You reported that very well.
    viagra without a doctors prescription
    viagra bought in mexico – https://www.viagraforsalemsn.com/
    viagra oz
    viapro better than viagra Patrickapefs 06202d9

  104. Your comment is awaiting moderation.

    Amazing plenty of fantastic data.
    buy viagra
    caverjet better viagra – https://genericviagrantx.com/
    viagra guaranteed certified
    viagra price strategy Lelandmem ff7b9ef

  105. Your comment is awaiting moderation.

    You actually revealed this really well!
    viagra online
    discount viagra perscription drug – https://genericviagramsl.com/
    blog viagra
    canad viagra Jasonunest 606202d

  106. Your comment is awaiting moderation.

    Fantastic material, Regards.
    viagra without a doctors prescription
    edinburgh uk viagra pages charles linskaill – https://www.genericonlineviagrarx.com/viagra-generic/
    viagra impotence pill
    best place to buy viagra EdwardCon fa0b1e8

  107. Your comment is awaiting moderation.

    Factor very well used..
    buy viagra online
    viagra news edinburgh tid moo search – https://genericviagrarcp.com/
    viagra eqivalent herbal tablets
    viagra and massage JoshuaScoca d94b8b1

  108. Your comment is awaiting moderation.

    Wonderful info. Many thanks.
    viagra
    viagra and pulmonary arterial hypertension – https://www.viagraforsalemsn.com/
    sideeffects of viagra
    mdma viagra combo on the street Patricksot 0b1e890

  109. Your comment is awaiting moderation.

    Great information. Cheers.
    viagra generic
    jalepeno and viagra – https://genericviagramsl.com/
    doxazosin and viagra
    subaction showcomments viagra smile watch LelandFiday 06202d9

  110. Your comment is awaiting moderation.

    Fine advice. Kudos!
    buy viagra online
    viagra drug store – https://genericviagrarcp.com/
    reliability of generic viagra
    va regulations viagra Kennethstarm 7b9efa0

  111. Your comment is awaiting moderation.

    Superb data. With thanks!
    generic viagra
    buy viagra line – https://www.viagraforsalemsn.com/
    cialis pharmacy rx viagra
    viagra c best supplier EdwardErers e890406

  112. Your comment is awaiting moderation.

    Thanks! Helpful stuff!
    generic viagra
    cheap viagra online – https://www.viagraonlinemsn.com/
    sean prentice viagra daily record
    fox news viagra women prozac JoshuaScien 94b8b14

  113. Your comment is awaiting moderation.

    Wow all kinds of helpful advice.
    cheap viagra
    viagra hamster jetlag – https://www.viagraonlinemsn.com/
    women on viagra
    does viagra result in longer erections PatrickSix 6202d94

  114. Your comment is awaiting moderation.

    You made your point very clearly!!
    viagra 100mg
    viagra equivalent for women – https://www.viagraonlinemsn.com/
    directions for taking viagra
    viagra stock ticker LelandKak a0b1e89

  115. Your comment is awaiting moderation.

    You have made your position extremely clearly.!
    buy viagra
    natural over the counter cialas viagra – https://genericviagrantx.com/
    viagra after effect
    redneck viagra Jasonhof b8b12_1

  116. Your comment is awaiting moderation.

    You explained that exceptionally well.
    generic viagra online
    health shop port louis generic viagra – https://genericviagrarcp.com/
    lexapro and viagra
    blue moon viagra KennethHix 606202d

  117. Your comment is awaiting moderation.

    With thanks, Excellent stuff!
    generic viagra online
    irish viagra – https://genericviagrantx.com/
    50mg viagra uk
    alternative to viagra Patricksow 5aff7b9

  118. Your comment is awaiting moderation.

    Kudos. I like it!
    cheap viagra
    who created extenze and viagra – https://www.viagraforsalemsn.com/
    men’s health natural viagra
    obtain a prescription for viagra Edwardcal 94b8b18

  119. Your comment is awaiting moderation.

    Nicely put, Thanks.
    generic viagra online
    india generic viagra legal – https://www.viagraforsalemsn.com/
    can companies make generic viagra
    legally purchase viagra LelandMub d94b8b1

  120. Your comment is awaiting moderation.

    Regards! Wonderful information.
    viagra
    viagra description – https://www.viagraonlinemsn.com/
    mccain avoids viagra cooper
    alcohol viagra Danielbub f7b9efa

  121. Your comment is awaiting moderation.

    You stated it wonderfully.
    viagra for sale uk
    edinburgh 18 pages viagra good search – https://genericviagrantx.com/
    generic viagra indian fda
    buy viagra search find 76k Patrickrah ff7b9ef

  122. Your comment is awaiting moderation.

    Incredible all kinds of helpful tips.
    cheap viagra
    viagra and 4 – https://www.viagraonlinemsn.com/
    find search viagra edinburgh computer
    viagra compare Kennethviona 9efa0b1

  123. Your comment is awaiting moderation.

    Thanks. I appreciate it.
    viagra for sale
    generic viagra from china – https://genericviagramsl.com/
    viagra good morning papa mp3
    viagra discard date LelandFiday 5aff7b9

  124. Your comment is awaiting moderation.

    Cheers! Lots of stuff!

    viagra without a doctor prescription
    viagra sweatshop – https://www.viagraforsalemsn.com/
    viagra wp
    viagra title object object Jasonhal 9040606

  125. Your comment is awaiting moderation.

    You made your position very clearly.!
    generic viagra online
    clarisa nora doctorovich viagra – https://www.viagraonlinemsn.com/
    viagra wisconsin teachers
    snow white and viagra Joshuaguism 06202d9

  126. Your comment is awaiting moderation.

    Regards. Loads of tips!

    viagra online
    viagra effect porn photos – https://genericviagrantx.com/
    cheap buy viagra
    non prescription viagra substitute Patrickstema 5aff7b9

  127. Your comment is awaiting moderation.

    Nicely voiced truly. .
    generic viagra online
    can you take viagra with lexapro – https://www.viagraonlinebuymsn.com/
    where to buy viagra in bangkok
    generic viagra differences Lelandfex 9040606

  128. Your comment is awaiting moderation.

    Many thanks! Lots of knowledge!

    buy generic viagra
    purchase and viagra soft – https://genericviagrantx.com/
    viagra moa
    herbal viagra cartridges Jasonlob d94b8b1

  129. Your comment is awaiting moderation.

    You have made your point!
    viagra pills
    viagra slaes – https://genericviagrantx.com/
    viagra on the internet
    erections while on viagra KennethhervE d94b8b1

  130. Your comment is awaiting moderation.

    Regards, Excellent information.
    cialis
    efficacite cialis 10mg – http://cialisbirx.com/cialis+40+mg+generic/cialis-online-ec-37
    cialis 20mg

  131. Your comment is awaiting moderation.

    Whoa a good deal of excellent facts!
    viagra 100mg
    best website use generic viagra – http://viagramsnrx.com/best+viagra/for-sale-viagra-42-3b
    viagra generic
    generic viagra verses brand viagra Kevingresk 9efa0b1

  132. Your comment is awaiting moderation.

    You have made your position pretty effectively.!
    cialis online
    daily cialis benefits – https://cialismsnrx.com/
    cialis 20mg
    can cialis raise psa Danielbub d94b8b1

  133. Your comment is awaiting moderation.

    Many thanks. I like this.
    cialis 20 mg
    hay cialis generico en las farmacias – https://cialismsnrx.com/
    cialis generic
    costo del cialis originale Danielclori 5aff7b9

  134. Your comment is awaiting moderation.

    You actually said it terrifically!
    generic cialis
    liquid cialis ingredients – http://cialisbirx.com/cialis+40+mg+generic/cialis-online-ec-37
    buy cialis pills

  135. Your comment is awaiting moderation.

    You said it nicely.!
    buy generic viagra
    insurance online viagra – http://viagramsnrx.com/best+viagra/for-sale-viagra-42-3b
    viagra online
    professional viagra cheap KevinNUM 7b9efa0

  136. Your comment is awaiting moderation.

    Thanks, Excellent stuff.
    cheap cialis
    cialis or viagra after prostatectomy – http://cialisbirx.com/cialis+40+mg+generic/cialis-online-ec-37
    cialis 20mg

  137. Your comment is awaiting moderation.

    Great posts. Regards.
    cheap drugs
    canadian prescription drugstore – http://canadianpharmacyntv.com/

    buy prescription drugs from canada mn 8904060

  138. Your comment is awaiting moderation.

    You said it nicely.!
    london drugs canada
    rx pharmacy – http://canadianpharmacyntv.com/

    canada meds Nax 5aff7b9

  139. Your comment is awaiting moderation.

    Terrific forum posts. Many thanks!
    buy medication without an rx
    mexican pharmacy online medications – https://www.canadianonlinepharmacytrust.com/

    online pharmacy without prescription nex aff7b9e

  140. Your comment is awaiting moderation.

    Many thanks! I like this!
    canadian viagra
    online pharmacy reviews – https://canadianpharmaciescubarx.com/

    canadian prescriptions Fap 5aff7b9

  141. Your comment is awaiting moderation.

    Thanks a lot, Useful information.
    canada pharmaceuticals online
    mexican pharmacies shipping to usa – http://canadianpharmaceuticalsonlinerx.com/

    prescription drugs online without doctor Fuct 2_66570

  142. Your comment is awaiting moderation.

    Kudos! I enjoy this.
    ed meds online without doctor prescription
    best online canadian pharcharmy – https://canadianpharmaciescubarx.com/

    online discount pharmacy Ma a0b1e89

  143. Your comment is awaiting moderation.

    Appreciate it. A lot of info.

    canadian pharmacy drugs online
    canadian pharmacy king – https://canadianpharmaciescubarx.com/

    canadian pharmacy king jast 5aff7b9

  144. Your comment is awaiting moderation.

    You actually explained this really well!
    canadian pharmacies
    canadian king pharmacy – http://canadianpharmacyntv.com/

    canada drugs mew 4060620

  145. Your comment is awaiting moderation.

    Wow plenty of good info. canadian drug best online pharmacy – https://www.canadianonlinepharmacytrust.com/ canada online pharmacy gob 8b17_8f

  146. Your comment is awaiting moderation.

    Great write ups. Cheers!
    ed meds online without doctor prescription
    canadian pharmacies without prescriptions – https://www.canadianonlinepharmacytrust.com/

    pharmacy northwest canada nex aff7b9e

  147. Your comment is awaiting moderation.

    Thanks. I value this.
    canadian pharmacies
    canadian drug stores online – http://canadianpharmacyntv.com/

    pharmacy prices compare Foes 0406062

  148. Your comment is awaiting moderation.

    Whoa all kinds of amazing tips.
    ed meds online without doctor prescription
    canadianpharmacyusa24h – https://www.canadianonlinepharmacytrust.com/

    24 hour pharmacy Ol 9040606

  149. Your comment is awaiting moderation.

    You actually stated it superbly.
    canada pharmaceuticals online
    northwestpharmacy – http://canadianpharmacyntv.com/

    mexican online pharmacies ot 1e89040

  150. Your comment is awaiting moderation.

    Seriously plenty of helpful data.
    trust pharmacy canada
    canadian pharmacy without prescription – http://canadianpharmaceuticalsonlinerx.com/

    online pharmacy without prescription Wame 5aff7b9

  151. Your comment is awaiting moderation.

    Valuable postings. Appreciate it.
    best 10 online canadian pharmacies
    canada medication pharmacy – https://www.canadianonlinepharmacytrust.com/

    drugs without prescription nex b1e8904

  152. Your comment is awaiting moderation.

    You expressed this fantastically!
    canadian pharmacy king
    canadian drugstore – https://www.canadianonlinepharmacytrust.com/

    online pharmacies canada Pa 202d94b

  153. Your comment is awaiting moderation.

    Helpful postings. Many thanks.
    canadian drugs
    online rx pharmacy – https://canadianpharmaciescubarx.com/

    londondrugs Bumn efa0b1e

  154. Your comment is awaiting moderation.

    Kudos, Lots of information.

    canadian viagra
    prescription cost – http://canadianpharmaceuticalsonlinerx.com/

    buy prescription drugs online Ma aff7b9e

  155. Your comment is awaiting moderation.

    Really all kinds of excellent material.
    canadian pharmacies
    canadian pharmacy viagra – http://canadianpharmaceuticalsonlinerx.com/

    canada pharmacies online Vaf aff7b9e

  156. Your comment is awaiting moderation.

    Information clearly used.!
    canadian pharmacies
    cialis from canada – https://www.canadianonlinepharmacytrust.com/

    drugstore online shopping Pt 1e89040

  157. Your comment is awaiting moderation.

    Excellent data. With thanks.
    canadian pharmacy drugs online
    canadianpharmacyservice.com – http://canadianpharmaceuticalsonlinerx.com/

    canadian mail order pharmacies kato ff7b9ef

  158. Your comment is awaiting moderation.

    Terrific posts. Thanks a lot.
    canada pharmaceuticals online
    drugs from canada with prescription – https://www.canadianonlinepharmacytrust.com/

    northwest pharmaceuticals canada fat 8b14_9e

  159. Your comment is awaiting moderation.

    You revealed this terrifically.
    ordering prescriptions from canada legally
    meds online without doctor prescription – https://canadianpharmaciescubarx.com/

    ed meds online Bum 8904060

  160. Your comment is awaiting moderation.

    Point clearly applied.!
    canadian pharmacy
    canadianpharmacyusa24h – http://canadianpharmaceuticalsonlinerx.com/

    legitimate online pharmacies View b18_791

  161. Your comment is awaiting moderation.

    Perfectly expressed really! .
    canadian online pharmacies
    canada medication pharmacy – http://canadianpharmacyntv.com/

    canada rx Pn 0406062

Leave a Reply