viewtopic.php?f=7&t=10626&start=40
At any rate, in my comments, I mentioned I would look for an old post that I made that gave a glimpse of why our consumer testing process requires these layers of complexity. Im going to post that old post below and link to it for anyone who wants to read about it. For context, the below was a comment I made in regards to somebody talking about the nonsense of strain names. Its a topic I have talked about many times, but I was in the mood to blabber that night and touched on another favorite topic of mine - personal chemistry. If you are here to see my example of how response bias plays into the quality of data, you can scroll down to the part that shows actual survey data. Note that this is not the most clear-cut example, in actuality compared to what we see this is quite tame. However, its quick and easy to share, and dinner is almost ready so it will do the trick!
OLD POST BELOW:
Kudos to the original poster – using the word “strain” is something that I have been silently protesting for years. You’ve touched on something that drives me nuts -without a doubt this will result in me typing on the keyboard for hours, lol. I’ve been intentionally calling them “batches” for a few years now and when I do use the word “strain” I’m pretty sure I always put the word in quotes because I don’t accept its current use. I’m not always sure people understand why I avoid the word “strain”, but I do know it sometimes causes confusion when I am interacting with others. I would highly encourage everyone to switch to the word “batch”. This will allow people like me to have more sanity, but more importantly, it will draw attention to why there was a change in terminology to begin with (leading to people becoming educated on what a “strain” is and isn’t). I mean the word “strain” is so misleading to newcomers because it implies that there is some kind of standardization in the industry.
Batch names (which I will try to call “strains” for the sake of clarity on this post) with batch codes are very helpful for the purpose of identifying batches that you liked from a vendor and would like to order more of. However, in the current state of the industry, that is just about all there is to gain from the name on a menu (there are some vendor-specific exceptions.) The original poster described how this could be different if there were people working on ways to improve kratom similarly to the cannabis industry – I assure you there are some serious efforts under way!!!
Until then, I always recommend that consumers should consider a new batch of any “strain” as if they were not related at all… Its just a name. In fact just this morning (at the time of me starting to write this) I had a customer request a 1kg of a “strain” that we just added to the menu because they loved last years version of it so much. As you can see, I discouraged this and made the recommendation that they make a smaller order to test it out first.
Kratom is just like people - just because two people share the same name, doesn’t mean they are going to be similar people. Although a silly comparison, but its absolutely true for two primary reasons:
1. The vast, vast, majority of vendors have zero idea of where the products they import actually come from. To be clear - this is not a dig at "most vendors" - Id be a fool to think that I know with 100% certainty what tree every leaf we import has come from
. This is of course is one of the things that makes kratom vending a stressful profession. I’m sure everyone has experienced the displeasure of ordering a kilogram and being very disappointed with what arrived. Now imagine how that feels to a vendor when they order 20-1000kg’s … yeah it can be brutal, lol. Even the vendors who genuinely believe they know everything about the origins of their kratom – I am telling you…. they most likely really don’t. There is a lot of smoke and mirrors in the supply chain, even some of the most “plugged in” vendors who think they are above the fray are still sold a boat of lies (again I would be a fool to think we have not bought that boat several times). A lot of kratom is sold to US vendors by middlemen who might not even physically see the kratom they sell. And even if the exporter is actually involved with the production, they are often just buying dried leaves in bulk from another middleman and milling them into different shared of powder. In most cases (again, not all) "strain names" are just randomly assigned to batches in the storehouses of kratom exporters as they are being boxes up for export. One exception of course to the randomness is if the exporter has an order for specific strains – then whatever “strains” were ordered are the names that will be assigned. Unfortunately, this is just how much of the industry is - unless you actually have boots-on-the-ground support to help you with sourcing, you are really at the mercy of whatever you are told. I want to add that none of this is a knock on how vendors source their products, not knowing the actual origins of a product doesn’t mean you don’t know how to get a great product. After all, if the product is safe, clean, and high quality, I think that most consumers are not as concerned with what region the product came from. I wouldn’t have a clue on how to pick a good ranch to buy cattle from, but I do know that if I pick up a pack of prime porterhouse steaks from Kroger that they are going to be dang tasty nearly every time! 2. What about those vendors who do actually know the source of their products - well again, they most likely actually don’t, lol. But if they did - is there a lot to be gained by the consumer from accurate “strain names”? The answer to that is there might be some slight advantages – but for the most part it’s not nearly as helpful as you would think. A primary reason for this is that there is zero standardization of "strains" in the kratom industry.... a "Green Bali" from vendor A is nothing like a "Green Bali" from vendor B. Also, there is no such thing as this “X” type of “strain” usually has “Y” kind of characteristics. It is true that environmental conditions can heavily influence the overall alkaloid profile of a batch, but few of those conditions are “region-specific”. In fact, you can have many varieties of different “alkaloid profiles” on the same plot of land (not that most farmers are leveraging this fact in any way). We sometimes carry multiple batches from the same general plot of land, and not one of them are the same (trust me - I obsess over menu construction... if they were the same we would not even consider carrying them).
Further, even if you only buy from a single vendor, that vendor's Green Bali from March is not going to not be the same thing as their next batch of Green Bali in May (assuming they are different batches... check those batch codes). There may be similarities, but not enough for you to go all in and say, “I really liked the last Green Bali, so I’m going to go ahead and order 2 kilograms of the new Green Bali”.
Again, I have to refer to point one above. It is difficult (but not impossible) to know with certainty that you are getting product from the same group of trees or even the same farm. If a vendor is buying from a skilled and trusted supplier, they may have be able to reliably get the same “grade” of product as their last shipment (there is such a thing as kratom grades), but it doesn’t necessarily mean it will be from the same source. Even when a vendor is buying directly from a farmer, they are not always getting that farmers product (which might be a good thing… more on why that is later). There will be times that farmer doesn’t have any product ready and ir/when that occurs, the farmer will just buy raw leaves from another farmer before turning his US customer away. Kratom is just a commodity, its bought/sold in Indonesia just like corn is here. If a corn farmer just sold his entire harvest and some dummy on the other side of the world is willing to pay 10x the price for corn, that farmer is just going to buy corn off his neighbor and resell it for a nice profit.
Here is the kicker though – almost all of the above doesn’t really matter to begin with. Even if a vendor does acquire a batch that was harvested from the same exact trees and they accounted for all post-harvest processing variables (which is extremely difficult), it's most likely going to be an entirely different batch. We have gone to great extremes to try and replicate batches with varying degrees of success. In some cases, it might be similar or close (depending on the individual), but you should never expect that it will be just like the last batch. The analogy I used earlier in this post applies here too. Anyone who is a parent of more than one child can tell you that kids from the same set of parents are NOT the same. They may have some similarities, but even twins can have radically different personalities. The reality of the matter just a few weeks is enough time for a tree's alkaloid profile to change – sometimes radically. There are just so many environmental conditions that alter the tree’s production of secondary metabolites. Various genetic, ontogenic, morphogenetic and environmental factors can influence the biosynthesis and accumulation of secondary metabolites (the alkaloids we look for). Secondary metabolite accumulation is strongly dependent on a variety of environmental factors such as photoperiod, light intensity, temperature, soil water, soil fertility and salinity, and many others I am not going to share here. Further a single factor may alter the content of secondary metabolites even if all of the other factors remain fairly constant. As you can imagine, some of these can be controlled to a certain extent, but many of them can’t in a practical manner… sometimes the best time to harvest is a matter of timing. Even with a concerted effort to manipulate the pre-harvest variables (which no farmer is going to do on their own accord), it’s extremely difficult to replicate a specific batch month apart (sometimes you can get close though).
Speaking of getting close, check out the email I received 2 days ago from a tester. He had recently tested a genericly labeled sample and was able tell it's our Deep Jungle source based by comparying the flavor to the previous version (which we haven't offered in almost a year and a half now). Wow!
Despite the annoying fact of not being able to recreate previous versions, I personally view these changes as a blessing, because it gives the consumer more variety and combats SSS (same “strain” syndrome). In my experience, I have never seen a consecutive batch that was so close to the previous batch that there was an issue with cross-tolerance in regards to SSS. This means if you bought our 2020 version of 30 Year Wild Red TFH, even though the 2021 version comes from the same exact trees it is going to be an entirely different "strain" (both have a permanent spot in my rotation).
I mean even if you don’t know anything about kratom production – a little critical thinking will go a long way to back up what I’m saying. Kratom varies batch to batch just the same way a lot of other agricultural products do. Just because last time you went to the grocery store the green grapes were sour, doesn’t mean that the next batch of green grapes is going to be the same (even if they do come from the safe farm). You wouldn’t ever go to the store and say that last batch of green grapes were the best ever, so I’m going to go ahead and buy 4 bags now because they are just that good. Now, take that principal and apply it to kratom times 10 because kratom is an industry in its infancy. I have no doubt that with time the quality and consistency between kratom batches will improve just as many agricultural products are incredibly consistent via genetic engineering and advanced farming (cotton candy grapes anyone?).
A lot has changed since we began our journey and when we started, we were basically amateur enthusiasts. By borrowing existing techniques from the tea industry, layering in old-fashioned trial, and the occasional consultation with botanists on reddit voila… we are doing something! A lot can be done quickly by people who are passionate (okay, obsessed) and determined, but I am really curious what could occur if you paired that with some serious funding and technical know-how. I mean imagine if a company like Lipton offered a kratom tea and got involved with kratom production – who knows where it could go (I’m not saying I would like to see a big commercial takeover of kratom – but I am curious as to what they would discover). Be a giant corporation or us little guys, I really do think we will soon reach a point where personal chemistry is the only obstacle a consumer will face while finding kratom works exceptionally well for them.
Speaking of the phenomenon I call "personal chemistry", I’d like to share my 2 cents. Its an incredibly relevant topic that I don’t see covered much (with the exception of an email here or there with members). I’ve been putting off post on it for a long time – I guess you can say its bit of a touchy subject for me.
Your personal body chemistry is a HUGE factor in how you experience a batch – there is just no way around that fact. Its importance is completely underestimated and misunderstood in our community. Sure, I hear mentions of it here and there, but it gets nowhere near the credence it deserves. As a consumer, I have been guilty of ignoring its importance for years - despite noticing antidotally that my own preferences would occasionally change (some batches I loved would begin to fall out of favor with me, and other batches I didn’t care for would suddenly earn a spot in my rotation). Truthfully, prior to being involved in our research I had really never given it that much thought. Now personal chemistry is the thing that keeps me up at night – no joke! It adds a whole other level of complexity to what otherwise should be a pretty black and white process of producing or finding the best kratom possible.
When we started our little kratom project, I thought wholeheartedly that if we could just discern exactly what makes “great” kratom that we could ensure every batch would be a home run for everyone… oh how naive I was, lol. Don’t get me wrong, starting with a quality product will ABSOLUTELY can increase your odds of finding a batch that works well for you… it makes a big difference. However, if you goal is to make every batch a home-run for every customer, producing an incredibly product is only like ¾ of the battle. You can make the best product in the world, but at the end of the day it’s only the “best” if the person consuming it agrees. If we do ever find a “solve” for obstacle that personal chemistry, it will change the whole industry and take the consumer experience to the next level. Currently there are semi-effective ways to account for personal chemistry with the implementation of preference profiling, but it requires a lot of data and is a very time-consuming process … its just really hard to implement practically to scale (I hope there is somebody out there smarter than I is working on it that disagrees, lol).
Seriously, the complications in our mission that are rooted in personal chemistry are vast I cannot even begin to explain how big of a pain in the rear it is when you are trying to do something as simple as trying to figure out if a batch of kratom is poor, average, good, or fantastic. Factoring for people’s different preferences, baselines, etc, make our consumer testing process like 10x’s more complicated than it would be without. Still, like any worthy nemesis, I have grown to have a healthy respect for the challenges it presents, and even an appreciation for the few advantages it provides us. A few interesting things about personal chemistry are:
1. Personal chemistry can completely dictate how you experience a batch. As if that weren’t enough, a person’s chemistry/preference can change…. sometimes wildly. We have all heard the stories about how people dug in the old “discard box” and discovered a powder that they are now in love with. Antidotal experiences are interesting, they are far from scientific evidence. What you need to draw a reliable conclusion is hard data… and we have that data. At first chemistry was not even on our radar to study, but it became apparent quickly that there is something major to it. Now we see it all the time that a person might a batch very low in consumer testing (sometimes multiple times), but down the road they unknowingly test the same sample again and promptly ask to purchase a kilogram of it, lol. (Literally one of our most trusted testers recently asked for 5kg of a “strain” that he gave very harsh scores to in consumer testing a few months ago). I have to say, although maddening, it is all quite entertaining, lol. What further complicates things is our bodies also experience “glitches in our chemistry matrix” its not uncommon for consumers to have brief periods of time where our bodies are completely out of whack and nothing will work well (due to things like the time of day, consumption schedule, something you ate, or maybe a minor virus that you are unknowingly battling, etc.) Sometimes our bodies are just primed to have a great kratom session, and sometimes we are not. With that said, If you ever feel that you got "bait and switched" by a vendor who sent out an amazing sample but when you ordered it it didn’t seem the same… I would recommend withholding judgment. If it came from a reputable vendor, the odds are it was you who switched, not the powder, lol.
2. Another thing that is very interesting is how people’s perception of a batch can change literally by the day. When testers unknowingly test the same batch multiple times, we can often observe a great fluctuation of overall quality scores. More often, scores will start high and then they will gradually decrease the more times they test the same batch (this probably doesn’t surprise some of you out there). However, there are also some rare birds out there who tend to like batches more each time they try it. We have done many blind tests to study how continued use of the same batch affects its overall quality score by sending the same batch under different names, sometimes all at once, and sometimes months apart. Most of the time what we see is people will be more generous with their scoring the very first time they sample a batch. With these people, consecutive uses of the same batch will most often result in lower scores each time - and for some people it's a one and done (they score the first time very high, and then every score after that is very low). On the flip side (and less frequently) there are times where people start out not appreciating a batch, but over time come to appreciate it more. We don’t exactly know what the cause of these different behaviors are, because they are not even always consistent to the individual.
*As a side note – I had mentioned that there are some things that I appreciate about personal chemistry as a vendor – the phenomenon I describe above is one of them. Understanding how repeated use of the same batch influences perceived quality is extremely helpful for us as a vendor, especially whenever we are working with a supplier in which we don’t have direct oversight of production. Unfortunately, it’s VERY common for Indonesian suppliers to make 10-20 “strains” from a single harvest of leaves (by blending different ratios of red and green leaves). The end results are that a US vendor can order 10-15 “strains” from a supplier and what they are actually getting is a bunch of blends of the same leaf (sometimes the Indo suppliers don’t even try that hard and will just write different names on the same exact blend). This happens all the time – its super common and a lot of vendors are completely oblivious to it. I’d be willing to bet that almost everyone on this forum has bought several different “strains” from a vendor that were nothing more than a blend of the same leaf – and not because the vendor intentionally sold it that way. Again – it’s very difficult for a vendor to know what they are actually getting unless they are well connected. Thankfully, there are things vendors can do to account for this problem (assuming they are aware of it), and if they are vigilant, they can eliminate it completely. A lot can be accomplished by simply having supply chain checks in place to oversee production on the Indonesian side of things. However, there are also things that can be done here in the states to identify those type of problems. For example - it stands out like a sour thumb in consumer testing whenever a source tries to pass “blends” as unique “strains”. Because most people will score consecutive servings of the same batch lower, you can very reliably detect when people are testing batches that are similar in composition. It’s easy to set up a control group that only receives 1 random strain from a particular shipment, and then make sure a separate test group receives all 5 strains from that shipment. If the control group gives good scores, but the test group initially give high marks, and then it falls off – it’s a strong indication that the 5 “strains” from that one supplier is basically just 1 blend. This principal also has a variety of other applications – such as using to test if different pre/post-harvest techniques/manipulations developed a different/unique alkaloid profile, etc...
Perhaps the most interesting thing we have learned through consumer testing is something that absolutely blew my mind. I think a lot of people will have a hard time believing this fact – just a few years ago I would be in that boat. Consumer testing is very reliable when it comes to testing for the "overall quality" of a batch but that’s about it. In general, people tend to agree on what is poor, average, good, and great kratom, even though there are outliers. With a decent sample size of consumer testers to counter outlying opinions (at least 20 testers… preferably 40), you can statistically gauge the overall quality of a batch. It’s a FANTASTIC tool for weeding out batches, however, overall quality is about where the consensus will end because it's really kind of rare that you will get a consensus on where it sits on the "spectrum" or how long its "legs" are in blind testing (the blind part is important as humans we are very susceptible to the power of persuasion… more on that later). What's also interesting is even when gaining a strong consensus on quality, it’s nearly impossible to get a full consensus. We can be testing some of the best kratom on earth and there will always be a handful of folks who will score it poorly.
Personal chemistry results in some interesting conflicts for me as a vendor. I often get asked a lot of questions by customers that I cannot in good faith just give a simple answer to. Take for example the most common question I find in my inbox - "What are a few batches on the current menu that really stand out". I would say that might be a good question for a vendor who is known to have some “iffy” choices on the menu. Surely any vendor worth their weight in kratom should know which batches are performing well with consumers. If they are a reputable vendor, they should be able to give you a scoop on what’s “good”. (If they are not reputable, they might recommend batches that they are having a hard time selling just to get rid of). This question is a tough one for me – and how I usually respond is probably annoying to some people, lol. Here I am, literally with spreadsheets of data that detail each batches characteristics, like ratios, and overall quality scores and yet I can’t ever provide a concrete answer of “you will love X”. Because of the way we compile our menu, it's not as black and white as you would think. Once you weed out the poor and average batches, everything that qualifies for the menu ends up in a similarly narrow range of overall quality scores and acceptance ratio’s... it truly is difficult to point to a few batches as “standouts”. I certainly have my personal favorites, but I know that once a product is in that narrow upper range, personal chemistry is going to dictate to what degree you love the product. I still answer the question because I know its annoying for me to reply with “we don’t have any” - but who have asked will have probably noticed that I always preface my reply with “These are some of my personal favorites right now….”
Like any kratom enthusiast, I like to read reviews on the forum (especially when they are on our products). Something I never noticed as a consumer but I certainly do now is that with some of the more popular vendors, it appears that everyone has a different “favorite strain”. A good vendor will make sure that they have a quality lineup which will dramatically increase the odds that you will find favor with some of their offerings. Which strain becomes your favorite is likely a matter of your personal chemistry. Given that fact it's not surprising that so many different batches can be declared as "favorites".
The 2nd most common question I get is "What strains on the menu are more on the lower/upper end of the spectrum?" This is a question to which I am sure my answer frustrates people, lol. I know people just want me to point them to some menu items, but it's hard to do that in good conscious whenever I know there is hardly ever a consensus on spectrums. (I will usually point again to what I personally recommend). How people perceive a batch on the spectrum is wildly all over the place, it is exceedingly rare that we have a general consensus on spectrum scores.
To give some perspective on just how differently people perceive the effects of the same batch of kratom, I will share a few snippets of data with you. Below is some old survey data that I have handy because I have shared it tons of times in email as an example. I've' trimmed out much of the data for the sake of simplicity, but let's look at the first and last columns.
The first column is an "overall quality score" which is on a scale of 1-10 (1=poor, 5 = average/what most vendors offer, and 10 = some of the best kratom you have had this year). The batch shown below did qualify for the menu, you can see there is some general consensus on its overall quality score. However, you can notice that in true fashion, there are a good handful of folks that scored it very low. If this were not a "menu item" you could expect to see a general consensus of lower overall quality scores… you can really tell when a batch is not well received, lol.
The third column reflects where testers score the batch in regards to where it lands on the "the spectrum" (1=low, 2=high). I know most people would expect to see some commonality amongst scores here, but as you can see, everyone has a different take on it. Now you can add up all of the ratings and calculate the average (4.97ish) but that is a rather lazy and ineffective way of interpreting the data. An average of 4.97 sounds like the perfect definition of a "mid", but that doesn't really reflect the experiences of the testers (just as an overall quality score average only tells half the story). If you look at the ratio of how many people scored it below a 3 or above a 7, it paints a different picture. This trend repeats no matter how we sort/filter the data – even when trying to account to consumption habits, the spectrum results still vary the majority of the time.
Now this doesn’t mean that there is never a spectrum consensus of a batch. There are times where we see it, even in blind testing. In those cases we will be very intentional with the name we give the batch to ensure it’s a descriptor that fits it well (Lazy Sundae, Blast off Bali, Afterburner Bali, etc...).
Notice in the above paragraph I said “in blind testing”, with an emphasis on “blind”. This is because we do see spectrum consensus more often whenever the tester believes they know what they are testing. When they don’t know the “vein color” of a batch or better yet, if they are misled on the “vein color” of a test sample its extremely rare to have a spectrum consensus. This is due to what I touched on briefly before and that is sadly, humans are very susceptible to the power of persuasion and/or confirming their beliefs. If you are told that you are testing a “white”, you are much more likely to notice “white characteristics” than if you were given no color assignment at all. To illustrate my point, check out the snapshot of consumer data below regarding two batches – Green 389 and White 389.
As you can see there is a variety of responses as to how each person experienced each batch. Looking at the data for Green 389, it would be hard to recommend to a person where on the spectrum it sits. One conclusion you can draw from the data however is that on average, the White 389 scored significantly faster than the Green 389… you might say this is on the higher end of the spectrum. The only problem is that in reality, both batches are scores from the same exact batch. We just labeled some samples as “green” and others as “white” in testing. This experiment repeats itself over and over again and as you might predict the differences in scoring become even more pronounced when we blind test a “red label” vs a “white label”. This doesn’t just apply to product testing - the same thing occurs when somebody announces on a review forum that “x” batch is incredibly “XYZ”. Based on that review, people will order it and be expecting that same experience. They then leave a review and the cycle continues, lol. I’m no behavioral phycologist… I don’t really know what this reality says about us. Do we just program our brains in advance to experience what we expect to experience, or do we really just do a very poor job of observing how kratom effects our bodies? I have to admit all of this was pretty astonishing to me when I first began our research. As a long-time consumer with my own preconceived notions, it was a very hard pill to swallow. If the data weren’t so abundantly clear, I would have a hard time believing it myself. In fact, early on I did all kinds of experiments to try and shoot holes in the data I was seeing. (accounting for testers serving size, grams per day, consumption experience, etc). Truthfully though – now that I know it, there are signs all around that have always pointed to this conclusion. Even with the echo chamber on review sites and forums, If you really pay attention to how people describe their experiences of any given batch, you will start to notice just how diverse our opinions can be.
This of course doesn’t mean that peer revies are a waste of time or even unreliable…. Just don’t take them as gospel. There is some real value when it comes to considering the overall the perceived quality of a batch, but I would recommend blindly going big on a batch just because somebody describes it with the characteristics that you love (unless you are priming yourself for the placebo effect - I suppose there is nothing wrong with that).
Finally, I mentioned that there are some situations in which “strain names” are relevant. Some vendors take the liberty or renaming the products to match the customers’ expectations. If I receive something as a “White Thai” that is more like a Relaxing Red – the name is getting changed. Also, we will often give names that are actual descriptors of their origin or technique applied during post-harvest processing. For example – our “30 Year Wild” batches really do come from wild trees where the youngest among them is estimated to be around 30 years old, Wild Hulu Kapuas are from wild trees along the Hulu Kapuas river, and our Lazy Sundae may lead to extremely low productivity, lol. Even with vendors such as ourselves, I will be the first to tell you that you still should not go by the name alone… not a single one of our “Balis” actually come from Bali (and neither does anyone else’s unless they picked it there while on vacation).
