TFH testing and naming

[gumbyke1]

Here is another idea for Chris and Angie to contemplate on the testing issue.

Would even help for trading post transactions.

What if we had a STICKY thread that cross referenced test batch numbers to new menu items and it would be the responsibility of the BUYER to check for duplicates.

Of course this would be Chris and Angie’s decision, but I thought I would throw it out there.....

Thanks for the suggestion K-Tom, you always have good stuff to share! Unfortunately, our obstacle won't be resolved with a sticky post because we will not be revealing the various testing IDs for each batch going forward. There are a variety of reasons for this which I have written about in various posts recently (I think the Red 707 post and Personal Chemistry post maybe?). I think because I write such long novel comments that there is way too much info to keep track of.

For those who don't have an hour to read through by books , Ill try to summerize that our wish is to get feedback specifically on the two options we presented in our post (ill quote-paste them at the end of this comment.). We obviously know there are some smart people here that might have a better idea, and if you do we would kindly ask if folks have other suggestions that you email to us. We just want to avoid the folks getting pumped around an idea that we cannot or will not implement because it doesn't fall within the parameters of our situation. Thanks!

An important thing to note that is easy to miss in my novel is that for a variety of reasons batches can have several different test IDs. Additionally, the "White 101" might not be the same "White 101" that others receive. For a variety of reasons that I have given insight into in other posts, we will be politely declining to share testing ID information going forward. This is where the whole problem originates - members will not know the testing ID's of menu items going forward - should we allow them to be purchased with that understanding, or do we just decline requests to do so?

I'm seeing a lot of public support for option 1 so far, and honestly, that is the easiest solution for us. However, even more, people are emailing us privately requesting #2. I think they are choosing to share thier voice on email instead of the forum because they don't want to be seen as "greedy" in the public forum, lol I tell them it's quite alright, as a former collector myself I would have voted number #2. Seriously, real connoisseurs of the leaf would not let a fantastic batch slip through their hands. The biggest thing that is speaking to me about option #2, is we do get a lot of order requests for items that never make it to the menu. Thanks to personal chemistry, we see batches all the time that may be phenomenal to some folks but not to the masses (since our game is about giving members the best overall odds of finding a batch that works well - those batches get weeded out). Option #2 at least allows folks to snag some powder of something they will basically never see again. Again the collector in me really appreciates that aspect.

Thanks again Tom!

Anyways, I thought I would ask the community real quick which of the below options would be preferred:

1. We do not allow the purchase of test samples, any requests to do so are politely declined.
2. We allow the purchase of test samples, but that comes with the understanding that the item you are purchasing might appear on the menu one day. When asked, we will politely decline requests to "confirm" what sample IDs it was tested as.

If anyone else has other suggestions that you want to share, we would love to hear them! Keep in mind we are looking for something low-tech, easy to implement, and does not compromise our testing methods (revealing the various testing IDs for each batch in testing). If you have an idea, we would just kindly ask that you email us directly and not post your suggestions here. We know there are probably some other alternatives and we want to hear about them, however, not in a public forum, please. Thanks!

[IndelibleDotInk]

Customer service to the max!

[Lokey]

Confused, are you saying one tester's red 707 could be different than another's?

[IndelibleDotInk]

Confused, are you saying one tester's red 707 could be different than another's?

Yes, and to further keeping the data fresh, Chris and Angie don't know either until they get to the ending phases of testing (if they are using that method.)

[IndelibleDotInk]

aw man, now I must place an order of kratom which is statistically analyzed to ensure people that liked it liked the majority of batches I like.

[Lokey]

Confused, are you saying one tester's red 707 could be different than another's?

Yes, and to further keeping the data fresh, Chris and Angie don't know either until they get to the ending phases of testing.

So the red 707 that Kelley loved could be different than the red 707 I tested and that others purchased based on his review?

[IndelibleDotInk]

if that's the way they are running the test, i don't know. It should be cause it's simple to implement it -It's called a double blind and can partially eliminate people-error.

I am so nerding out, lol.

[gumbyke1]

Confused, are you saying one tester's red 707 could be different than another's?

It could be... any batch could be. However, I will throw everyone a bone and confirm that we did not need to do that with Red 707. The Red 707 that everyone got is exactly the same... no trickery there. I have like 3 different posts where I am commenting about this stuff, I should put it all together to give anyone who is reading this post the proper context. Im going to paste them below.

[gumbyke1]

Im just pasting a post I made on another thread in the reviews section (Red 707) because it gives more context to what we are discussing on this thread. See below:





707 was the same for everyone, and we pretty much got all enough survey data to make a decision before anyone even started talking about it on here. It's officially going to make the menu at some time in the future, but we havent even began to consider when that might be. However, I have been getting a lot of inquiries about the complexity of a testing process, so Ill give a sneak peek of 707 since we are on the subject Note that this is just part of the survey data, I'm not giving the full picture here to try and keep it breif but I think its enough to illustrate what I'm about to break down.



Just looking at the survey data of this batch, you can see that its average overall quality score is 8.22 (on a scale of 1-10). When we first began our research, the average score was basically all we looked at. I mean it makes sense, right? However, just looking at the average score is actually a lazy and ineffective way to use this data to evaluate overall quality - there is a lot more that must be considered. There are even other steps that come between collection and analysis that should be considered such as “survey cleaning”, but that’s a deeper rabbit hole that I wont go down here for the sake of brevity (who am I kidding in thinking this might be brief, lol). When reading survey data, if the details are ignored or the data is read poorly, it can really limit our ability to capture valuable insight and it dramatically reduces the credibility of our findings. Over the years we have become much more refined in how we collect and interpret data, I’ll give just a few high-level examples of some other ways to view the data.

Now back to the overall quality score of 8.22. On the surface that doesn't sound very impressive. It’s not bad, but its not great either. However, you have to first consider what the scale of 1-10 represents.

1 = Poor Quality
5 = Average – comparable to what you will find from most online vendors
10 = Exceptional - some of the best leaf currently available.

Following that scale, if a "5" is average, then "8.22" is more than above average, it's actually creeping into the range of really good. However, that's just part of the story that is being told here.
Our experience has taught us that you will hardly ever find a batch that is a home run for every person. Thanks to our "friend" personal chemistry, its exceedingly rare that we find a batch where 100% or even 95% of testers score it above a 7 or 8.... it just doesn't happen very often. Since our goal is to construct a menu where the consumer will on have a higher probability of finding batches that work well for them, it's really important for us to take into consideration the overall approval ratio of each batch.

A score of “7” is generally defined as “above average” and most consumers give “menu approval” to items they score 7 or higher. Out of 32 surveys, only 3 people scored it below a "7" with the lowest score being a "5" (6, 6, 5). Accepting that standard, this batch has an approval ratio of 29/32, or roughly a 91% favorability rate.

I know that 91% doesn’t sound like anything special if this were ratings on Amazon…it would basically equate to a 4.5 of 5 stars. Whatever the product was, that would be good enough for me to feel comfortable pulling the trigger, however, it wouldn’t give me the impression that this thing is particularly awesome. However, in the world of kratom (or “kratom roulette”), having 9 out of 10 people consider the batch as “above average” it’s actually pretty good. Back in my days as a consumer, if I were told I have a 90% chance of being pleased with a batch it would absolutely make my list of batches to order.

However, the story is still not complete. Let’s take a look at how the scores of those who approved are broken down:
10 = 7 respondents
9 = 7 respondents
8 = 8 respondents
7 = 7 respondents
We can see here that of those who liked this batch, roughly 50% of surveyors found it to be one of the best batches available or very close to it (9 or 10). The other 50% found it to be above average/good (7 or 8). So if we break down the probabilities that a person will like it, we have the following:

0% will find it as poor quality
9% will find it as average
46% will find it as good/ very good
44% will find it as the best you can get or very close to it

Those odds are pretty promising when you break it down like that. As a consumer, when its broken down this way it is much more appealing than if I saw it had an average score of 8.2. We are in the business of providing what the consumer wants, so its really important that we look at things this way.

But that’s not even where the analysis ends. There are other ways to analyze the data such as seeking out a central tendency by removing the highest and lowest 10% of survey scores (this can help offset some of the one-offs that might exist because of a tester having an "off" day). There are tons of different techniques to apply here, and they can all help give you a better picture.

Also, remember that I am only sharing part of the data here. There are other things I can consider here such as providing different “weights” to each individual surveyor's score. Perhaps I want to decrease the weight of surveys from people who have been consuming kratom for less than one year, and give more weight to surveyors with 5+ years of experience. I can also assign weight by a surveyor's average score or even cross-reference it with how a surveyor has scored other high-quality batches in the past. I could also look at the surveyor's other history to remove “straight line” answers. I could also begin breaking down correlations with the preference profiles of individual surveyors. I'll stop with the possibilities, but as you can imagine, sometimes it pays to get into the nitty-gritty especially if we are having a hard time making a decision.

Another thing to consider is the possibility of just having “bad data”. I always tell surveyors that I would rather have no data than bad data, but it still happens. For example:

- There are people who might have submitted 7 surveys at once and they are just assigning scores to each sample by memory (this usually does not provide good data).
- Sometimes scoring from one tester may appear to be random or contrary (such as straight-line answers, or answers that with each sample contradict the crowd). This could indicate that the surveyor was just submitting them to check it off as complete, or perhaps they are fighting a virus or something and just can't get a good read on things.

These are just a couple of things that should be taken into consideration, especially if you don’t have a very big pool of data to play with. Occasionally we might see a great divide between the people who absolutely love a batch and those who score it a 1-3. I'm looking at a real-life example right now where we only have 20 surveys in and 15 people (75%) rated it between “7-10” and 5 people (25%) rated it a “3” or below. The average turns out to be 6.65, which right away looks like it’s not menu worthy. However, with a good chunk of people loving it and such a small pool of data, it's hard not to wonder if maybe 1 or 2 of those who scored it low were just not primed for kratom that day and no matter what they tested, it was going to be a “1”. If we were to just turn 2 of the low scores into an “8”, well suddenly the overall quality score is a “7.25”… that getting in the territory that might need to expand testing or look at the data in a different way (especially since some people really seemed to like it).

In the scenario above, we might decide to simply expand the testing pool or maybe target those testers who scored low with the same sample in the future, but under a different tester ID. A lot of you have noticed that sometimes it takes a long time for a batch to appear on the menu, this can sometimes be the reason (there are other reasons too that have to do with a planned release schedule to ensure our batches are consistent year-round).

Im finding myself getting into territory where I could talk forever here, so I am just going to wrap up. It wasn’t my intention to get this deep, but I think there are likely one or two fellow nerds here who might appreciate it.

[Lokey]

Thanks again Chris. The only explanation I can think of 707 not working for me like others is the physical and mental stress I was feeling that particular day or just a fluke thing but wanted to make sure it was the same one.

[gumbyke1]

Thanks again Chris. The only explanation I can think of 707 not working for me like others is the physical and mental stress I was feeling that particular day or just a fluke thing but wanted to make sure it was the same one.

That's entirely possible, there are times when no matter what you take, it's not going to work for you. However, its also entirely possible that your chemistry just didnt agree with it. I recently just reposted a comment I made on another thread on personal chemistry that dives into the fact a little further. Ive been recycling posts a lot lately, lol.... Its because Ive been writing on 3 seperate posts about a subject that all ties together. I'll post the link that talks a little bit about personal chemistry below.

viewtopic.php?f=7&t=10660

I think whats good to note is that although roughly 50% of people found 707 to be amazing, there are also 50% of people who found it to be just "pretty good". With that, if you order it expecting it to blow your socks off, there is only a 50% chance of that...about half of you will just find it as a nice solid batch (except for the couple who wont like it at all). Trust me, its baffling as all can be. A long time ago I would try a batch first before even sending it through consumer testing, kind of acting as the "first filter" to determine whether a batch is worthy of the expense of consumer testing (the consumer testing costs really add up...seriously!). However, I quickly learned that being a "first filter" was a fool's errand and since then Ive seen plenty of batches thrive that I thought for sure would be a dud.

Consumer testing is a wonderful tool for knowing whether most people find a batch's quality as poor, average, good, or great. However, its usuelly much less useful to determine where a batch is on the spectrum (or really any other characteristics for that matter). HOWEVER, I should say that occasionally we do see somewhat of a general consensus on surveys, but its hardly ever a slam dunk where 90% of folks are on the same page. When surveyers do scores characteristics similarly, we will usually give the batch a name that speaks to that (Afterburner Bali, Lazy Sundae, etc). As a matter of fact, will be soon adding to the menu a new 30-Year Wild batch called "Red Hangat Hulu". The word "Hangat" means "warm" in Indonesian. This batch is a slower cousin to the 30-Year Wild Jade Hulu (same harvest but a different cure). At any rate, the batch got this "Hangat" designation because the actual word "warm" kept appearing in the comments of several of our consumer surveys - it was at least enough times for us to take notice and say it wasn't a coincidence.

[Arob1000]

As a matter of fact, will be soon adding to the menu a new 30-Year Wild batch called "Red Hangat Hulu". The word "Hangat" means "warm" in Indonesian. This batch is a slower cousin to the 30-Year Wild Jade Hulu (same harvest but a different cure). At any rate, the batch got this "Hangat" designation because the actual word "warm" kept appearing in the comments of several of our consumer surveys - it was at least enough times for us to take notice and say it wasn't a coincidence.

Already getting FOMO for this!

[Kelleytoons]

Already getting FOMO for this!

LOL - I was thinking EXACTLY the same thing (for me it was the "red" - greens are always kind of meh to me, but reds? Can't stay away from any, particularly from TFH).

[Roofdog]

If its tied in any way to the 30yr hulu im in. That is a great batch. Im putting a 250g limit per batch on myself now though. I am running out of storage.

[uspatriot995]

I like an option #3:

You may place an order of a maximum of 1 (one) of your test samples, there are two conditions. a) Test sample orders are limited to 250gm and one strain only; and b) You agree and understand that in doing so you may unknowingly purchase the same strain from the menu at some point in the future if the strain makes the menu--which, wouldn't be a bad thing because you thought enough of it to purchase some of the sample anyway.

This does several things: Eliminates the possibility that you tested a sample that was a "10" in your book, but it never made the menu, so you'll never have a chance to have any more. The people who do order test samples are able to enjoy a nice 250gm bag of their favorite tester and are informed it may end up and be a duplicate of something they order later. Since it's limited to one tester bag per person, it will lighten the "load" of having to fill a large order of a tester, and keeps everything fair and even. And as stated, they liked it enough to buy a 250 of it in testing, so they should be delighted to get more later.

[Kelleytoons]

If its tied in any way to the 30yr hulu im in. That is a great batch. Im putting a 250g limit per batch on myself now though. I am running out of storage.

LOL - any limits I once had are blown WAY out of the water (I JUST put an order - my last for this year - and missed out on this wonder. So now... SECOND last of this year. Truly. I promise).