### AB Test

 Source khan academyWe have a website with white background. The mean amount of time people spend on it is μ = 20 minutesWe change the background to yellow and want to see if it improves user time on our siteNull hypothesis H0The change had no effect, there is no news hereThat would mean the mean is still 20 minutes even after the changeAlternative hypothesis Haμ > 20people actually spend more time on itWe set a threshold significance level α = 0.05Take sample of people visiting yellow website and calculate mean like sample mean, standard deviation, if the null hypothesis is true, what is the probability of getting a sample with statistics that we get?if that probability is lower than our significance level. if it is less than 0.05 (5%) then we reject the null hypothesis and say we have evidence for the alternative. However if the probability of getting samples are at significance level or higher, then we say hey we can't reject the null hypothesis and we aren't able to have evidence for the alternative.Step 3: take 100 samples from yellow page and calculate mean and sample standard deviation and calculate p-value (X_bar = 25, STD) Step 4: p-value: p(sample mean X_bar >= 25  |  H0 is true) Step 5: if p-value < α   => reject null hypothesis                if p-value >= α  we do not reject the null hypothesis.Hypothesis testing and p-values | Inferential statistics | Probability and Statistics | Khan AcademyA neurologist is testing the effects of a drug on rat response time, by injecting 100 rats with a unit dose of the drug. Neurologist knows thatmean response time of the rats not injected with drug is 1.2 seconds,mean response time of the rats injected with drug is 1.05 seconds, with a sample standard deviation of 0.5 seconds. Do you think that the drug has an effect on response time?H0: drug has no effect: μ is 1.2 even with drugHa: drug has an effect: μ is not 1.2 when drug is givenShould we accept alternative hypothesis or stick with null hypothesis?Approach:Let's assume null hypothesis is true, what is the prob we would have got these results with samples Chi-square goodness-of-fit tests  Khan academy: very good!https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2_contingency.htmlhttps://www.mathsisfun.com/data/chi-square-test.htmlX^2 = sigma (observed - expected)^2 / expecteddegree of freedom = number of options  - 1then look up table to see for which degree of freedom what is the prob of this particular X^2. That becomes your p-valuehttps://www.statisticshowto.datasciencecentral.com/probability-and-statistics/chi-square/The McNemar’s test operates upon a contingency table. of two classifiers working on same dataset. So you get two results per samplehttps://www.statisticshowto.datasciencecentral.com/mcnemar-test/On Comparing Classifiers: Pitfalls to Avoid and a Recommended ApproachStatistical Comparisons of Classifiers over Multiple Data SetsApproximate statistical tests for comparing supervised classification learning algorithmsPaired or unpaired statistical significancePython-based statistical hypothesis testingshttps://medium.com/@sasidhar.konda/ab-testing-in-python-1b5608207d86https://towardsdatascience.com/the-art-of-a-b-testing-5a10c9bb70a4https://www.facebook.com/business/help/927019534094639/?helpref=hc_fnav&bc[0]=AHCv1&bc[1]=Ads%20Help&bc[2]=Measure%20%26%20OptimizeAB test confidence onlinehttp://www.evanmiller.org/ab-testing/chi-squared.html#Google udacity AB testing https://www.udacity.com/course/ab-testing--ud257https://towardsdatascience.com/the-math-behind-a-b-testing-with-example-code-part-1-of-2-7be752e1d06f