← Back
Editing: _page_trend_test.cpython-311.pyc
� d�cbJ � � � d dl mZ d dlZd dlZddlmZ d dlZd dl m Z e dd� � Z dd �Z d � Z d � Zd � Zd� Z G d� d� � Z e� � ZdS )� )� permutationsN� )�norm)�make_dataclass�PageTrendTestResult�� statistic�pvalue�methodF�autoc �d � t t dd�}||vr t dt |� � � �� � �t j | d�� � }|j dk rt d� � �|j \ }}|dk s|dk rt d � � �t j t j | � � � � rt d � � �|rK|� � � d k r#|� � � |j d k st d � � �n!t j � | d �� � }|�t j d |d z � � }nxt j |d�� � }|j d k sDt |� � t t! d |d z � � � � k st# |� � |k rt d|� d�� � �t% |� � t& urt) d� � �t+ ||� � }|dk rt- |� � }|| } | |||� � } t/ || |�� � } | S )a_0 Perform Page's Test, a measure of trend in observations between treatments. Page's Test (also known as Page's :math:`L` test) is useful when: * there are :math:`n \geq 3` treatments, * :math:`m \geq 2` subjects are observed for each treatment, and * the observations are hypothesized to have a particular order. Specifically, the test considers the null hypothesis that .. math:: m_1 = m_2 = m_3 \cdots = m_n, where :math:`m_j` is the mean of the observed quantity under treatment :math:`j`, against the alternative hypothesis that .. math:: m_1 \leq m_2 \leq m_3 \leq \cdots \leq m_n, where at least one inequality is strict. As noted by [4]_, Page's :math:`L` test has greater statistical power than the Friedman test against the alternative that there is a difference in trend, as Friedman's test only considers a difference in the means of the observations without considering their order. Whereas Spearman :math:`\rho` considers the correlation between the ranked observations of two variables (e.g. the airspeed velocity of a swallow vs. the weight of the coconut it carries), Page's :math:`L` is concerned with a trend in an observation (e.g. the airspeed velocity of a swallow) across several distinct treatments (e.g. carrying each of five coconuts of different weight) even as the observation is repeated with multiple subjects (e.g. one European swallow and one African swallow). Parameters ---------- data : array-like A :math:`m \times n` array; the element in row :math:`i` and column :math:`j` is the observation corresponding with subject :math:`i` and treatment :math:`j`. By default, the columns are assumed to be arranged in order of increasing predicted mean. ranked : boolean, optional By default, `data` is assumed to be observations rather than ranks; it will be ranked with `scipy.stats.rankdata` along ``axis=1``. If `data` is provided in the form of ranks, pass argument ``True``. predicted_ranks : array-like, optional The predicted ranks of the column means. If not specified, the columns are assumed to be arranged in order of increasing predicted mean, so the default `predicted_ranks` are :math:`[1, 2, \dots, n-1, n]`. method : {'auto', 'asymptotic', 'exact'}, optional Selects the method used to calculate the *p*-value. The following options are available. * 'auto': selects between 'exact' and 'asymptotic' to achieve reasonably accurate results in reasonable time (default) * 'asymptotic': compares the standardized test statistic against the normal distribution * 'exact': computes the exact *p*-value by comparing the observed :math:`L` statistic against those realized by all possible permutations of ranks (under the null hypothesis that each permutation is equally likely) Returns ------- res : PageTrendTestResult An object containing attributes: statistic : float Page's :math:`L` test statistic. pvalue : float The associated *p*-value method : {'asymptotic', 'exact'} The method used to compute the *p*-value See Also -------- rankdata, friedmanchisquare, spearmanr Notes ----- As noted in [1]_, "the :math:`n` 'treatments' could just as well represent :math:`n` objects or events or performances or persons or trials ranked." Similarly, the :math:`m` 'subjects' could equally stand for :math:`m` "groupings by ability or some other control variable, or judges doing the ranking, or random replications of some other sort." The procedure for calculating the :math:`L` statistic, adapted from [1]_, is: 1. "Predetermine with careful logic the appropriate hypotheses concerning the predicted ording of the experimental results. If no reasonable basis for ordering any treatments is known, the :math:`L` test is not appropriate." 2. "As in other experiments, determine at what level of confidence you will reject the null hypothesis that there is no agreement of experimental results with the monotonic hypothesis." 3. "Cast the experimental material into a two-way table of :math:`n` columns (treatments, objects ranked, conditions) and :math:`m` rows (subjects, replication groups, levels of control variables)." 4. "When experimental observations are recorded, rank them across each row", e.g. ``ranks = scipy.stats.rankdata(data, axis=1)``. 5. "Add the ranks in each column", e.g. ``colsums = np.sum(ranks, axis=0)``. 6. "Multiply each sum of ranks by the predicted rank for that same column", e.g. ``products = predicted_ranks * colsums``. 7. "Sum all such products", e.g. ``L = products.sum()``. [1]_ continues by suggesting use of the standardized statistic .. math:: \chi_L^2 = \frac{\left[12L-3mn(n+1)^2\right]^2}{mn^2(n^2-1)(n+1)} "which is distributed approximately as chi-square with 1 degree of freedom. The ordinary use of :math:`\chi^2` tables would be equivalent to a two-sided test of agreement. If a one-sided test is desired, *as will almost always be the case*, the probability discovered in the chi-square table should be *halved*." However, this standardized statistic does not distinguish between the observed values being well correlated with the predicted ranks and being _anti_-correlated with the predicted ranks. Instead, we follow [2]_ and calculate the standardized statistic .. math:: \Lambda = \frac{L - E_0}{\sqrt{V_0}}, where :math:`E_0 = \frac{1}{4} mn(n+1)^2` and :math:`V_0 = \frac{1}{144} mn^2(n+1)(n^2-1)`, "which is asymptotically normal under the null hypothesis". The *p*-value for ``method='exact'`` is generated by comparing the observed value of :math:`L` against the :math:`L` values generated for all :math:`(n!)^m` possible permutations of ranks. The calculation is performed using the recursive method of [5]. The *p*-values are not adjusted for the possibility of ties. When ties are present, the reported ``'exact'`` *p*-values may be somewhat larger (i.e. more conservative) than the true *p*-value [2]_. The ``'asymptotic'``` *p*-values, however, tend to be smaller (i.e. less conservative) than the ``'exact'`` *p*-values. References ---------- .. [1] Ellis Batten Page, "Ordered hypotheses for multiple treatments: a significant test for linear ranks", *Journal of the American Statistical Association* 58(301), p. 216--230, 1963. .. [2] Markus Neuhauser, *Nonparametric Statistical Test: A computational approach*, CRC Press, p. 150--152, 2012. .. [3] Statext LLC, "Page's L Trend Test - Easy Statistics", *Statext - Statistics Study*, https://www.statext.com/practice/PageTrendTest03.php, Accessed July 12, 2020. .. [4] "Page's Trend Test", *Wikipedia*, WikimediaFoundation, https://en.wikipedia.org/wiki/Page%27s_trend_test, Accessed July 12, 2020. .. [5] Robert E. Odeh, "The exact distribution of Page's L-statistic in the two-way layout", *Communications in Statistics - Simulation and Computation*, 6(1), p. 49--61, 1977. Examples -------- We use the example from [3]_: 10 students are asked to rate three teaching methods - tutorial, lecture, and seminar - on a scale of 1-5, with 1 being the lowest and 5 being the highest. We have decided that a confidence level of 99% is required to reject the null hypothesis in favor of our alternative: that the seminar will have the highest ratings and the tutorial will have the lowest. Initially, the data have been tabulated with each row representing an individual student's ratings of the three methods in the following order: tutorial, lecture, seminar. >>> table = [[3, 4, 3], ... [2, 2, 4], ... [3, 3, 5], ... [1, 3, 2], ... [2, 3, 2], ... [2, 4, 5], ... [1, 2, 4], ... [3, 4, 4], ... [2, 4, 5], ... [1, 3, 4]] Because the tutorial is hypothesized to have the lowest ratings, the column corresponding with tutorial rankings should be first; the seminar is hypothesized to have the highest ratings, so its column should be last. Since the columns are already arranged in this order of increasing predicted mean, we can pass the table directly into `page_trend_test`. >>> from scipy.stats import page_trend_test >>> res = page_trend_test(table) >>> res PageTrendTestResult(statistic=133.5, pvalue=0.0018191161948127822, method='exact') This *p*-value indicates that there is a 0.1819% chance that the :math:`L` statistic would reach such an extreme value under the null hypothesis. Because 0.1819% is less than 1%, we have evidence to reject the null hypothesis in favor of our alternative at a 99% confidence level. The value of the :math:`L` statistic is 133.5. To check this manually, we rank the data such that high scores correspond with high ranks, settling ties with an average rank: >>> from scipy.stats import rankdata >>> ranks = rankdata(table, axis=1) >>> ranks array([[1.5, 3. , 1.5], [1.5, 1.5, 3. ], [1.5, 1.5, 3. ], [1. , 3. , 2. ], [1.5, 3. , 1.5], [1. , 2. , 3. ], [1. , 2. , 3. ], [1. , 2.5, 2.5], [1. , 2. , 3. ], [1. , 2. , 3. ]]) We add the ranks within each column, multiply the sums by the predicted ranks, and sum the products. >>> import numpy as np >>> m, n = ranks.shape >>> predicted_ranks = np.arange(1, n+1) >>> L = (predicted_ranks * np.sum(ranks, axis=0)).sum() >>> res.statistic == L True As presented in [3]_, the asymptotic approximation of the *p*-value is the survival function of the normal distribution evaluated at the standardized test statistic: >>> from scipy.stats import norm >>> E0 = (m*n*(n+1)**2)/4 >>> V0 = (m*n**2*(n+1)*(n**2-1))/144 >>> Lambda = (L-E0)/np.sqrt(V0) >>> p = norm.sf(Lambda) >>> p 0.0012693433690751756 This does not precisely match the *p*-value reported by `page_trend_test` above. The asymptotic distribution is not very accurate, nor conservative, for :math:`m \leq 12` and :math:`n \leq 8`, so `page_trend_test` chose to use ``method='exact'`` based on the dimensions of the table and the recommendations in Page's original paper [1]_. To override `page_trend_test`'s choice, provide the `method` argument. >>> res = page_trend_test(table, method="asymptotic") >>> res PageTrendTestResult(statistic=133.5, pvalue=0.0012693433690751756, method='asymptotic') If the data are already ranked, we can pass in the ``ranks`` instead of the ``table`` to save computation time. >>> res = page_trend_test(ranks, # ranks of data ... ranked=True, # data is already ranked ... ) >>> res PageTrendTestResult(statistic=133.5, pvalue=0.0018191161948127822, method='exact') Suppose the raw data had been tabulated in an order different from the order of predicted means, say lecture, seminar, tutorial. >>> table = np.asarray(table)[:, [1, 2, 0]] Since the arrangement of this table is not consistent with the assumed ordering, we can either rearrange the table or provide the `predicted_ranks`. Remembering that the lecture is predicted to have the middle rank, the seminar the highest, and tutorial the lowest, we pass: >>> res = page_trend_test(table, # data as originally tabulated ... predicted_ranks=[2, 3, 1], # our predicted order ... ) >>> res PageTrendTestResult(statistic=133.5, pvalue=0.0018191161948127822, method='exact') N)� asymptotic�exactr z`method` must be in F)�copy� z`data` must be a 2d array.� zVPage's L is only appropriate for data with two or more rows and three or more columns.z9`data` contains NaNs, which cannot be ranked meaningfullyr zD`data` is not properly ranked. Rank the data or pass `ranked=False`.�����axisz6`predicted_ranks` must include each integer from 1 to z0 (the number of columns in `data`) exactly once.z`ranked` must be boolean.r r )�_l_p_asymptotic� _l_p_exact� ValueError�set�np�array�ndim�shape�any�isnan�min�max�scipy�stats�rankdata�arange�range�len�type�bool� TypeError� _l_vectorized�_choose_methodr ) �data�ranked�predicted_ranksr �methods�ranks�m�n�L�p_fun�p� page_results �>/usr/lib/python3/dist-packages/scipy/stats/_page_trend_test.py�page_trend_testr9 sq � �L -�"�� � �G� �W� � ��>��G� � �>�>�?�?�?� �H�T�� &� &� &�E� �z�Q����5�6�6�6� �;�D�A�q��1�u�u��A���� C� D� D� D� �v�b�h�t�n�n� � � )�� (� )� )� )� � 4� � � � � �q� � �U�Y�Y�[�[�E�K��N�%B�%B� � 4� 5� 5� 5� &C� � �$�$�T��$�3�3�� � � �)�A�q��s�+�+��� �(�?��?�?�?�� � �1� $� $��_�%�%��U�1�a��c�]�]�);�);�;�;��_�%�%��*�*� � 6�*+� 6� 6� 6� 7� 7� 7� �F�|�|�4� � ��3�4�4�4� �e�_�-�-�A� ���� ��&�&�� �F�O�E� ��a��A���A�%��!�F�K�K�K�K� �� c �T � | j \ }}|dk s|dk r|dk s|dk rd}nd}|S )z1Choose method for computing p-value automatically� � r � r r )r )r1 r2 r3 r s r8 r, r, l s? � � �;�D�A�q��1�u�u��R���A��E�E�a�"�f�f� ����� �Mr: c � � | � dd�� � }||z }|� d�� � }|j dk r|d n|� � � }|S )z:Calculate's Page's L statistic for each page of a 3d array���T)r �keepdimsr r r r )�sum�size�ravel)r1 r/ �colsums�products�Lss r8 r+ r+ v sX � ��i�i�R�$�i�/�/�G� ��(�H� � � �2� � � �B��'�Q�,�,��A���B�H�H�J�J�B� �Ir: c � � ||z |dz dz z dz }||dz z |dz z |dz dz z dz }| |z t j |� � z }t j |� � }|S )zBCalculate the p-value of Page's L from the asymptotic distributionr r � � )r �sqrtr �sf)r4 r2 r3 �E0�V0�Lambdar6 s r8 r r so � � �A�#�q��s�Q�h�,�� �B� �A�q�D�&�!�A�#�,��1��Q�� �� $�B���d�B�G�B�K�K� �F� �����A� �Hr: c �� � t | � � t |� � t |� � }}} t � |� � t � | |� � S )z)Calculate the p-value of Page's L exactly)�int� _pagel_state�set_krL )r4 r2 r3 �ks r8 r r � sK � � �!�f�f�c�!�f�f�c�!�f�f�!�q�A����q���� �?�?�1�a� � � r: c �0 � e Zd ZdZd� Zd� Zd� Zd� Zd� ZdS )�_PageLz4Maintains state between `page_trend_test` executionsc � � i | _ dS )zLightweight initializationN)�all_pmfs)�selfs r8 �__init__z_PageL.__init__� s � ��� � � r: c �t � || _ ||dz z |dz z dz ||dz z d|z dz z dz c| _ | _ dS )z4Calculate lower and upper limits of L for single rowr r � N)rT �a�b)rY rT s r8 rS z _PageL.set_k� sJ � �����Q�q�S�'�1�Q�3�-�!�+�a��1��g�q��s�1�u�o��-A�������r: c �~ � �� �� fd�t |�� j z dz � � D � � }t j |� � S )z'Survival function of Page's L statisticc �< �� g | ]}�� |�� � ��S � )�pmf)�.0�lr3 rY s ��r8 � <listcomp>z _PageL.sf.<locals>.<listcomp>� s% �� � =� =� =��d�h�h�q�!�n�n� =� =� =r: r )r& r^ r rB )rY rd r3 �pss ` ` r8 rL z _PageL.sf� sB ��� � =� =� =� =� =�e�A�q���x�!�|�&<�&<� =� =� =���v�b�z�z�r: c �t � t d| j dz � � }t j t t |� � � � � � }||z � d�� � }t j |t j | j dz | j dz � � � � d }|t j | j � � z S )z@Relative frequency of each L value over all possible single rowsr r g �?g �?r ) r& rT r r �listr rB � histogramr% r] r^ �math� factorial)rY r1 � rank_permsrG �countss r8 �p_l_k_1z_PageL.p_l_k_1� s� � � �a�����"�"���X�d�<��#6�#6� 7� 7�8�8� ��J� � #� #�� #� +� +��� �b�"�)�D�F�3�J���s� �"C�"C�D�D�Q�G���d�n�T�V�,�,�,�,r: c �X � || j vr i | j |< | j | j | vri | j | | j < || j | | j v r | j | | j | S |dk r|| � � � }t | j | j dz � � }d� t ||� � D � � | j | | j < | j | | j | S d}t ||dz | j z z | j � � }t ||dz | j z z | j � � }t ||dz � � D ]<}| � ||z |dz � � } | � |d� � } || | z z }�=|| j | | j |< |S )z=Recursive function to evaluate p(l, k, n); see [5] Equation 1r c � � i | ]\ }}||�� S ra ra )rc rd r6 s r8 � <dictcomp>z _PageL.pmf.<locals>.<dictcomp>� s � �'E�'E�'E���A��1�'E�'E�'Er: r ) rX rT rn r& r] r^ �zipr! r rb ) rY rd r3 rf �lsr6 �low�high�t�p1�p2s r8 rb z _PageL.pmf� s� � � �D�M� !� !� !�D�M�!� � �6�� �q�)� )� )�')�D�M�!� �T�V� $� �� �a� ���(� (� (��=��#�D�F�+�A�.� .� ��6�6�� � � � �B��t�v�t�v�a�x�(�(�B�'E�'E��R����'E�'E�'E�D�M�!� �T�V� $��=��#�D�F�+�A�.� .� ���!�Q�q�S�$�&�L�.�$�&�)�)���1�a��c�4�6�\�>�4�6�*�*�� �s�D��F�#�#� � �A����!�A�#�q��s�#�#�B����!�Q� � �B� ��B��J�A�A�&'�� �a���� ��#��r: N) �__name__� __module__� __qualname__�__doc__rZ rS rL rn rb ra r: r8 rV rV � sh � � � � � �>�>�� � �B� B� B� � � � -� -� -� � � � � r: rV )FNr )� itertoolsr �numpyr rj �_continuous_distnsr � scipy.statsr" � dataclassesr r r9 r, r+ r r rV rR ra r: r8 �<module>r� s �� "� "� "� "� "� "� � � � � � � � � $� $� $� $� $� $� � � � � &� &� &� &� &� &� %�n�%:�%F�H� H� �\� \� \� \�~ � � �� � � � � �"!� !� !�?� ?� ?� ?� ?� ?� ?� ?�F �v�x�x� � � r:
Save File
Cancel