• 14.5 2012联邦选举委员会数据库
    • 根据职业和雇主统计赞助信息
    • 对出资额分组
    • 根据州统计赞助信息

    14.5 2012联邦选举委员会数据库

    美国联邦选举委员会发布了有关政治竞选赞助方面的数据。其中包括赞助者的姓名、职业、雇主、地址以及出资额等信息。我们对2012年美国总统大选的数据集比较感兴趣(http://www.fec.gov/disclosurep/PDownload.do)。我在2012年6月下载的数据集是一个150MB的CSV文件(P00000001-ALL.csv),我们先用pandas.read_csv将其加载进来:

    1. In [184]: fec = pd.read_csv('datasets/fec/P00000001-ALL.csv')
    2. In [185]: fec.info()
    3. <class 'pandas.core.frame.DataFrame'>
    4. RangeIndex: 1001731 entries, 0 to 1001730
    5. Data columns (total 16 columns):
    6. cmte_id 1001731 non-null object
    7. cand_id 1001731 non-null object
    8. cand_nm 1001731 non-null object
    9. contbr_nm 1001731 non-null object
    10. contbr_city 1001712 non-null object
    11. contbr_st 1001727 non-null object
    12. contbr_zip 1001620 non-null object
    13. contbr_employer 988002 non-null object
    14. contbr_occupation 993301 non-null object
    15. contb_receipt_amt 1001731 non-null float64
    16. contb_receipt_dt 1001731 non-null object
    17. receipt_desc 14166 non-null object
    18. memo_cd 92482 non-null object
    19. memo_text 97770 non-null object
    20. form_tp 1001731 non-null object
    21. file_num 1001731 non-null int64
    22. dtypes: float64(1), int64(1), object(14)
    23. memory usage: 122.3+ MB

    该DataFrame中的记录如下所示:

    1. In [186]: fec.iloc[123456]
    2. Out[186]:
    3. cmte_id C00431445
    4. cand_id P80003338
    5. cand_nm Obama, Barack
    6. contbr_nm ELLMAN, IRA
    7. contbr_city TEMPE
    8. ...
    9. receipt_desc NaN
    10. memo_cd NaN
    11. memo_text NaN
    12. form_tp SA17A
    13. file_num 772372
    14. Name: 123456, Length: 16, dtype: object

    你可能已经想出了许多办法从这些竞选赞助数据中抽取有关赞助人和赞助模式的统计信息。我将在接下来的内容中介绍几种不同的分析工作(运用到目前为止已经学到的方法)。

    不难看出,该数据中没有党派信息,因此最好把它加进去。通过unique,你可以获取全部的候选人名单:

    1. In [187]: unique_cands = fec.cand_nm.unique()
    2. In [188]: unique_cands
    3. Out[188]:
    4. array(['Bachmann, Michelle', 'Romney, Mitt', 'Obama, Barack',
    5. "Roemer, Charles E. 'Buddy' III", 'Pawlenty, Timothy',
    6. 'Johnson, Gary Earl', 'Paul, Ron', 'Santorum, Rick', 'Cain, Herman',
    7. 'Gingrich, Newt', 'McCotter, Thaddeus G', 'Huntsman, Jon',
    8. 'Perry, Rick'], dtype=object)
    9. In [189]: unique_cands[2]
    10. Out[189]: 'Obama, Barack'

    指明党派信息的方法之一是使用字典:

    1. parties = {'Bachmann, Michelle': 'Republican',
    2. 'Cain, Herman': 'Republican',
    3. 'Gingrich, Newt': 'Republican',
    4. 'Huntsman, Jon': 'Republican',
    5. 'Johnson, Gary Earl': 'Republican',
    6. 'McCotter, Thaddeus G': 'Republican',
    7. 'Obama, Barack': 'Democrat',
    8. 'Paul, Ron': 'Republican',
    9. 'Pawlenty, Timothy': 'Republican',
    10. 'Perry, Rick': 'Republican',
    11. "Roemer, Charles E. 'Buddy' III": 'Republican',
    12. 'Romney, Mitt': 'Republican',
    13. 'Santorum, Rick': 'Republican'}

    现在,通过这个映射以及Series对象的map方法,你可以根据候选人姓名得到一组党派信息:

    1. In [191]: fec.cand_nm[123456:123461]
    2. Out[191]:
    3. 123456 Obama, Barack
    4. 123457 Obama, Barack
    5. 123458 Obama, Barack
    6. 123459 Obama, Barack
    7. 123460 Obama, Barack
    8. Name: cand_nm, dtype: object
    9. In [192]: fec.cand_nm[123456:123461].map(parties)
    10. Out[192]:
    11. 123456 Democrat
    12. 123457 Democrat
    13. 123458 Democrat
    14. 123459 Democrat
    15. 123460 Democrat
    16. Name: cand_nm, dtype: object
    17. # Add it as a column
    18. In [193]: fec['party'] = fec.cand_nm.map(parties)
    19. In [194]: fec['party'].value_counts()
    20. Out[194]:
    21. Democrat 593746
    22. Republican 407985
    23. Name: party, dtype: int64

    这里有两个需要注意的地方。第一,该数据既包括赞助也包括退款(负的出资额):

    1. In [195]: (fec.contb_receipt_amt > 0).value_counts()
    2. Out[195]:
    3. True 991475
    4. False 10256
    5. Name: contb_receipt_amt, dtype: int64

    为了简化分析过程,我限定该数据集只能有正的出资额:

    1. In [196]: fec = fec[fec.contb_receipt_amt > 0]

    由于Barack Obama和Mitt Romney是最主要的两名候选人,所以我还专门准备了一个子集,只包含针对他们两人的竞选活动的赞助信息:

    1. In [197]: fec_mrbo = fec[fec.cand_nm.isin(['Obama, Barack','Romney, Mitt'])]

    根据职业和雇主统计赞助信息

    基于职业的赞助信息统计是另一种经常被研究的统计任务。例如,律师们更倾向于资助民主党,而企业主则更倾向于资助共和党。你可以不相信我,自己看那些数据就知道了。首先,根据职业计算出资总额,这很简单:

    1. In [198]: fec.contbr_occupation.value_counts()[:10]
    2. Out[198]:
    3. RETIRED 233990
    4. INFORMATION REQUESTED 35107
    5. ATTORNEY 34286
    6. HOMEMAKER 29931
    7. PHYSICIAN 23432
    8. INFORMATION REQUESTED PER BEST EFFORTS 21138
    9. ENGINEER 14334
    10. TEACHER 13990
    11. CONSULTANT 13273
    12. PROFESSOR 12555
    13. Name: contbr_occupation, dtype: int64

    不难看出,许多职业都涉及相同的基本工作类型,或者同一样东西有多种变体。下面的代码片段可以清理一些这样的数据(将一个职业信息映射到另一个)。注意,这里巧妙地利用了dict.get,它允许没有映射关系的职业也能“通过”:

    1. occ_mapping = {
    2. 'INFORMATION REQUESTED PER BEST EFFORTS' : 'NOT PROVIDED',
    3. 'INFORMATION REQUESTED' : 'NOT PROVIDED',
    4. 'INFORMATION REQUESTED (BEST EFFORTS)' : 'NOT PROVIDED',
    5. 'C.E.O.': 'CEO'
    6. }
    7. # If no mapping provided, return x
    8. f = lambda x: occ_mapping.get(x, x)
    9. fec.contbr_occupation = fec.contbr_occupation.map(f)

    我对雇主信息也进行了同样的处理:

    1. emp_mapping = {
    2. 'INFORMATION REQUESTED PER BEST EFFORTS' : 'NOT PROVIDED',
    3. 'INFORMATION REQUESTED' : 'NOT PROVIDED',
    4. 'SELF' : 'SELF-EMPLOYED',
    5. 'SELF EMPLOYED' : 'SELF-EMPLOYED',
    6. }
    7. # If no mapping provided, return x
    8. f = lambda x: emp_mapping.get(x, x)
    9. fec.contbr_employer = fec.contbr_employer.map(f)

    现在,你可以通过pivot_table根据党派和职业对数据进行聚合,然后过滤掉总出资额不足200万美元的数据:

    1. In [201]: by_occupation = fec.pivot_table('contb_receipt_amt',
    2. .....: index='contbr_occupation',
    3. .....: columns='party', aggfunc='sum')
    4. In [202]: over_2mm = by_occupation[by_occupation.sum(1) > 2000000]
    5. In [203]: over_2mm
    6. Out[203]:
    7. party Democrat Republican
    8. contbr_occupation
    9. ATTORNEY 11141982.97 7.477194e+06
    10. CEO 2074974.79 4.211041e+06
    11. CONSULTANT 2459912.71 2.544725e+06
    12. ENGINEER 951525.55 1.818374e+06
    13. EXECUTIVE 1355161.05 4.138850e+06
    14. ... ... ...
    15. PRESIDENT 1878509.95 4.720924e+06
    16. PROFESSOR 2165071.08 2.967027e+05
    17. REAL ESTATE 528902.09 1.625902e+06
    18. RETIRED 25305116.38 2.356124e+07
    19. SELF-EMPLOYED 672393.40 1.640253e+06
    20. [17 rows x 2 columns]

    把这些数据做成柱状图看起来会更加清楚(’barh’表示水平柱状图,如图14-12所示):

    1. In [205]: over_2mm.plot(kind='barh')

    图14-12 对各党派总出资额最高的职业

    你可能还想了解一下对Obama和Romney总出资额最高的职业和企业。为此,我们先对候选人进行分组,然后使用本章前面介绍的类似top的方法:

    1. def get_top_amounts(group, key, n=5):
    2. totals = group.groupby(key)['contb_receipt_amt'].sum()
    3. return totals.nlargest(n)

    然后根据职业和雇主进行聚合:

    1. In [207]: grouped = fec_mrbo.groupby('cand_nm')
    2. In [208]: grouped.apply(get_top_amounts, 'contbr_occupation', n=7)
    3. Out[208]:
    4. cand_nm contbr_occupation
    5. Obama, Barack RETIRED 25305116.38
    6. ATTORNEY 11141982.97
    7. INFORMATION REQUESTED 4866973.96
    8. HOMEMAKER 4248875.80
    9. PHYSICIAN 3735124.94
    10. ...
    11. Romney, Mitt HOMEMAKER 8147446.22
    12. ATTORNEY 5364718.82
    13. PRESIDENT 2491244.89
    14. EXECUTIVE 2300947.03
    15. C.E.O. 1968386.11
    16. Name: contb_receipt_amt, Length: 14, dtype: float64
    17. In [209]: grouped.apply(get_top_amounts, 'contbr_employer', n=10)
    18. Out[209]:
    19. cand_nm contbr_employer
    20. Obama, Barack RETIRED 22694358.85
    21. SELF-EMPLOYED 17080985.96
    22. NOT EMPLOYED 8586308.70
    23. INFORMATION REQUESTED 5053480.37
    24. HOMEMAKER 2605408.54
    25. ...
    26. Romney, Mitt CREDIT SUISSE 281150.00
    27. MORGAN STANLEY 267266.00
    28. GOLDMAN SACH & CO. 238250.00
    29. BARCLAYS CAPITAL 162750.00
    30. H.I.G. CAPITAL 139500.00
    31. Name: contb_receipt_amt, Length: 20, dtype: float64

    对出资额分组

    还可以对该数据做另一种非常实用的分析:利用cut函数根据出资额的大小将数据离散化到多个面元中:

    1. In [210]: bins = np.array([0, 1, 10, 100, 1000, 10000,
    2. .....: 100000, 1000000, 10000000])
    3. In [211]: labels = pd.cut(fec_mrbo.contb_receipt_amt, bins)
    4. In [212]: labels
    5. Out[212]:
    6. 411 (10, 100]
    7. 412 (100, 1000]
    8. 413 (100, 1000]
    9. 414 (10, 100]
    10. 415 (10, 100]
    11. ...
    12. 701381 (10, 100]
    13. 701382 (100, 1000]
    14. 701383 (1, 10]
    15. 701384 (10, 100]
    16. 701385 (100, 1000]
    17. Name: contb_receipt_amt, Length: 694282, dtype: category
    18. Categories (8, interval[int64]): [(0, 1] < (1, 10] < (10, 100] < (100, 1000] < (1
    19. 000, 10000] <
    20. (10000, 100000] < (100000, 1000000] < (1000000,
    21. 10000000]]

    现在可以根据候选人姓名以及面元标签对奥巴马和罗姆尼数据进行分组,以得到一个柱状图:

    1. In [213]: grouped = fec_mrbo.groupby(['cand_nm', labels])
    2. In [214]: grouped.size().unstack(0)
    3. Out[214]:
    4. cand_nm Obama, Barack Romney, Mitt
    5. contb_receipt_amt
    6. (0, 1] 493.0 77.0
    7. (1, 10] 40070.0 3681.0
    8. (10, 100] 372280.0 31853.0
    9. (100, 1000] 153991.0 43357.0
    10. (1000, 10000] 22284.0 26186.0
    11. (10000, 100000] 2.0 1.0
    12. (100000, 1000000] 3.0 NaN
    13. (1000000, 10000000] 4.0 NaN

    从这个数据中可以看出,在小额赞助方面,Obama获得的数量比Romney多得多。你还可以对出资额求和并在面元内规格化,以便图形化显示两位候选人各种赞助额度的比例(见图14-13):

    1. In [216]: bucket_sums = grouped.contb_receipt_amt.sum().unstack(0)
    2. In [217]: normed_sums = bucket_sums.div(bucket_sums.sum(axis=1), axis=0)
    3. In [218]: normed_sums
    4. Out[218]:
    5. cand_nm Obama, Barack Romney, Mitt
    6. contb_receipt_amt
    7. (0, 1] 0.805182 0.194818
    8. (1, 10] 0.918767 0.081233
    9. (10, 100] 0.910769 0.089231
    10. (100, 1000] 0.710176 0.289824
    11. (1000, 10000] 0.447326 0.552674
    12. (10000, 100000] 0.823120 0.176880
    13. (100000, 1000000] 1.000000 NaN
    14. (1000000, 10000000] 1.000000 NaN
    15. In [219]: normed_sums[:-2].plot(kind='barh')

    图14-13 两位候选人收到的各种捐赠额度的总额比例

    我排除了两个最大的面元,因为这些不是由个人捐赠的。

    还可以对该分析过程做许多的提炼和改进。比如说,可以根据赞助人的姓名和邮编对数据进行聚合,以便找出哪些人进行了多次小额捐款,哪些人又进行了一次或多次大额捐款。我强烈建议你下载这些数据并自己摸索一下。

    根据州统计赞助信息

    根据候选人和州对数据进行聚合是常规操作:

    1. In [220]: grouped = fec_mrbo.groupby(['cand_nm', 'contbr_st'])
    2. In [221]: totals = grouped.contb_receipt_amt.sum().unstack(0).fillna(0)
    3. In [222]: totals = totals[totals.sum(1) > 100000]
    4. In [223]: totals[:10]
    5. Out[223]:
    6. cand_nm Obama, Barack Romney, Mitt
    7. contbr_st
    8. AK 281840.15 86204.24
    9. AL 543123.48 527303.51
    10. AR 359247.28 105556.00
    11. AZ 1506476.98 1888436.23
    12. CA 23824984.24 11237636.60
    13. CO 2132429.49 1506714.12
    14. CT 2068291.26 3499475.45
    15. DC 4373538.80 1025137.50
    16. DE 336669.14 82712.00
    17. FL 7318178.58 8338458.81

    如果对各行除以总赞助额,就会得到各候选人在各州的总赞助额比例:

    1. In [224]: percent = totals.div(totals.sum(1), axis=0)
    2. In [225]: percent[:10]
    3. Out[225]:
    4. cand_nm Obama, Barack Romney, Mitt
    5. contbr_st
    6. AK 0.765778 0.234222
    7. AL 0.507390 0.492610
    8. AR 0.772902 0.227098
    9. AZ 0.443745 0.556255
    10. CA 0.679498 0.320502
    11. CO 0.585970 0.414030
    12. CT 0.371476 0.628524
    13. DC 0.810113 0.189887
    14. DE 0.802776 0.197224
    15. FL 0.467417 0.532583