文章目录

【作者介绍】擅长爬虫与JS加密逆向分析Python领域优质创作者、CSDN博客专家、阿里云博客专家、华为云享专家。

一路走来长期坚守并致力于Python与爬虫领域研究与开发工作

【作者推荐】对爬虫领域以及JS逆向分析感兴趣的朋友可以关注《爬虫JS逆向实战》《深耕爬虫领域》

未来作者会持续更新所用到、学到、看到的技术知识包括但不限于各类验证码突防、爬虫APP与JS逆向分析、RPA自动化、分布式爬虫、Python领域等相关文章

作者声明文章仅供学习交流与参考严禁用于任何商业与非法用途否则由此产生的一切后果均与作者无关如有侵权请联系作者本人进行删除

写在前面

这个网站也是作者最近接触到的一个APP应用市场类网站。

讲实话还是蛮适合新手朋友去动手学习的。

毕竟爬虫领域要想进步还是需要多实战、多分析该网站中的一些小细节也是能够锻炼分析能力的也有反爬虫处理。

甚至是下载APP的话在Web端是无法拿到APK下载的直链需要去APP端接口数据获取

接口分析

需要抓取的内容为整个游戏板块当然可以是所有板块甚至是关键词去搜素命中。

游戏板块包含了所有分类与子分类下APP信息如下所示

import

https://appgallery.huawei.com/,Interface-Code:

6bc2bec970e616747dc0a99b57aa6730_1716973000358,sec-ch-ua-mobile:

?0,User-Agent:

Safari/537.36,sec-ch-ua-platform:

\macOS\

https://web-drcn.hispace.dbankcloud.com/edge/uowap/index

params

internal.getTabDetail,serviceType:

20,reqPageNum:

b2b4752f0a524fe5ad900870f88c11ed,maxResults:

25,zone:

paramsparams)print(response.json())这里直接请求会失败因为其中有一个细节的反爬虫检测代码运行会提示接口验证失败如下所示

{rtnCode:

failed.}这个是什么原因根据返回提示中接口检验的问题回看请求头内携带有Interface-Code参数大概率是这个参数的问题重放失败这个参数是动态的不过好在并非算法加密生成

Interface-Code参数其实相当于一个动态注册的令牌在我们请求接口的时候。

需要去固定的接口请求并拿到值然后携带这个参数的值进行后续的任何请求

如上图动态获取Interface-Code参数的接口可每次在提交页面请求之前从这个接口拿动态值携带

爬虫开发

这里我们经过分析可以找到属于游戏板块全量分类的一个唯一ID拿它可以获取到所有的子分类ID如下所示

完整链接地址如下

https://web-drcn.hispace.dbankcloud.com/edge/uowap/index?methodinternal.getTabDetailserviceType20reqPageNum1uri33ef450cbac34770a477cfa78db4cf8cmaxResults25zonelocaleen这个接口请求之后去解析拿到tabInfo下面的tabId字段即可这是子分类板块的ID用来后续请求细分栏目下的所有游戏类APP所需要的字段同样放到上面链接替换uri如下所示

appid

fhttps://appgallery.huawei.com/#/app/{appid}encoded_appid

quote(fapp|{appid})encoded_detail_link

quote(quote(detail_link))detail_json_link

(https://web-drcn.hispace.dbankcloud.cn/uowap/index?methodinternal.getTabDetailserviceType20reqPageNum1maxResults25furi{encoded_appid}shareTocurrentUrl{encoded_detail_link}accessIdfappid{appid}zonelocalezh)response

headersheaders)

对详情页面接口构造完请求后同样将会得到JSON数据需要对结构化数据进行解析实现代码如下

import

dateparser.parse(pub_time).strftime(%Y-%m-%d

%H:%M:%S)

process_downloads_times(appdata.get(intro,

如上拿到完整的APP信息数据包括名称、版本、发布时间、大小、下载量、开发者…其中process_downloads_times方法是一个自定义方法对下载量数据进行清洗页面下载量如下

import

process_downloads_times(data):if

not

.join(pa.findall(data))try:return

str(int(float(num)

接下来最重要的就是对APK包的下载了在Web端可以看到有一个getAppDownloadUrl的接口但是它并不是APK包的下载链接它是华为应用商店的APP下载链接所以注意不要被误导了如下所示

要获取APK包的下载链接需要从APP端入手Web网站内不再提供APK的下载直链以前是可以的早期的下载链接直接拼appid即可早期链接如下所示

ttps://appgallery.cloud.huawei.com/appdl/C100210735这里需要对手机端的华为应用商店APP进行抓包分析拿到下载链接接口最终APK在移动端的下载接口如下所示

https://store-drcn.hispace.dbankcloud.com/hwmarket/api/clientApi这里下载接口的请求参数巨长好在没有加密的参数字段如下所示

data

apsid1705374325915brandgooglechannelIdbackgroundclientPackagecom.huawei.appmarketcno4010001code0200contentPkgdataFilterSwitchdeviceIda7a6dcc273d12e3f63865060533dec04f24e5042e48a621c1b5e601953e1bb69deviceIdRealType4deviceIdType9deviceSpecParams%7B%22abis%22%3A%22arm64-v8a%2Carmeabi-v7a%2Carmeabi%22%2C%22deviceFeatures%22%3A%22U%2Ccom.verizon.hardware.telephony.lte%2Ccom.verizon.hardware.telephony.ehrpd%2CP%2CB%2C0c%2Ce%2C0J%2Cp%2Ca%2Cb%2C04%2Cm%2Candroid.hardware.wifi.rtt%2Ccom.google.android.feature.PIXEL_2017_EXPERIENCE%2C08%2C03%2CC%2CS%2C0G%2Cq%2Ccom.google.android.feature.PIXEL_2018_EXPERIENCE%2CL%2C2%2C6%2Ccom.google.android.feature.GOOGLE_BUILD%2CY%2C0M%2Candroid.hardware.vr.high_performance%2Cf%2C1%2C07%2C8%2C9%2Candroid.hardware.sensor.hifi_sensors%2Candroid.hardware.strongbox_keystore%2CO%2CH%2Ccom.google.android.feature.TURBO_PRELOAD%2Candroid.hardware.vr.headtracking%2CW%2Cx%2CG%2Co%2C06%2C0N%2Ccom.google.android.feature.PIXEL_EXPERIENCE%2C3%2CR%2Cd%2CQ%2Cn%2Candroid.hardware.telephony.carrierlock%2Cy%2CT%2Ci%2Cr%2Cu%2Ccom.google.android.feature.WELLBEING%2Cl%2C4%2C0Q%2CN%2Candroid.software.device_id_attestation%2CM%2C01%2C09%2CV%2C7%2C5%2C0H%2Cg%2Cs%2Cc%2CF%2Ct%2C0L%2C0W%2Ccom.google.hardware.camera.easel_2018%2Ccom.google.android.apps.dialer.SUPPORTED%2C0X%2Ck%2C00%2Ccom.google.android.feature.GOOGLE_EXPERIENCE%2Ccom.google.android.feature.EXCHANGE_6_2%2Ccom.google.android.apps.photos.PIXEL_2018_PRELOAD%2Candroid.hardware.sensor.assist%2Ccom.google.android.feature.DREAMLINER%2Candroid.hardware.audio.pro%2CK%2CE%2C02%2CI%2CJ%2Cj%2CD%2Ch%2Candroid.hardware.wifi.aware%2CX%2Cv%22%2C%22dpi%22%3A560%2C%22openglExts%22%3A%221%2C0y%2C1d%2C1e%2C1f%2C0c%2CM%2CO%2C0q%2C0r%2C0s%2CB%2CA%2C5%2C4%2C0p%2CD%2CE%2C0m%2C0n%2C7%2C0i%2C0j%2C0k%2C8%2C0t%2C0w%2C1g%2C1o%2C1m%2C1n%2CH%2C0u%2C1i%2C1l%22%2C%22preferLan%22%3A%22zh%22%2C%22usesLibrary%22%3A%225%2C6%2Ccom.vzw.apnlib%2C09%2C0D%2Ccom.google.android.camera.experimental2018%2CA%2C0E%2C0L%2Ccom.google.android.poweranomalydatamodeminterface%2C9%2C2%2Cb%2C0C%2Ccom.android.ims.rcsmanager%2CE%2C1%2Ccom.verizon.embms%2Ccom.qualcomm.qti.uim.uimservicelibrary%2Ccom.google.android.lowpowermonitordevicefactory%2Ccom.google.android.lowpowermonitordeviceinterface%2Ccom.google.android.poweranomalydatafactory%2C0H%2C08%2C7%2Ccom.google.android.dialer.support%2Ccom.google.vr.platform%2CF%2Ccom.verizon.provider%2Cd%2C0A%2Ccom.google.android.hardwareinfo%2CB%2C0K%2CC%2C07%22%7Dfid0globalTracenullgradeLevel0gradeTypehardwareType0isSupportPage1manufacturerGooglemaxResults25methodclient.getTabDetailnet1outside0recommendSwitch1reqPageNum1roamingTime0runMode2serviceType0shellApkVer0sid1705386105840signh9001090fp00010920000000000001000a0000000600100000011190000010000040240b0100001001000%40000F6C248B294FAD99F665A13B23795DthirdPartyPkgcom.huawei.appmarkettranslateFlag1ts{ts}uriapp%7C{appid}__HiAd__fed345ef91e24079b46e93c92d7e7d4e__cds_111122701__21__null____null%3Faglocation%3D%257B%2522cres%2522%253A%257B%2522lPos%2522%253A0%252C%2522lid%2522%253A%2522905310%2522%252C%2522pos%2522%253A21%252C%2522relResId%2522%253A%2522{appid}%2522%252C%2522relType%2522%253A%2522app%2522%252C%2522resid%2522%253A%2522{appid}%2522%252C%2522rest%2522%253A%2522app%2522%252C%2522tid%2522%253A%2522dist_83ae03cda9994714a499347e25bbccd1%2522%257D%252C%2522ftid%2522%253A%2522dist_25f5040f852a48c1b3bb52c3a99f4cf3%2522%252C%2522pres%2522%253A%257B%2522lPos%2522%253A0%252C%2522pos%2522%253A1%252C%2522resid%2522%253A%252283ae03cda9994714a499347e25bbccd1%2522%252C%2522rest%2522%253A%2522tab%2522%252C%2522tid%2522%253A%2522dist_25f5040f852a48c1b3bb52c3a99f4cf3%2522%257D%257D%26templateId%3D1428744958c44b80aad5091114d7fce4%26maple%3D0%26trackId%3D0%26attribution%3D%257B%2522taskid%2522%253A%2522910297447%2522%252C%2522subTaskId%2522%253A%25229102974470000%2522%252C%2522RTAID%2522%253A%2522%2522%252C%2522callback%2522%253A%2522security%253A44D8410ED7A32A50E8AABFD6%253A03EC36808D57EC58A89C207497BB1CED2B6192ED09D44BAC9D825C50BF49E14A95A9E0E6281F83A1DA4C1F2E6D7A8A%2522%252C%2522channel%2522%253A%25220%2522%257D%26appoid%3Ddde9f2d763154f37acc76b8620d195a9%26listId%3Dcds_111122701%26listIdType%3Dcds%26adFlag%3DHiAd%26cdrInfo%3D20240116142810el21s692259%255E%257BopType%257D%255E910297447%255E{appid}%255Ecds_111122701%255E21%255E608743720197185117_p21%255E608743720197185117%255E14effd3e76d34d76d6aa4185d8d08e5bb340dfdcb8ce0227285d09b8c9001ce2f315fce6f842b5449b4eac5ebebbd78bf7f6f470baaadc081ae820e43b6653825f66a6e7b6f2892cd760d063df840dd9954254fb48fa6a9f19259d68c7170c54%255Ectr%255Efn5-Y29tLmh1YXdlaS5hcHBtYXJrZXR-LTF-MTcwNTMxMDUxODQ0OQ%255E2024-01-16%2B14%253A28%253A10%255E%255EU8800%255E0.000126%255E0%255E2.0%255E2.0%255E9102974470000%255E900011964%255E%255E%255E%255E%255E11.4.1.303%255E1705374325914%255E0%255E1%255E%255E%255E%255E9%255E00000000-0000-0000-0000-000000000000%255E1%255E%255E%255ECN%255ECNY%255E243728192482956951%255Eappgallery%255E%255E%255E%255E2ea67221765846d69b880f43a78a7ab3%255ECPD%255E69%255E%255E%255E243728192570833732%255EWIFI%255EMA%255E%255EZXhwU3RyYXRlZ3k9I2V4cFBhcmFtcz0xMzAxLmFncmVjX2Fkdl9leHRlbmRfdWJyX2Rvd25fYXBwXzF5X3dpdGhfcmVhbHRpbWVfcXVlcnkjYnVja2V0SWQ9I2V4cERlcz0xMzAxLmFncmVjX2Fkdl9leHRlbmRfdWJyX2Rvd25fYXBwXzF5X3dpdGhfcmVhbHRpbWVfcXVlcnkjZXhwSWQ9MTMxN19WMi5hZ18xMzE3X1YyX2FsbF90YXJnZXRfb3B0aW1pemF0aW9uX29uZWZvcmFsbHwxMzAxLmFncmVjX2Fkdl9leHRlbmRfdWJyX2Rvd25fYXBwXzF5X3dpdGhfcmVhbHRpbWVfcXVlcnl8MTMwN19WMi5hZ18xMzA3X1YyX2NvbnRyb2xsZXI%255E%255E%255E%255E1%255E1%255E4010001%255E%255E3G%255E%255E0%255E%255E%255E%255E%255E%255E0%255E1%255E1027162%255E%255E1%255EP8%255E%255E%255Ecom.huawei.appmarket%255E%255E%255E%255E%255E%255E%255E%255E%255Ecom.huawei.appmarket%255E0.000252%255E%255E%255E20240116142810el21s692259%255E20240116142810el21s692182%255E1%255E0%255E%255E%255Esign%253A987064c954dab704e2b30dfe17f5ca63dae4245392d4fff32e75670e44da8998%26version%3D2%26phase%3D0%26requestId%3D2ea67221765846d69b880f43a78a7ab3%26algExp%3D%257B%2522expStrategy%2522%253A%2522246353%2522%252C%2522expParams%2522%253A%25221112.appstore_mtp_fulllist_new_baseline_dataops%2522%252C%2522bucketId%2522%253A%252236%2522%252C%2522expDes%2522%253A%25221112.appstore_mtp_fulllist_new_baseline_dataops%2522%252C%2522expId%2522%253A%25221112.appstore_mtp_fulllist_new_baseline_dataops%2522%257Dver1.1这里我们拿一个APP的appid对移动端的详情进行请求可以看到接口数据内包含APK的下载链接如下所示