python去除文本中重复的字符串 急求：如何用python删除文本中的重复行？
python\u4e2d\u600e\u6837\u5220\u6389\u5177\u6709\u76f8\u540c\u5143\u7d20\u7684\u5b57\u7b26\u4e32L=[('c', 'u'), ('x', 'f'), ('i', 'h'), ('j', 'v'), ('c', 'g'), ('r', 'w'), ('h', 's'), ('n', 'o'), ('k', 'b'), ('l', '\n'), ('i', 'y'), ('z', 'z'), ('d', 'q'), ('t', 'z'), ('o', 'x'), ('s', 'o'), ('c', 'n'), ('\n', 'f')........]newL=[i for i in L if '\n' not in i]
1.\u5982\u679c\u4f60\u7684txt \u6587\u4ef6\u4e0d\u5927\u7684\u8bdd \u53ef\u4ee5\u76f4\u63a5 tmp = open('**.txt').readlines() #\u628a\u5185\u5bb9\u4e00\u6b21\u6027\u5168\u90e8\u8bfb\u53d6\u51fa\u6765 \u662f\u4e00\u4e2a\u5217\u8868set(tmp) #\u8fd9\u4e2a\u5c31\u662f\u628a\u5217\u8868 \u53bb\u91cd\u590d \u7136\u540e \u4f60\u53ef\u4ee5\u628a \u8fd9\u4e2a\u53bb\u91cd\u540e\u7684tmp \u5199\u5165\u5230\u65b0\u7684\u6587\u4ef62.txt\u5f88\u5927\uff0c\u90a3\u4e48\u53ea\u80fd\u4e00\u884c\u4e00\u884c\u7684\u8bfb\u53d6\u53bb\u91cd\u4e86#!/usr/bin/env python# coding=utf-8# python 2.7outfile = open('result-readline.txt', 'w') #\u65b0\u7684\u6587\u4ef6list_1=[]for line in open('test.txt'): #\u8001\u6587\u4ef6 tmp = line.strip() if tmp not in list_1: list_1.append(tmp) outfile.write(line)outfile.close()
你的数据都是一行一行的吗?
是的话这样试试
input = open("a.txt", "r").read()()
output = open("b.txt", "w+")
patterns = []
for line in input.split("
"):
if line not in patterns:
print line
patterns.append(line + "
")
for pattern in patterns:
output.write(pattern)
output.close()测试了下满足你的输入输出
output=[]
with open('a.txt') as fp:
lines=fp.readlines()
for i in lines:
if i not in output:
output.append(i)
with open('b.txt','w',encoding='utf-8') as fp1:
for i in output:
fp1.write(i)
虽然已经几年过去了，还是写一下吧。以防刚好有人搜到。
Thx.
绛旓細寤鸿涓や釜鏂囦欢鐨勬湯灏鹃兘鐣欎竴涓┖琛岋紝鍚﹀垯鏈鍚庝竴琛屽彲鑳藉尮閰嶄笉鍒 fa = open('A.txt')a = fa.readlines()fa.close()fb = open('B.txt')b = fb.readlines()fb.close()c = [i for i in a if i in b]fc = open('C.txt', 'w')fc.writelines(c)fc.close()...
绛旓細鍒╃敤闆嗗悎鐨勪笉閲嶅灞炴э紝鍙互鍏堣浆鎹㈣嚦闆嗗悎锛屽啀鐢╨ist()鍑芥暟杞崲鍥炴潵鍗冲彲銆傛瘮濡傦紝a鏄竴涓垪琛紝a=list(set(a))锛屽嵆鍙畬鎴愬垪琛ㄥ幓閲嶃
绛旓細浣犵殑鏁版嵁閮芥槸涓琛屼竴琛岀殑鍚?鏄殑璇濊繖鏍疯瘯璇 input = open("a.txt", "r").read()output = open("b.txt", "w+")patterns = []for line in input.split("\n"): if line not in patterns: print line patterns.append(line + "\n")for pattern in patterns: output.write...
绛旓細瀛楀吀涔熸槸澶ф嫭鍙穥}锛屼絾鏄窡闆嗗悎杩樻槸鏈夊尯鍒1.闆嗗悎娌℃湁閲嶅鐨鍏冪礌锛屽垪琛ㄥ彲浠ユ湁閲嶅鍏冪礌 闆嗗悎浼氳嚜鍔ㄥ皢閲嶅鐨勫瓧绗︾粰鍒犳帀锛岃屽垪琛ㄤ細鍘熸牱杈撳嚭鏄剧ず 2.闆嗗悎涓鐨勫厓绱犱笌椤哄簭鏃犲叧锛岃屽垪琛ㄤ腑鐨勫厓绱犱笌椤哄簭鏈夊叧 1.闆嗗悎{}娌℃湁閲嶅鐨勫厓绱 2.闆嗗悎{}涓殑鍏冪礌璺熼『搴忔棤鍏 3.灏嗗垪琛╗],鍏冪粍() 杞崲鎴愰泦鍚堝悗...
绛旓細鍋囪浣犵殑鏂囦欢鍚嶆槸a.txt锛屽啓鍒癰.txt d = {}for line in open('a.txt'): d[line] = d.get(line, 0) + 1 fd = open('b.txt', 'w')for k, v in d.items(): if v > 1: fd.write(k)fd.close()
绛旓細lista=[1,7,5,7]#缁欏畾鏁版嵁 listb=[]#瀹氫箟绌哄垪琛 for j in lista:#閬嶅巻lista '''(Tab)澶勭缉杩涗唬鐮'''(Tab)if lista.count(j)==1:#鍗曟鏉′欢 (Tab)(Tab)listb.append(j)#绗﹀悎鏉′欢鍒欏瓨琛 print(listb)#杈撳嚭缁撴灉 '''杩愯鏁堟灉 [1, 5]'''
绛旓細python data_unique = data.drop_duplicates(keep=False)浠ヤ笂灏辨槸鍦Python涓鍒犻櫎DataFrame涓畬鍏閲嶅鐨琛岀殑鏂规硶銆傚鏋滀綘闇瑕佸熀浜庢煇浜涘垪鏉ュ垹闄ら噸澶嶇殑琛岋紙鍗宠繖浜涘垪瀹屽叏鐩稿悓鍗充负閲嶅锛夛紝浣犲彲浠ュ皢鍒楀悕鏀惧叆涓涓垪琛ㄤ腑锛岀劧鍚庝紶閫掔粰drop_duplicates鏂规硶鐨剆ubset鍙傛暟銆備緥濡傦紝鍙熀浜庡垪'A'鍜'B'鏉ュ垹闄ら噸澶嶇殑琛岋細python...
绛旓細鎶婇敭鈥榓鈥欏搴旂殑鍊糩3,4,4,4,3]锛岀敤set杞负闆嗗悎灏卞彲浠ュ幓閲嶃傝ˉ鍏咃紝set鏄泦鍚堬紝鏃犲簭涓斾笉閲嶅锛屾湁閲嶅鐨涔熶細鑷姩鍘婚噸
绛旓細鍙互鏀瑰彉涓嬫濊矾,鍑忓皯寰幆娆℃暟:list杞负set,鐒跺悗&涓庝笅鎵惧埌鐩稿悓鍊,鎺ョ潃鍐嶅拰str2寰幆in鐨剅emove鎺;鎴栬卻et鍚庣洿鎺ュ噺 濡傛灉瀹炲湪澶(瓒呰繃1w涓瓧绗)鍙﹀涓涓濊矾鏄敤绾跨▼,鍗冲涓や釜list鍒囩墖,鐒跺悗澶氱嚎绋嬪鐞.
绛旓細2. 鍘熷瓧绗︿覆浠ラ楀彿鍒嗛殧鐨勶紝鍚庨潰鏈変竴涓垨澶氫釜瀛楃涓诧紝鎵浠e.split(', | ')銆3. 鎵цre.split(r', | ', S)鎿嶄綔涔嬪悗锛屽垪琛ㄤ腑浼氫骇鐢熷ぇ閲忕殑''锛屽氨闇瑕佸皢filter杩囨护鎺夈4. 浣跨敤L.count(x) == 1 鎴栬 L.count(x) > 1鏉ヤ繚鐣閲嶅椤规垨锛岄潪閲嶅椤广5. set(L)鍒欐槸淇濈暀鍒楄〃涓殑鍞竴椤癸紝...
