U AcjU @sxUddlmZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z ddlmZmZmZmZmZdd lmZdd lmZmZdd lmZmZmZmZmZmZe d Z!e"Z#e#$e%d e&edddZ'de(d<d.dddddddddddd d d!Z)d/d"ddddddddddd# d$d%Z*d0d&ddddddddddd' d(d)Z+d1d*ddddddddddd+ d,d-Z,dS)2) annotationsN) lru_cache)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDIANA_SUPPORTED_SIMILARTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bomis_multi_byte_encodingshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)scCs t| SN)rencodingr:/tmp/pip-unpacked-wheel-k0k9q145/charset_normalizer/api.py.r)keyz list[str]IANA_SUPPORTED_MB_FIRST皙?TF皙?zbytes | bytearrayintfloatzlist[str] | Noneboolr) sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainlanguage_thresholdenable_fallbackreturnc D Cs&t|ttfs tdt||r>tj} tt t t t |} | dkrt d|rptt t | tt|dddgdgS|dk rtt d d |d d |D}ng}|dk rtt d d |dd |D}ng}| ||krtt d||| d}| }|dkr4| ||kr4t| |}t |tk} t |tk} | rftt d| n| r~tt d| g}|rt|nd}|dk r||tt d|t}g}g}t}t}i}tddt}tddt}d}t}d}d}d}d}d}d}t} t}!t|\}"}#|"dk rR||"tt dt |#|"|dd|krp|d|tD ]}$|r|$|krqx|r|$|krqx|$|krqx||$d}%|"|$k}&|&ot|$}'|$dkr|&stt d|$qx|$dkr|&stt d|$qx|$|kr8tt d|$qx|$|krTtt d|$qxz t |$}(Wn.t!t"fk rtt d|$YqxYnX|r|(stt#|$})n tt$|$})|)%|stt d|$|)|qx|r|(s||krtt d |$||qx|r"|(s"tt d!|$qx|( o.| }*z| rr|(srt&|'sV|dtd"n|t |#td"|$d#nf|*s|$d$kr|&rt&||$d#}%|%r|%dd%kr|%dd}%n"t&|'s|n|t |#d|$d#}%Wn\t't(fk r6}+z8t|+t(stt d&|$t&|+||$WYqxW5d}+~+XYnXt)|&sDdnt |#| t| |},|(ot|%dk ott |%| k}-|-rtt d'|$tt |,d(}.t*|.d)}.d}/d}0g}1g}2zt+||$|,||&|'|#|(|%|* D]t}3|1|3|2||3||o dt |kod)kn|2d*|kr&|/d7}/|/|.ks<|&r|'sqFqWnrt't(fk r}+zN|*rtt d&|$t&|+||$WY$qxtt d+|$t&|+|.}/d,}0W5d}+~+XYnX|0s:| r:|(s:z|td-dj,|$d.d/WnLt'k r8}+z,tt d0|$t&|+||$WYqxW5d}+~+XYnX|2rPt-|2t |2nd}4|4|ksh|/|.kr||$|$t.kr|/t.|$|%dk r|(s|0t1|%|4gdftt d1|$|/t2|4d2d3d4| rx|$dd|d5d6fkrx|0sx|%dkrXz&t&|'s|n|t |#d|$d#}%Wn.t't(fk rLtt d7|$YqxYnX| rXd}%t||$||&g|%|d8}5|$|kr~|5}n|$dkr|5}n|5}qx|* rz&t&|'s|n|t |#d|$d#}%WnPt't(fk r}+z,tt d&|$t&|+||$WYqxW5d}+~+XYnX|%dk r|( st1|%}6|3|6}7|7dk r|7\}8}9}:|: rt||$|8|&|9| rr|$|ddfk rv|%nd|d8};| |;||$tt d9|$t2|8d2d3d4|$|ddfk r|8d:k r|8dk r t d;|;j4| rtt t | t|;gS|!|;t |!rx|dk s2||krxd|krxd|krx|!5}|$t&|=g}>|$dk r|= rjd?|=nd}?|1D]}3||3||?}@|>|@ qrt6|>}A|A rtt d@|A|$t||$|4|&|A| r|$|ddfk r|%nd|d8}B| |B|%dk r|( s|0t1|%|4|Ad,f| r0|( s0|4dAk r0|d7}|$|ddfk r|4d:k r|4dk rt d;|Bj4| r|tt t | t|BgS|!|Bt |! r|dk s||k rd|k rd|k r|!5}lszfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSsg|]}t|dqSr6r7r8rrrr;wsz^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.)maxsizezIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_32utf_16z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.zY%s is deemed too similar to a code page that was already considered unsuited. Continuing!zESkipping %s: already fast-tracked from a similar successful encoding.z2Encoding %s does not provide an IncrementalDecoderzbSkipping %s: definitive match already found, this encoding targets different languages (%s vs %s).zXSkipping %s: already accumulated %d same-family results after definitive match (cap=%d).zCSkipping single-byte %s: multi-byte definitive match already found.gArrAuz9Code page %s does not fit given bytes sequence at ALL. %szpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sTgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsr@r?z>%s does not decode the whole payload: fallback entry withheld.)Zpreemptive_declarationzM%s fast-tracked (identical decoded payload to a prior encoding, chaos=%f %%).r%z.Encoding detection: %s is most likely the one.zZ%s fast-skipped (identical decoded payload to a prior encoding that failed chaos probing).z=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}g{Gz?css|]\}}|VqdSrr)r9_vrrr Cszfrom_bytes..)defaultg?zyDefinitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.g\(\?> utf_16_ler?r4r@ utf_16_berA utf_8_sig utf_32_be utf_32_lezjMulti-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)8 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerrrlogjoinr&rr rappendsetrrrrr!addrrModuleNotFoundError ImportErrorrr intersectionstrUnicodeDecodeError LookupErrorrangemaxrdecodesumr update setdefaulthashroundgetrbestr fingerprint)Dr)r*r+r,r-r.r/r0r1r2Zprevious_logger_levellengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZsoft_failure_skipZsuccess_fast_trackedZpayload_result_cacheZcached_mess_ratioZcached_coherence_ratioZdefinitive_match_foundZdefinitive_target_languagesZ post_definitive_sb_success_countZPOST_DEFINITIVE_SB_CAPZmb_definitive_match_foundZfallback_asciiZ fallback_u8Zfallback_specifiedresultsZearly_stop_resultsZ sig_encodingZ sig_payloadZ encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decoderZ enc_languagesZdeferred_decodingeZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratioschunkZmean_mess_ratioZfallback_entryZ payload_hashcachedZ cached_messZ cached_cdZ cached_passedZ fast_matchZprobable_resultZtarget_languagesZ cd_ratiosZ lg_inclusionZchunk_languagesZcd_ratios_mergedZ current_matchZbest_coherencerrr from_bytes2s*                                    "                                                          r}r) fpr*r+r,r-r.r/r0r1r2r3c Cst|||||||||| S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )r}read) r~r*r+r,r-r.r/r0r1r2rrrfrom_fpsrzstr | bytes | PathLike) pathr*r+r,r-r.r/r0r1r2r3c Cs<t|d(} t| ||||||||| W5QRSQRXdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rr*r+r,r-r.r/r0r1r2r~rrr from_paths rz!PathLike | str | BinaryIO | bytes) fp_or_path_or_payloadr*r+r,r-r.r/r0r1r2r3c Cszt|ttfr,t|||||||||| d } nHt|ttfrXt|||||||||| d } nt|||||||||| d } | S)a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) r*r+r,r-r.r/r0r1r2)rTrjrrrVrUr}r) rr*r+r,r-r.r/r0r1r2Zguessesrrr is_binarysX   r) r"r#r$NNTFr%T) r"r#r$NNTFr%T) r"r#r$NNTFr%T) r"r#r$NNTFr%F)- __future__rlogging functoolsrosrtypingrZcdrrr r Zconstantr r r rrZmdrmodelsrrutilsrrrrrr getLoggerrZ StreamHandlerr] setFormatter Formattersortedr!__annotations__r}rrrrrrrs       $$ $!