o i1@s UddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZd d lmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$e ed duddZ%e ed dvddZ&e ed dwddZ'e(dde)DZ*de+d<dde*DZ,de+d<e ed dxd!d"Z-e ed dvd#d$Z.e ed dvd%d&Z/e ed dvd'd(Z0e ed dvd)d*Z1e ed dvd+d,Z2e ed dvd-d.Z3e ed dvd/d0Z4e ed dvd1d2Z5e ed dvd3d4Z6e ed dvd5d6Z7e ed dvd7d8Z8e ed dvd9d:Z9e ed dvd;d<Z:e ed dvd=d>Z;e e Ddzd{dHdIZ?e dJd d|dLdMZ@d}dOdPZAd~dRdSZBdddWdXZCdd\d]ZDdd^d_ZEd`ejFdafddedfZG dddsdtZHdS)) annotationsN) bisect_right)IncrementalDecoder)aliases) lru_cache)findall) Generator)MultibyteIncrementalDecoder)ENCODING_MARKSIANA_SUPPORTED_SIMILARRE_POSSIBLE_ENCODING_INDICATIONUNICODE_RANGES_COMBINEDUNICODE_SECONDARY_RANGE_KEYWORDUTF8_MAXIMAL_ALLOCATIONCOMMON_CJK_CHARACTERS_LATIN_CJK_HANGUL _KATAKANA _HIRAGANA_THAI_ARABIC_ARABIC_ISOLATED_FORM_ACCENT_KEYWORDS _ACCENTUATED)maxsize characterstrreturnintcCszt|}Wn tyYdSwd}d|vr|tO}d|vr$|tO}d|vr,|tO}d|vr4|tO}d|vr<|tO}d|vrD|tO}d|vrT|t O}d |vrT|t O}t D] }||vrc|t O}|SqV|S) zRCompute all name-based classification flags with a single unicodedata.name() call.rZLATINCJKZHANGULZKATAKANAZHIRAGANAZTHAIZARABICz ISOLATED FORMN) unicodedataname ValueErrorrrrrrrrrrr)rdescflagskwr(^C:\Users\psant\Downloads\PAF backend\school\venv\Lib\site-packages\charset_normalizer/utils.py_character_flags&s: r*boolcCtt|t@SN)r+r*rrr(r(r)is_accentuatedIr/cCs.t|}|s |S|d}tt|ddS)N r)r" decompositionsplitchrr )rZ decomposedcodesr(r(r) remove_accentNs  r7ccs"|] \}}|j|j|fVqdSr-)startstop).0r#Z ord_ranger(r(r) [s  r;zlist[tuple[int, int, str]]_UNICODE_RANGES_SORTEDcCsg|]}|dqS)rr()r:er(r(r) _sr>z list[int]_UNICODE_RANGE_STARTS str | NonecCs<t|}tt|d}|dkrt|\}}}||kr|SdS)zK Retrieve the Unicode range official name from a single character. r rN)ordrr?r<)rZ character_ordidxr8r9r#r(r(r) unicode_rangebsrCcCr,r-)r+r*rr.r(r(r)is_latinsr0rDcCs2t|}d|vr dSt|}|durdSd|vS)NPTFZ Punctuationr"categoryrCrcharacter_categorycharacter_ranger(r(r)is_punctuationxs rKcCsBt|}d|vs d|vrdSt|}|durdSd|vo |dkS)NSNTFZFormsZLorFrHr(r(r) is_symbols rNcCs$t|}|dur dSd|vpd|vS)NFZ EmoticonsZ Pictographs)rC)rrJr(r(r) is_emoticonsrOcCs.|s|dvr dSt|}d|vp|dvS)N>>ZPdZPcZPo)isspacer"rG)rrIr(r(r) is_separators rUcCs||kSr-)islowerisupperr.r(r(r)is_case_variabler0rXcCr,r-)r+r*rr.r(r(r)is_cjkr0rYcCr,r-)r+r*rr.r(r(r) is_hiraganar0rZcCr,r-)r+r*rr.r(r(r) is_katakanar0r[cCr,r-)r+r*rr.r(r(r) is_hangulr0r\cCr,r-)r+r*rr.r(r(r)is_thair0r]cCr,r-)r+r*rr.r(r(r) is_arabicr0r^cCr,r-)r+r*rr.r(r(r)is_arabic_isolated_formr0r_cCs|tvSr-)rr.r(r(r)is_cjk_uncommonsr` range_namecstfddtDS)Nc3s|]}|vVqdSr-r()r:keywordrar(r)r;sz-is_unicode_range_secondary..)anyrrcr(rcr)is_unicode_range_secondarysrecCs(|duo|duo|dko|dkS)NFu)rT isprintabler.r(r(r)is_unprintables  rh sequencebytes | bytearray search_zonecCst|ttfs tt|}tt|dt||jddd}t|dkr&dS|D]'}| dd}t D]\}}||krD|S||krN|Sq6q(dS)zW Extract using ASCII-only decoder any specified encoding in the first n-bytes. Nasciiignoreerrorsr-_) isinstancebytes bytearray TypeErrorlenrr mindecodelowerreplaceritems)rjrlZseq_lenresultsZspecified_encodingencoding_alias encoding_ianar(r(r)any_specified_encodings&   rr#cCs |dvpttd|jtS)zQ Verify is a specific encoding is a multi byte one based on it IANA name > utf_16_beutf_8utf_32 utf_32_leutf_16 utf_16_leZ utf_8_sigutf_7 utf_32_be encodings.N) issubclass importlib import_modulerr )r#r(r(r)is_multi_byte_encoding s  rtuple[str | None, bytes]cCsJtD] }t|}t|tr|g}|D]}||r!||fSqqdS)z9 Identify and extract SIG/BOM in given sequence. )NN)r rsrt startswith)rj iana_encodingZmarksmarkr(r(r)identify_sig_or_boms  rrcCs|dvS)N>rrr()rr(r(r)should_strip_sig_or_bom0srTcp_namestrictcCsN|dd}tD]\}}|||fvr|Sq |r%td|d|S)zIReturns the Python normalized encoding name (Not the IANA official name).rqrrzUnable to retrieve IANA for ''N)rzr{rr|r$)rrr~rr(r(r) iana_name4s r iana_name_a iana_name_bfloatc Cst|st|r dStd|j}td|j}|dd}|dd}d}tdD]}t|g}||||krA|d7}q,|dS)Ngrrnrorr )rrrrrangertry) rrZ decoder_aZ decoder_bZid_aZid_bZcharacter_match_countiZ to_be_decodedr(r(r) cp_similarityEs    rcCs|tvo |t|vS)z Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using the function cp_similarity. N)r )rrr(r(r) is_cp_similarYs rZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)slevel format_stringNonecCs:t|}||t}|t|||dSr-)logging getLoggersetLevel StreamHandler setFormatter Formatter addHandler)r#rrloggerhandlerr(r(r)set_logging_handlerds  r sequencesroffsetsr chunk_sizebom_or_sig_availablestrip_sig_or_bom sig_payloadrtis_multi_byte_decoderdecoded_payloadGenerator[str, None, None]c cs&|r|dur|D]} || | |} | sdS| Vq dS|D]p} | |} | t|dkr/q || | |} |rA|durA|| } | j||rHdndd} |r| dkrt|d} |r| d| |vrt| | dd D]#}||| } |r{|dur{|| } | j|dd} | d| |vrqqi| Vq dS) NFrnrrorr2)rwryrxr)rrrrrrrrrrchunkZ chunk_endZ cut_sequenceZchunk_partial_size_chkjr(r(r)cut_sequence_chunksqsF       r)rrrr )rrrr+)rrrr)rrrr@)rarrr+)ri)rjrkrlr rr@)r#rrr+)rjrkrr)rrrr+)T)rrrr+rr)rrrrrr)rrrrrr+)r#rrr rrrrr-)rrkrrrrrr rr+rr+rrtrr+rr@rr)I __future__rrrr"bisectrcodecsrZencodings.aliasesr functoolsrrertypingrZ_multibytecodecr Zconstantr r r rrrrrrrrrrrrrrr*r/r7sortedr|r<__annotations__r?rCrDrKrNrOrUrXrYrZr[r\r]r^r_r`rwrerhrrrrrrrINFOrrr(r(r(r)s       L "                      "