Firepup650/bitbot-3.11-fork
2
0
Fork 0
Code Issues Pull requests Activity Actions
4488 commits 2 branches 0 tags 4.4 MiB
Commit graph

394 commits

Author SHA1 Message Date
jesopo
7ee65f8f8c remove src/utils/irc/protocol.py 2019-10-28 10:56:33 +00:00
jesopo
1bddc3b37f Revert "remove unneeded import" 
This reverts commit 8425c11c97.
 2019-10-27 10:32:13 +00:00
jesopo
8425c11c97 remove unneeded import 2019-10-27 10:25:37 +00:00
jesopo
8f4b5a0e70 move IRCLine related code from utils.irc to IRCLine.py 2019-10-27 10:19:00 +00:00
jesopo
3a755bb15f don't consume past 2nd digit in e.g. "\03033,123" 2019-10-25 17:12:24 +01:00
jesopo
f64131a10f support utf8 hostnames by punycode (idna) encoding 2019-10-18 10:58:24 +01:00
jesopo
2c19bdb949 add a fairly basic file locking mechanism with src/LockFile.py 
closes #96
 2019-10-10 12:11:03 +01:00
jesopo
0331b763ff refactor multi-line-to-line normalisation to utils.parse.line_normalise(), use it in rss.py 
closes #174
 2019-10-10 10:33:18 +01:00
jesopo
68aa89f16f commit FunctionSetting changes i forgot to commit yesterday 2019-10-08 11:38:56 +01:00
jesopo
9ab817ca58 parse out content_type in Response ctor 2019-10-05 22:56:56 +01:00
jesopo
b2473a4ac4 parse content-type out in utils.http.request, put it on Response object 2019-10-04 13:07:09 +01:00
jesopo
3466a3c43e Allow utils.Setting_ parse functions to throw detailed errors 2019-10-04 10:25:48 +01:00
jesopo
f306213cb8 'is_localhost()' -> 'host_permitted()' 2019-09-30 15:15:20 +01:00
jesopo
b9c64b7cf1 use ipaddress is_loopback etc to do better forbidden ranges 
closes #87
 2019-09-30 15:12:01 +01:00
jesopo
7db17c0962 add utils.parse.try_int() because .isdigit() isnt good enough 2019-09-26 13:44:38 +01:00
jesopo
2f49fb99e9 assume http fallback_encoding by content-type (utf8 for json) 2019-09-25 15:32:09 +01:00
jesopo
72649a90c2 only BeautifulSoup for finding encoding when it's a html-ish type 2019-09-20 13:38:00 +01:00
jesopo
efc0e197e5 Allow passing source Hostmask to IRCBatch 2019-09-19 18:16:10 +01:00
jesopo
e34259f967 log call was replaced with Exception but [] on args remained 2019-09-19 15:30:27 +01:00
jesopo
88a69aaa66 give Requests, use them in utils.http.request_many() 2019-09-19 14:54:44 +01:00
jesopo
d8e3a1c7ee utils.http.request_() has no self, let alone self.log 2019-09-19 14:02:48 +01:00
jesopo
b69c9146b2 should be using pair_start/pair_end throughout for 2019-09-19 13:51:27 +01:00
jesopo
cd0d39ee5e also show "bad" data in HTTPParsingException when a message is provided 2019-09-18 14:20:59 +01:00
jesopo
312f8906ae show "bad" data in HTTPParsingException message 2019-09-18 10:52:05 +01:00
jesopo
a003c97fba move q.close() to where it will be called even if deadline is hit 2019-09-18 10:24:01 +01:00
jesopo
dce6eee8c9 move _raise_deadline() out of except block to clean up printed stacktrace 2019-09-18 10:21:40 +01:00
jesopo
ee6360be22 don't check already-read data when checking for too-large requests 
this check was here because the first read will return empty if it was an
invalid byte sequence for e.g. gzip because we needed to receive more data. the
second read will always return data (not decoded) so regardless of what the
already-read data is, the second read is the only criteria we need.
 2019-09-17 17:33:23 +01:00
jesopo
1ac7f2697e log which URL caused an error in request_many 2019-09-17 17:09:19 +01:00
jesopo
98545a9fb4 only decode content-types in DECODE_CONTENT_TYPES 2019-09-17 16:12:03 +01:00
jesopo
8ca0d30fef Response.__init__() needs encoding now 2019-09-17 14:11:12 +01:00
jesopo
b7dd78ef1a restore 5 second (instead of default 10) deadline for http.request 2019-09-17 13:44:14 +01:00
jesopo
94c3ff962b use utils.deadline_process() in utils.http._request() so background threads can 
call _request()
 2019-09-17 13:41:11 +01:00
jesopo
fa95eaa9eb add .get() to CaseInsensitiveDict 2019-09-17 13:40:37 +01:00
jesopo
d454f9b732 use Queue.get() with timeout, not Process.join() for timeout 
this was because the threads spawned by multiprocessing.Queue seemed to be
making Process.join() believe the subprocess had not exited.
 2019-09-17 13:39:23 +01:00
jesopo
1ed14f9a17 first draft of multiprocess.Process deadline system 2019-09-17 11:56:30 +01:00
jesopo
334d580c57 'seperate_hostmask()' -> 'parse_hostmask()' 2019-09-16 18:43:57 +01:00
jesopo
47735421b8 add json_body arg to Request to json-encode body, only return from body if 
not null
 2019-09-16 10:57:18 +01:00
jesopo
f9d13dc373 support '0' as an IntSetting value 2019-09-15 22:22:30 +01:00
jesopo
d950eb3660 add utils.SensitiveSetting, to .format() hide value 2019-09-12 12:17:31 +01:00
jesopo
ba0911f2e7 add utils.Setting.format() so subtypes can format differently 2019-09-12 12:17:09 +01:00
jesopo
9d33354d16 translate INVITE from [channel_name, target] to [target, channel_name] 2019-09-12 11:24:25 +01:00
jesopo
540c7b8c44 Revert "INVITE should be [channel_name, target]" 
This reverts commit f3d8ffad2c.
 2019-09-12 11:23:29 +01:00
jesopo
f3d8ffad2c INVITE should be [channel_name, target] 2019-09-12 11:21:29 +01:00
jesopo
77f50187c5 allow Requests to specify a useragent 2019-09-12 10:41:50 +01:00
jesopo
9d6a3982ed add a helper utils.http.Client static object 2019-09-11 17:53:49 +01:00
jesopo
51dc26d113 add proxy to Request objects 2019-09-11 17:53:37 +01:00
jesopo
4a97c9eb0d refactor utils.http.requests to support a Request object 2019-09-11 17:44:07 +01:00
jesopo
8f8cf92ae2 automatically decode certain http content types 2019-09-11 15:28:13 +01:00
jesopo
a9b106c6be Don't try to .decode non-html things, default iso-lat-1 for non-html too 2019-09-09 16:17:26 +01:00
jesopo
b83f5d9e30 add flag to disable encoding detection 2019-09-09 14:59:08 +01:00
jesopo
5ef2b7af27 'str.split' -> 's.split' 2019-09-09 14:53:11 +01:00
jesopo
1df82c1cb2 still default to iso-latin-1 if no on-page or in-header content-type is present 2019-09-09 14:48:26 +01:00
jesopo
0a67659637 only look for <meta>-related tags when there are meta tags 2019-09-09 14:39:19 +01:00
jesopo
0a1077c5cd add explicit None return for _find_encoding (mypy) 2019-09-09 14:25:01 +01:00
jesopo
ff9c82bf67 change utils.http.request to best-effort detect on-page encoding 
closes #113
 2019-09-09 14:11:18 +01:00
jesopo
007bb78d30 make utils.from_pretty_time() format much stricter 2019-09-04 11:22:56 +01:00
jesopo
397cfa8e7e correctly qualify DeadlineExceededException namespace 2019-09-03 14:54:59 +01:00
jesopo
b7b2f31c1c use utils.deadline() in utils.http.request, not raw sigalrm 2019-09-02 15:50:21 +01:00
jesopo
d42d694e64 move deadline alarm time check inside try/finally 2019-09-02 15:50:12 +01:00
jesopo
9cc1ee98eb Pass the content of a webpage to HTTPParsingException 2019-09-02 13:27:44 +01:00
jesopo
408b89aeb7 use \S+ for url regex (for non-ascii chars), use url_sanitize to catch <> 2019-09-02 13:25:48 +01:00
jesopo
9e43ea1599 add per-user fediverse setting 2019-09-02 08:51:56 +01:00
jesopo
722d6418c8 switch to using signal.setitimer so we can restore previous timer 2019-08-30 18:36:48 +01:00
jesopo
a57a06b1cc switch utils.deadline to a context manager 2019-08-30 18:13:12 +01:00
jesopo
e97e3c8f30 add utils.deadline(func, seconds) to put hard limits on code exec time 2019-08-30 17:17:03 +01:00
jesopo
4bfb4c3200 Add IntRangeSetting, reorder OptionsSetting arg order 2019-08-30 14:40:54 +01:00
jesopo
58a4fd7e74 round microseconds up to a whole number and zero-pad to 3 chars 2019-08-30 13:46:00 +01:00
jesopo
d3ed08e336 Don't use 'Z' for "utc timezone" - actually parse timezone out of datetimes 2019-08-30 13:25:19 +01:00
jesopo
b6361ba4e3 Remove extraneous space 2019-08-07 14:00:56 +01:00
jesopo
20042edfd9 Allow bypass of content-type check in utils.http.request 2019-08-05 15:41:02 +01:00
jesopo
d093027431 not all HTTP responses have content-type 2019-08-02 17:33:16 +01:00
jesopo
77dfc76591 switch to function/module magic being a single object 2019-07-26 11:58:06 +01:00
jesopo
c19c6c0e14 asyncio.gather -> asyncio.wait (with timeout) 2019-07-08 14:50:11 +01:00
jesopo
469c725675 tell asyncio.gather which loop to use 2019-07-08 14:41:12 +01:00
jesopo
a1438abf66 close event loop when we're done with it (request_many()) 2019-07-08 13:59:48 +01:00
jesopo
81c7af8ab5 Don't try/except async http exceptions 2019-07-08 13:51:02 +01:00
jesopo
ee0ec0eca1 switch request_many() to use asyncio.gather 2019-07-08 13:46:27 +01:00
jesopo
b62ba469d7 catch async exceptions in utils.http.request_many() 2019-07-08 13:18:59 +01:00
jesopo
078681eddf add missing schema in utils.http.sanitise_url, use in rss.py 2019-07-08 12:54:06 +01:00
jesopo
ecb8364d0d switch to using asyncio's event loop 2019-07-08 12:45:10 +01:00
jesopo
15e143fcff implement utils.http.request_many as a tonado ioloop yield 2019-07-08 11:43:09 +01:00
jesopo
637067c62c url_validate() -> url_sanitise() 2019-07-02 14:15:49 +01:00
jesopo
534854127b Add utils.http.url_validate() for best-effort url tidying 2019-07-02 14:10:18 +01:00
jan6
b6e9713a14 fixed capabilities 2019-07-01 14:41:47 +03:00
jesopo
19f716ab4c Implemnt _options_factory for OptionsSetting to have dynamic options 2019-06-29 22:26:28 +01:00
jesopo
b4c762eb4e Automatically format example text for OptionsSetting (showing options) 2019-06-29 21:33:26 +01:00
jesopo
7091860e54 Setting.__init__ call in OptionsSetting ctor needs self param 2019-06-28 23:27:56 +01:00
jesopo
756396d758 Implement utils.OptionsSetting 2019-06-28 23:26:42 +01:00
jesopo
ae9d099a41 Refactor set/channelset/serverset/botset in to 'utils.Setting' objects 2019-06-28 23:16:05 +01:00
jesopo
f9eb017466 message arg for HTTPWrongContentTypeException/HTTPParsingException 2019-06-28 23:01:21 +01:00
jesopo
97810db8df Give descriptions to utils.http.HTTPException subclasses 2019-06-27 18:28:08 +01:00
jesopo
3c3f0449b6 add CaseInsensitiveDict.__contains__ 2019-06-26 17:57:49 +01:00
jesopo
16d331dd43 add allow_redirects kwarg to utils.http.request() 2019-06-26 17:53:16 +01:00
jesopo
c5785a2d14 implement @utils.kwarg() magic, use it for command.regex hooks 2019-06-26 14:37:41 +01:00
jesopo
90c90e5bbd Implement dependency system for CAPs 2019-06-21 18:05:11 +01:00
jesopo
8a0d99f969 Adda system of aliases for CAPs, mostly for changeable draft specs and creating 
dependence between moving specs
 2019-06-21 17:15:46 +01:00
jesopo
01ddc04bb9 fix hostmask_match_many type hinting 2019-06-20 17:58:49 +01:00
jesopo
b6e194a450 Implement hostmask_match_any, for more efficient matching of multiple hostmasks 2019-06-20 14:21:11 +01:00
jesopo
792a42be6e Implement utils.irc.hostmask_match() as regex 2019-06-20 14:10:10 +01:00
jesopo
9457430c15 Add MessageTag.present, to see if a tag is present (even without value) 2019-06-19 22:36:42 +01:00
jesopo
ced59a0f74 Allow channel-access and channel-mode check for channels people are not in 2019-06-16 20:48:31 +01:00
jesopo
919a488e86 Implement client-to-server BATCHes 2019-06-16 09:52:54 +01:00
jesopo
8ab4880c5c Change from yield checks to a func in events that pass up EventErrors 2019-06-15 18:42:14 +01:00
jesopo
e0d99fe8b1 Fix utils.Check args type hinting 2019-06-14 12:12:38 +01:00
jesopo
f05fc209b0 Add a way to __or__ utils.Checks, as a "if one of these passes" thing 2019-06-14 12:01:55 +01:00
jesopo
d7fa2cfa24 Catch yields in command callbacks for e.g. permission checks 2019-06-14 11:42:12 +01:00
jesopo
6c5dc958d0 Add .assure() to ParsedLine, to make it immune from .valid() 2019-06-10 10:45:02 +01:00
jesopo
2d46fe0cbf labels: Use BatchType to match, pass lines not line, batch ID isn't a label, 
fire event on label response
 2019-06-04 17:54:19 +01:00
jesopo
2470c1ec03 Add new BatchType object, to match like how Capability and MessageTag do 2019-06-04 17:51:55 +01:00
jesopo
a802e66dcf Defer decoding http payload bytestring until after checking ContentType 2019-06-04 13:47:03 +01:00
jesopo
ef221f4165 v1.8.0 release 2019-06-03 13:30:39 +01:00
jesopo
c212d70b68 Hold on to BATCH args, allow event hooks to manipulate batches 2019-06-02 15:18:36 +01:00
jesopo
5204ac3300 fix some CAP related type hints 2019-06-02 10:14:09 +01:00
jesopo
f9f637e084 Remove IRCSendBatch, fix some batch-related type hints 2019-06-02 10:13:51 +01:00
jesopo
367eb1c60e Make labeled-resposnes it's own module, tag and track every sent line 2019-05-30 15:36:10 +01:00
jesopo
0be9046669 Pass str object to BeautifulSoup, not bytes. closes #56 2019-05-28 10:22:35 +01:00
jesopo
eeee2458d3 Fix utils.parse.hashflags type hints 2019-05-25 21:42:42 +01:00
jesopo
d0634bb54e Don't store hashflags as a dict before returning them - allow duplicate keys 2019-05-25 12:58:07 +01:00
jesopo
ab1074cf4d Remove mention of ipv4 - detect address family automatically 2019-05-21 10:11:09 +01:00
jesopo
db33e12c8c Remove Capability.enabled() - it's unused 2019-05-20 16:36:02 +01:00
jesopo
b12c38c326 typing.List -> typing.Sequence 2019-05-20 16:35:50 +01:00
jesopo
e752b60f6c use fnmatchcase( so the strings are not case-"normalised" 2019-05-19 16:28:46 +01:00
jesopo
c607a0e67c Implement utils.irc.hostmask_match for glob-like hostmask matching 2019-05-19 16:24:38 +01:00
jesopo
58d6b16744 Remove _name and _draft_name (unused) and add Capability.copy() 2019-05-19 11:00:58 +01:00
jesopo
eccd43d357 Add a way to not put spaces between docstring lines when stripping \n 2019-05-18 18:34:27 +01:00
jesopo
38ba656bda Add type hints to utils.irc.MessageTag 2019-05-14 23:07:42 +01:00
jesopo
421d9c10c7 Add utils.irc.MessageTag and use it for both draft and ratified msgid 2019-05-14 13:02:45 +01:00
jesopo
d291cd5063 Revamp how CAPs are tracked through REQ and ACK/NAK etc 2019-05-11 18:22:40 +01:00
jesopo
295f7a751c Refactor CAPs in to objects 2019-05-11 15:11:09 +01:00
Patrick Nappa
2c344c9ddd forgot the beautiful % 2019-05-03 13:50:51 +10:00
Patrick Nappa
471c11e229 ensure that non-url characters not separated by whitespace aren't consumed 2019-05-03 13:43:08 +10:00
jesopo
0afe798b83 Support parsing iso8601 with microseconds 2019-04-28 12:11:48 +01:00
jesopo
bdcb4b5db2 Add missing ":" 2019-04-25 17:50:41 +01:00
jesopo
1240b154cb Support interfaces that don't have AF_INET and/or AF_INET6 2019-04-25 17:48:51 +01:00
jesopo
7643a962bd Refuse to get the title for any url that points locall 2019-04-25 15:58:58 +01:00
jesopo
dffee4d223 Move REGEX_URL out of isgd.py and title.py in to utils.http 2019-04-24 15:46:54 +01:00
jesopo
e095c56f77 Record and show when a !to was created 2019-04-24 14:32:56 +01:00
jesopo
032f6fbc3b IRCServer.alias is no longer optional 2019-04-24 11:45:21 +01:00
jesopo
87340bcbf9 change utils.iso8601_format to default to no milliseconds, switch to using 
utils.iso8601 functions in badges.py
 2019-04-23 21:27:43 +01:00
jesopo
0eaeca8f83 Put spaces between units in output from utils.to_pretty_time 2019-03-21 17:59:42 +00:00
jesopo
5d7f017c9c Enum values shouldn't be all upper case 2019-03-10 13:14:25 +00:00
jesopo
6b59830a6e Only print ANSI reset characters when appropriate 2019-03-06 21:48:31 +00:00
jesopo
197ae2e053 Raise a specific exception in utils.http.request for "wrong content type" 2019-02-28 23:28:45 +00:00
jesopo
9335ca5d16 Treat both a and a= as None for message-tags 2019-02-27 22:20:01 +00:00
jesopo
846b881e52 Throw ValueError when utils.http.request tries to soup non-html/xml data 2019-02-27 15:16:08 +00:00
jesopo
cfaf6864fc Don't try to parse non-html/xml stuff with BeautifulSoup 2019-02-26 11:18:50 +00:00
jesopo
d627ed49e2 Pull "is main thread" logic out to utils, force Database to be accessed on main 
thread
 2019-02-25 10:36:17 +00:00
jesopo
8c94bcf6ca Move utils.irc.IRCParsedLine to IRCLine.ParsedLine, improve truncation 
mechanism, don't convert sent line from ParsedLine to text to ParsedLine for
line_handler handling
 2019-02-23 21:33:04 +00:00
jesopo
c6699c509e .upper() command in IRCParsedLine.format() 2019-02-23 16:11:26 +00:00
jesopo
54218b747f Only allow the last arg of a formatted line to be :trailing 2019-02-23 16:07:20 +00:00