jesopo
7db17c0962
add utils.parse.try_int() because .isdigit() isnt good enough
2019-09-26 13:44:38 +01:00
jesopo
2f49fb99e9
assume http fallback_encoding by content-type (utf8 for json)
2019-09-25 15:32:09 +01:00
jesopo
72649a90c2
only BeautifulSoup for finding encoding when it's a html-ish type
2019-09-20 13:38:00 +01:00
jesopo
efc0e197e5
Allow passing source Hostmask to IRCBatch
2019-09-19 18:16:10 +01:00
jesopo
e34259f967
log call was replaced with Exception but [] on args remained
2019-09-19 15:30:27 +01:00
jesopo
88a69aaa66
give Requests, use them in utils.http.request_many()
2019-09-19 14:54:44 +01:00
jesopo
d8e3a1c7ee
utils.http.request_() has no self, let alone self.log
2019-09-19 14:02:48 +01:00
jesopo
b69c9146b2
should be using pair_start/pair_end throughout for
2019-09-19 13:51:27 +01:00
jesopo
cd0d39ee5e
also show "bad" data in HTTPParsingException when a message is provided
2019-09-18 14:20:59 +01:00
jesopo
312f8906ae
show "bad" data in HTTPParsingException message
2019-09-18 10:52:05 +01:00
jesopo
a003c97fba
move q.close() to where it will be called even if deadline is hit
2019-09-18 10:24:01 +01:00
jesopo
dce6eee8c9
move _raise_deadline() out of except block to clean up printed stacktrace
2019-09-18 10:21:40 +01:00
jesopo
ee6360be22
don't check already-read data when checking for too-large requests
...
this check was here because the first read will return empty if it was an
invalid byte sequence for e.g. gzip because we needed to receive more data. the
second read will always return data (not decoded) so regardless of what the
already-read data is, the second read is the only criteria we need.
2019-09-17 17:33:23 +01:00
jesopo
1ac7f2697e
log which URL caused an error in request_many
2019-09-17 17:09:19 +01:00
jesopo
98545a9fb4
only decode content-types in DECODE_CONTENT_TYPES
2019-09-17 16:12:03 +01:00
jesopo
8ca0d30fef
Response.__init__() needs encoding now
2019-09-17 14:11:12 +01:00
jesopo
b7dd78ef1a
restore 5 second (instead of default 10) deadline for http.request
2019-09-17 13:44:14 +01:00
jesopo
94c3ff962b
use utils.deadline_process() in utils.http._request() so background threads can
...
call _request()
2019-09-17 13:41:11 +01:00
jesopo
fa95eaa9eb
add .get() to CaseInsensitiveDict
2019-09-17 13:40:37 +01:00
jesopo
d454f9b732
use Queue.get() with timeout, not Process.join() for timeout
...
this was because the threads spawned by multiprocessing.Queue seemed to be
making Process.join() believe the subprocess had not exited.
2019-09-17 13:39:23 +01:00
jesopo
1ed14f9a17
first draft of multiprocess.Process deadline system
2019-09-17 11:56:30 +01:00
jesopo
334d580c57
'seperate_hostmask()' -> 'parse_hostmask()'
2019-09-16 18:43:57 +01:00
jesopo
47735421b8
add json_body arg to Request to json-encode body, only return from body if
...
not null
2019-09-16 10:57:18 +01:00
jesopo
f9d13dc373
support '0' as an IntSetting value
2019-09-15 22:22:30 +01:00
jesopo
d950eb3660
add utils.SensitiveSetting, to .format() hide value
2019-09-12 12:17:31 +01:00
jesopo
ba0911f2e7
add utils.Setting.format() so subtypes can format differently
2019-09-12 12:17:09 +01:00
jesopo
9d33354d16
translate INVITE from [channel_name, target] to [target, channel_name]
2019-09-12 11:24:25 +01:00
jesopo
540c7b8c44
Revert "INVITE should be [channel_name, target]"
...
This reverts commit f3d8ffad2c
2019-09-12 11:23:29 +01:00
jesopo
f3d8ffad2c
INVITE should be [channel_name, target]
2019-09-12 11:21:29 +01:00
jesopo
77f50187c5
allow Requests to specify a useragent
2019-09-12 10:41:50 +01:00
jesopo
9d6a3982ed
add a helper utils.http.Client static object
2019-09-11 17:53:49 +01:00
jesopo
51dc26d113
add proxy to Request objects
2019-09-11 17:53:37 +01:00
jesopo
4a97c9eb0d
refactor utils.http.requests to support a Request object
2019-09-11 17:44:07 +01:00
jesopo
8f8cf92ae2
automatically decode certain http content types
2019-09-11 15:28:13 +01:00
jesopo
a9b106c6be
Don't try to .decode non-html things, default iso-lat-1 for non-html too
2019-09-09 16:17:26 +01:00
jesopo
b83f5d9e30
add flag to disable encoding detection
2019-09-09 14:59:08 +01:00
jesopo
5ef2b7af27
'str.split' -> 's.split'
2019-09-09 14:53:11 +01:00
jesopo
1df82c1cb2
still default to iso-latin-1 if no on-page or in-header content-type is present
2019-09-09 14:48:26 +01:00
jesopo
0a67659637
only look for <meta>-related tags when there are meta tags
2019-09-09 14:39:19 +01:00
jesopo
0a1077c5cd
add explicit None return for _find_encoding (mypy)
2019-09-09 14:25:01 +01:00
jesopo
ff9c82bf67
change utils.http.request to best-effort detect on-page encoding
...
closes #113
2019-09-09 14:11:18 +01:00
jesopo
007bb78d30
make utils.from_pretty_time() format much stricter
2019-09-04 11:22:56 +01:00
jesopo
397cfa8e7e
correctly qualify DeadlineExceededException namespace
2019-09-03 14:54:59 +01:00
jesopo
b7b2f31c1c
use utils.deadline() in utils.http.request, not raw sigalrm
2019-09-02 15:50:21 +01:00
jesopo
d42d694e64
move deadline alarm time check inside try/finally
2019-09-02 15:50:12 +01:00
jesopo
9cc1ee98eb
Pass the content of a webpage to HTTPParsingException
2019-09-02 13:27:44 +01:00
jesopo
408b89aeb7
use \S+ for url regex (for non-ascii chars), use url_sanitize to catch <>
2019-09-02 13:25:48 +01:00
jesopo
9e43ea1599
add per-user fediverse setting
2019-09-02 08:51:56 +01:00
jesopo
722d6418c8
switch to using signal.setitimer so we can restore previous timer
2019-08-30 18:36:48 +01:00
jesopo
a57a06b1cc
switch utils.deadline to a context manager
2019-08-30 18:13:12 +01:00
jesopo
e97e3c8f30
add utils.deadline(func, seconds) to put hard limits on code exec time
2019-08-30 17:17:03 +01:00
jesopo
4bfb4c3200
Add IntRangeSetting, reorder OptionsSetting arg order
2019-08-30 14:40:54 +01:00
jesopo
58a4fd7e74
round microseconds up to a whole number and zero-pad to 3 chars
2019-08-30 13:46:00 +01:00
jesopo
d3ed08e336
Don't use 'Z' for "utc timezone" - actually parse timezone out of datetimes
2019-08-30 13:25:19 +01:00
jesopo
b6361ba4e3
Remove extraneous space
2019-08-07 14:00:56 +01:00
jesopo
20042edfd9
Allow bypass of content-type check in utils.http.request
2019-08-05 15:41:02 +01:00
jesopo
d093027431
not all HTTP responses have content-type
2019-08-02 17:33:16 +01:00
jesopo
77dfc76591
switch to function/module magic being a single object
2019-07-26 11:58:06 +01:00
jesopo
c19c6c0e14
asyncio.gather -> asyncio.wait (with timeout)
2019-07-08 14:50:11 +01:00
jesopo
469c725675
tell asyncio.gather which loop to use
2019-07-08 14:41:12 +01:00
jesopo
a1438abf66
close event loop when we're done with it (request_many())
2019-07-08 13:59:48 +01:00
jesopo
81c7af8ab5
Don't try/except async http exceptions
2019-07-08 13:51:02 +01:00
jesopo
ee0ec0eca1
switch request_many() to use asyncio.gather
2019-07-08 13:46:27 +01:00
jesopo
b62ba469d7
catch async exceptions in utils.http.request_many()
2019-07-08 13:18:59 +01:00
jesopo
078681eddf
add missing schema in utils.http.sanitise_url, use in rss.py
2019-07-08 12:54:06 +01:00
jesopo
ecb8364d0d
switch to using asyncio's event loop
2019-07-08 12:45:10 +01:00
jesopo
15e143fcff
implement utils.http.request_many as a tonado ioloop yield
2019-07-08 11:43:09 +01:00
jesopo
637067c62c
url_validate() -> url_sanitise()
2019-07-02 14:15:49 +01:00
jesopo
534854127b
Add utils.http.url_validate() for best-effort url tidying
2019-07-02 14:10:18 +01:00
jan6
b6e9713a14
fixed capabilities
2019-07-01 14:41:47 +03:00
jesopo
19f716ab4c
Implemnt _options_factory for OptionsSetting to have dynamic options
2019-06-29 22:26:28 +01:00
jesopo
b4c762eb4e
Automatically format example text for OptionsSetting (showing options)
2019-06-29 21:33:26 +01:00
jesopo
7091860e54
Setting.__init__ call in OptionsSetting ctor needs self param
2019-06-28 23:27:56 +01:00
jesopo
756396d758
Implement utils.OptionsSetting
2019-06-28 23:26:42 +01:00
jesopo
ae9d099a41
Refactor set/channelset/serverset/botset in to 'utils.Setting' objects
2019-06-28 23:16:05 +01:00
jesopo
f9eb017466
message arg for HTTPWrongContentTypeException/HTTPParsingException
2019-06-28 23:01:21 +01:00
jesopo
97810db8df
Give descriptions to utils.http.HTTPException subclasses
2019-06-27 18:28:08 +01:00
jesopo
3c3f0449b6
add CaseInsensitiveDict.__contains__
2019-06-26 17:57:49 +01:00
jesopo
16d331dd43
add allow_redirects kwarg to utils.http.request()
2019-06-26 17:53:16 +01:00
jesopo
c5785a2d14
implement @utils.kwarg() magic, use it for command.regex hooks
2019-06-26 14:37:41 +01:00
jesopo
90c90e5bbd
Implement dependency system for CAPs
2019-06-21 18:05:11 +01:00
jesopo
8a0d99f969
Adda system of aliases for CAPs, mostly for changeable draft specs and creating
...
dependence between moving specs
2019-06-21 17:15:46 +01:00
jesopo
01ddc04bb9
fix hostmask_match_many type hinting
2019-06-20 17:58:49 +01:00
jesopo
b6e194a450
Implement hostmask_match_any, for more efficient matching of multiple hostmasks
2019-06-20 14:21:11 +01:00
jesopo
792a42be6e
Implement utils.irc.hostmask_match() as regex
2019-06-20 14:10:10 +01:00
jesopo
9457430c15
Add MessageTag.present, to see if a tag is present (even without value)
2019-06-19 22:36:42 +01:00
jesopo
ced59a0f74
Allow channel-access and channel-mode check for channels people are not in
2019-06-16 20:48:31 +01:00
jesopo
919a488e86
Implement client-to-server BATCHes
2019-06-16 09:52:54 +01:00
jesopo
8ab4880c5c
Change from yield checks to a func in events that pass up EventErrors
2019-06-15 18:42:14 +01:00
jesopo
e0d99fe8b1
Fix utils.Check args type hinting
2019-06-14 12:12:38 +01:00
jesopo
f05fc209b0
Add a way to __or__ utils.Checks, as a "if one of these passes" thing
2019-06-14 12:01:55 +01:00
jesopo
d7fa2cfa24
Catch yields in command callbacks for e.g. permission checks
2019-06-14 11:42:12 +01:00
jesopo
6c5dc958d0
Add .assure() to ParsedLine, to make it immune from .valid()
2019-06-10 10:45:02 +01:00
jesopo
2d46fe0cbf
labels: Use BatchType to match, pass lines not line, batch ID isn't a label,
...
fire event on label response
2019-06-04 17:54:19 +01:00
jesopo
2470c1ec03
Add new BatchType object, to match like how Capability and MessageTag do
2019-06-04 17:51:55 +01:00
jesopo
a802e66dcf
Defer decoding http payload bytestring until after checking ContentType
2019-06-04 13:47:03 +01:00
jesopo
ef221f4165
v1.8.0 release
2019-06-03 13:30:39 +01:00
jesopo
c212d70b68
Hold on to BATCH args, allow event hooks to manipulate batches
2019-06-02 15:18:36 +01:00
jesopo
5204ac3300
fix some CAP related type hints
2019-06-02 10:14:09 +01:00
jesopo
f9f637e084
Remove IRCSendBatch, fix some batch-related type hints
2019-06-02 10:13:51 +01:00
jesopo
367eb1c60e
Make labeled-resposnes it's own module, tag and track every sent line
2019-05-30 15:36:10 +01:00
jesopo
0be9046669
Pass str object to BeautifulSoup, not bytes. closes #56
2019-05-28 10:22:35 +01:00
jesopo
eeee2458d3
Fix utils.parse.hashflags type hints
2019-05-25 21:42:42 +01:00
jesopo
d0634bb54e
Don't store hashflags as a dict before returning them - allow duplicate keys
2019-05-25 12:58:07 +01:00
jesopo
ab1074cf4d
Remove mention of ipv4 - detect address family automatically
2019-05-21 10:11:09 +01:00
jesopo
db33e12c8c
Remove Capability.enabled() - it's unused
2019-05-20 16:36:02 +01:00
jesopo
b12c38c326
typing.List -> typing.Sequence
2019-05-20 16:35:50 +01:00
jesopo
e752b60f6c
use fnmatchcase( so the strings are not case-"normalised"
2019-05-19 16:28:46 +01:00
jesopo
c607a0e67c
Implement utils.irc.hostmask_match for glob-like hostmask matching
2019-05-19 16:24:38 +01:00
jesopo
58d6b16744
Remove _name and _draft_name (unused) and add Capability.copy()
2019-05-19 11:00:58 +01:00
jesopo
eccd43d357
Add a way to not put spaces between docstring lines when stripping \n
2019-05-18 18:34:27 +01:00
jesopo
38ba656bda
Add type hints to utils.irc.MessageTag
2019-05-14 23:07:42 +01:00
jesopo
421d9c10c7
Add utils.irc.MessageTag and use it for both draft and ratified msgid
2019-05-14 13:02:45 +01:00
jesopo
d291cd5063
Revamp how CAPs are tracked through REQ and ACK/NAK etc
2019-05-11 18:22:40 +01:00
jesopo
295f7a751c
Refactor CAPs in to objects
2019-05-11 15:11:09 +01:00
Patrick Nappa
2c344c9ddd
forgot the beautiful %
2019-05-03 13:50:51 +10:00
Patrick Nappa
471c11e229
ensure that non-url characters not separated by whitespace aren't consumed
2019-05-03 13:43:08 +10:00
jesopo
0afe798b83
Support parsing iso8601 with microseconds
2019-04-28 12:11:48 +01:00
jesopo
bdcb4b5db2
Add missing ":"
2019-04-25 17:50:41 +01:00
jesopo
1240b154cb
Support interfaces that don't have AF_INET and/or AF_INET6
2019-04-25 17:48:51 +01:00
jesopo
7643a962bd
Refuse to get the title for any url that points locall
2019-04-25 15:58:58 +01:00
jesopo
dffee4d223
Move REGEX_URL out of isgd.py and title.py in to utils.http
2019-04-24 15:46:54 +01:00
jesopo
e095c56f77
Record and show when a !to was created
2019-04-24 14:32:56 +01:00
jesopo
032f6fbc3b
IRCServer.alias is no longer optional
2019-04-24 11:45:21 +01:00
jesopo
87340bcbf9
change utils.iso8601_format to default to no milliseconds, switch to using
...
utils.iso8601 functions in badges.py
2019-04-23 21:27:43 +01:00
jesopo
0eaeca8f83
Put spaces between units in output from utils.to_pretty_time
2019-03-21 17:59:42 +00:00
jesopo
5d7f017c9c
Enum values shouldn't be all upper case
2019-03-10 13:14:25 +00:00
jesopo
6b59830a6e
Only print ANSI reset characters when appropriate
2019-03-06 21:48:31 +00:00
jesopo
197ae2e053
Raise a specific exception in utils.http.request for "wrong content type"
2019-02-28 23:28:45 +00:00
jesopo
9335ca5d16
Treat both a and a= as None for message-tags
2019-02-27 22:20:01 +00:00
jesopo
846b881e52
Throw ValueError when utils.http.request tries to soup non-html/xml data
2019-02-27 15:16:08 +00:00
jesopo
cfaf6864fc
Don't try to parse non-html/xml stuff with BeautifulSoup
2019-02-26 11:18:50 +00:00
jesopo
d627ed49e2
Pull "is main thread" logic out to utils, force Database to be accessed on main
...
thread
2019-02-25 10:36:17 +00:00
jesopo
8c94bcf6ca
Move utils.irc.IRCParsedLine to IRCLine.ParsedLine, improve truncation
...
mechanism, don't convert sent line from ParsedLine to text to ParsedLine for
line_handler handling
2019-02-23 21:33:04 +00:00
jesopo
c6699c509e
.upper() command in IRCParsedLine.format()
2019-02-23 16:11:26 +00:00
jesopo
54218b747f
Only allow the last arg of a formatted line to be :trailing
2019-02-23 16:07:20 +00:00
jesopo
47c2cbb0fa
Split line_handler.py out in to different files for different categories
2019-02-22 11:23:36 +00:00
jesopo
62eab40968
Revert "Treat both a and a= as having a value of None in message-tags (utils.irc)"
...
This reverts commit e7ec4b5716
2019-02-21 07:29:19 +00:00
jesopo
e7ec4b5716
Treat both a and a= as having a value of None in message-tags (utils.irc)
2019-02-20 22:26:31 +00:00
jesopo
ddb69685d3
again, r"\" isn't valid.
2019-02-20 16:53:40 +00:00
jesopo
98215f66eb
r"\" isn't valid - use "\\" (utils.irc)
2019-02-20 16:52:14 +00:00
jesopo
c1881fbafb
Remove '\' from post-unescaped message-tags
2019-02-20 16:34:14 +00:00
jesopo
47841ed154
don't replace '\\' with '\\' with message-tag unescape
2019-02-20 16:33:48 +00:00
jesopo
81d82413d4
draft/message-tags-0.3 (JSON tags) was rejected
2019-02-20 14:46:40 +00:00
jesopo
577fae7cea
Don't pass None around as tag default, fix some missing return type args
2019-02-19 15:57:00 +00:00
jesopo
3f8ac871ed
typo, 'utls' -> 'utils' (utils.irc)
2019-02-19 15:44:23 +00:00
jesopo
114d1054a5
Add .message( and .notice( to IRCSendBatch
2019-02-19 15:41:17 +00:00
jesopo
bc772ef5cc
Default IRCParsedLine.tags to empty dict, not None
2019-02-19 15:40:56 +00:00
jesopo
53e5a7fe7e
contextualise JOIN key check, otherwise it doesn't send channel_name
2019-02-19 09:32:17 +00:00
jesopo
9f946e236d
copy paste fail, batch end should be "-" not "+"
2019-02-19 08:25:25 +00:00
